DeepImageSearch: Agentic Multi-Step Image Retrieval over Visual Histories

This skill teaches Claude to build image retrieval systems that go beyond single-query semantic matching. Based on the DeepImageSearch paradigm, it reformulates image retrieval as an autonomous exploration task where an agent plans search trajectories, coordinates fine-grained perception tools, and connects scattered clues distributed across temporal sequences of images. The core insight is that target relevance often depends on other images in the collection (P(R|Q,C)), not just isolated query-image similarity -- requiring corpus-level contextual reasoning with a modular agent framework and dual-memory architecture.

When to Use

• When the user needs to search a large personal photo library using contextual clues that span multiple events (e.g., "find photos of the dog we saw at the same park where we had Sarah's birthday")
• When building an agent that must reason across temporal sequences of images rather than matching against individual frames
• When implementing a retrieval system where queries are ambiguous without cross-referencing metadata, timestamps, locations, or recurring entities across images
• When designing a modular tool-use agent with memory management for long-horizon visual navigation tasks
• When constructing benchmarks for evaluating multimodal agents on context-dependent retrieval
• When the user wants to build a memory graph over a photo collection linking people, places, events, and visual clues

Key Technique

Agentic Retrieval vs. Embedding-Based Retrieval: Traditional systems encode a query and each image independently, then rank by cosine similarity. DeepImageSearch shows this hits a fundamental ceiling (~10-14% Recall@3) on context-dependent queries because the answer depends on relationships between images. The agentic approach instead gives a model a toolkit -- similarity search, metadata filtering, photo viewing, web lookup -- and lets it plan multi-step reasoning chains: narrowing candidates by time/location, visually inspecting subsets, cross-referencing recurring entities, and iteratively refining.

Dual-Memory System for Long Horizons: When exploring large photo histories (100K+ images spanning years), conversation context fills up fast. The framework uses two memory layers: (1) Explicit state memory -- named photo subsets persisted as dictionary mappings (subset_name -> photo_IDs) enabling chained operations like filter->save->search_within->filter, and (2) Compressed context memory triggered at token limits, which distills the full interaction history into a session memory (high-level goals and key findings) plus a working memory (current subgoal and immediate action plans). This lets the agent maintain reasoning coherence across dozens of tool calls without losing track of its search trajectory.