Media Understanding

Use qwen-mm-plugins-api to understand media through configured model services. The tools are grouped by model family:

• VL model (Qwen-VL, OpenAI-compatible endpoint): vision_chat, ocr, grounding.
• Omni model (Qwen-Omni — combines video frames and audio): perceive_media, omni_asr, omni_asr_timestamped, omni_multi_speaker_asr, omni_av_caption, omni_av_grounding, omni_av_counting, omni_music_caption.
• Other services: transcribe_audio (Qwen3-ASR), segmentation (a SAM3 server).

Check the qwen-mm-plugins-api tools in your tool list for full schemas and parameters. For file reading, rendering, or metadata inspection, use core.

When to Use Which Tool

VL model (single images/videos, spatial reasoning):

• Ask a VLM about images/videos (caption, VQA, free-form) → vision_chat
• Extract text from an image → ocr
• Detect/locate objects in an image (bounding boxes, spatial WHERE) → grounding. It sends EXIF-corrected pixels, so returned 0–1000 boxes address the displayed image and can be passed directly to core crop/draw_bbox or search image_search.

Omni model (audio + video together, temporal reasoning):