Multimodal Orchestration

AI interactions increasingly span multiple modalities — text, images, voice, code, tools, structured data. Designing how these modalities work together is orchestration. The risk on every team: switching modality because it's available, not because it serves the user.

The point of multimodal isn't to use every modality. It's to use the right one at the right moment, and to handle the seams.

Modality selection

Each modality has strengths:

• Text: precise, editable, referenceable. Best for instructions, explanations, and nuanced content.
• Image: spatial, holistic, immediate. Best for layouts, diagrams, and visual concepts.
• Voice: natural, hands-free, emotional. Best for conversational flow and accessibility.
• Code: executable, precise, verifiable. Best for technical specifications and automation.
• Structured data: tables, forms, JSON. Best for comparison, configuration, and data entry.
• Tool use: actions in external systems. Best for execution, not generation.

The designer decides which modality the AI uses for each part of a response — and which modality the user is invited to use for input.

Cross-modal transitions