tokenize-then-parse

When to Use

• Building interpreters or compilers
• Processing parenthesized expressions
• Structured text with clear token boundaries
• Multi-stage processing pipeline

When NOT to Use

• Simple fixed format (just split)
• Very complex grammar (use parser generator)
• No clear token boundaries

The Pattern

Tokenize text into a stream of tokens, then parse tokens into a structure.