GPT-2 Code Golf

Overview

This skill provides guidance for implementing GPT-2 or similar transformer model inference in minimal code, typically for code golf challenges. These tasks require parsing binary checkpoint formats, implementing BPE tokenization, and performing forward passes through transformer architectures—all within strict size constraints.

Critical Principles

1. Verify Before Optimizing

Never optimize for size before functionality is verified. A working 10KB solution is better than a broken 4KB solution. Size optimization should be the final step, not a constraint during development.

2. Incremental Development and Testing

Build and test components independently before integration:

1. Checkpoint parsing - Verify weights load correctly
2. Tokenization - Verify BPE encoding/decoding works
3. Forward pass - Verify matrix operations produce reasonable outputs