file-processing Skill

Goal

This skill guides the handling of binary data and large datasets without blowing up RAM.

Steps

1. Streaming (Chunks)

• Problem: Reading 1GB CSV into memory (RAM).
• Solution: Stream it line by line.
  • nodejs: fs.createReadStream().pipe(csvParser()).
  • python: pd.read_csv(chunksize=1000).

2. Uploads (Presigned URLs)

• Anti-Pattern: Client -> Server (RAM buffer) -> S3.
• Pattern: Client -> (Request URL) -> Server; Server -> (Signed URL) -> Client; Client -> (Direct Upload) -> S3.
• Benefit: Server CPU/RAM is zero during upload. Infinite scalability.