Add-on: Header/Footer Cleanup (Repeated Lines)

Use this skill to implement the cleanup stage that removes repeated header/footer noise from LLM-facing text while preserving raw page content for traceability.

Inputs

Collect:

• REPEAT_THRESHOLD_RATIO: default 0.6 (line must appear on >= 60% of pages to be a removal candidate).
• TOP_BAND_LINES: default 3 (lines considered “header band”).
• BOTTOM_BAND_LINES: default 3 (lines considered “footer band”).
• MIN_LINE_LEN: default 6 (ignore tiny lines for repetition counting).

Algorithm (Deterministic)

1. For each page, split into lines.
2. Normalize each line for comparison:
   • NFKC normalize
   • lowercase
   • collapse whitespace