incremental-fetch
Incremental Fetch
Build data pipelines that never lose progress and never re-fetch existing data.
The Two Watermarks Pattern
Track TWO cursors to support both forward and backward fetching:
| Watermark | Purpose | API Parameter |
|---|---|---|
newest_id |
Fetch new data since last run | since_id |
oldest_id |
Backfill older data | until_id |
A single watermark only fetches forward. Two watermarks enable:
- Regular runs: fetch NEW data (since
newest_id) - Backfill runs: fetch OLD data (until
oldest_id) - No overlap, no gaps
More from shipshitdev/library
financial-operations-expert
Use this skill when users need help with business finances, tax planning, bookkeeping, profit/loss analysis, cash flow management, or multi-business financial tracking. Activates for "am I profitable," tax questions, accounting setup, or financial health checks.
1.6Kyoutube-video-analyst
Forensic-level deconstruction of YouTube videos to extract viral formulas, hooks, retention mechanics, and emotional engineering. Use when analyzing video transcripts to clone success patterns for new content.
897nestjs-testing-expert
Testing patterns for NestJS apps using Jest, covering unit, integration, and e2e tests.
546copywriter
Brand voice guardian and conversion-focused copywriter, specializing in direct, no-fluff copy that adapts to project's brand voice
378brand-name-generator
Generate creative brand names, company names, product names, or startup names. Includes naming strategies, brandability scoring, and domain availability checking.
317competitive-intelligence-analyst
Use this skill when users need to analyze competitors, monitor market movements, benchmark features/pricing, identify market gaps, or understand competitive positioning. Activates for "what are competitors doing," market analysis, or differentiation strategy.
301