incremental-fetch
Incremental Fetch
Build data pipelines that never lose progress and never re-fetch existing data.
The Two Watermarks Pattern
Track TWO cursors to support both forward and backward fetching:
| Watermark | Purpose | API Parameter |
|---|---|---|
newest_id |
Fetch new data since last run | since_id |
oldest_id |
Backfill older data | until_id |
A single watermark only fetches forward. Two watermarks enable:
- Regular runs: fetch NEW data (since
newest_id) - Backfill runs: fetch OLD data (until
oldest_id) - No overlap, no gaps
More from shipshitdev/skills
security-expert
Expert in application security, OWASP Top 10, authentication, authorization, data protection, and security best practices for React, Next.js, and NestJS applications
8testing-expert
Expert in testing strategies for React, Next.js, and NestJS applications covering unit tests, integration tests, E2E tests, and testing best practices
8performance-expert
Expert in performance optimization for React, Next.js, NestJS applications covering frontend rendering, API response times, database queries, and infrastructure optimization
8brand-architect
Use this skill when users need to develop brand strategy, choose a company name, define brand positioning, create brand voice, or build brand identity from day one. Activates for "what should I name it," "brand strategy," "positioning," or identity questions.
8expo-architect
Scaffold a production-ready Expo React Native app with working screens, navigation, and optional Clerk auth. Generates complete mobile app structure that runs immediately with `bun start`.
7nextjs-validator
Validate Next.js 16 configuration and detect/prevent deprecated patterns. Ensures proxy.ts usage, Turbopack, Cache Components, and App Router best practices. Use before any Next.js work or when auditing existing projects.
7