chkit ingestion authoring

Implement a recoverable source reader with finite runs. Inspect the installed plugin version, config, and existing streams; preserve the user's choices.

Decide before coding

Establish auth, response shape, pagination, record IDs, change filters/cursors, deletes, volume, and freshness needs from code and provider docs. Ask for missing facts that affect implementation. Verify provider capabilities.

Recommend normalized raw ingestion by default: sync provider resources as directly as possible into independent streams and raw destinations, with source and parent identifiers for relationships. Normalize resource boundaries while preserving each provider object's fields and structure. Perform field transformations, business mappings, joins, enrichment, and denormalization afterward in ClickHouse SQL views, materialized views, or derived tables. Keep readers focused on source retrieval, pagination, checkpointing, and the minimal identity/envelope mapping needed to store raw observations. Use a shaped ingestion schema or projected/embedded document model when the user explicitly requests it; retain inseparable bounded objects as the provider returns them.

Choose the strongest appropriate sync mechanism the provider actually supports: a change feed or sync token, reliable modification windows, or full reads with safe replay. A simple full read is appropriate when the API lacks reliable change selection and its cost is acceptable, as well as for small catalogs or explicitly requested prototypes. Use the default loader unless another publication contract is needed. Explain consequential choices and coverage limits.

Use one pipeline per account or installation, with independently selectable streams per resource type or configured collection. People and Companies need independent streams; individual people remain records within their collection. Prefer a raw destination per resource type; share one when resource types have the same provider representation and retain an explicit resource discriminator. A pipeline shares execution defaults; its stream order does not provide discovery dependencies. Parent-scoped endpoints may require parent enumeration inside the child reader, but each child stream performs its own discovery and owns its checkpoints. Parent publication does not wait for child streams.

Prefer existing paginate() and incremental strategies. Add custom state only when a provider's recovery contract needs progress beyond the executor's journal; ordinary fullSync() already records successful work, including empty reads. Keep editable configuration, authenticated request handling, and meaningful resource readers distinct without creating one-line proxy modules or a generic scan framework. Bind factory configuration to both readers and strategies, keep setup-time raw table definitions explicit, and test that configured streams can run independently. Put repeated generic mechanics in the library when the existing API cannot express them concisely.

Prefer coarse page or parent checkpoints when replay cost is acceptable. Read a unit and its required children with ordinary pagination, then publish its completion state after its rows; interruption replays the unfinished unit. Add finer child positions only when needed to make progress within execution budgets. Choose revisit windows from the user's volume and freshness requirements; an explicit recent-only policy does not require retained-object queues or historical reconciliation.

paginate() yields full { items, next, metadata? } pages, including empty terminal pages. Use page.items for rows and carry provider checkpoint candidates in metadata, then explicitly yield them as chunk state with cursorState. next is the request continuation; metadata is not automatically durable progress. Keep checkpoint snapshots fresh and complete, and let paginate own the request attempt rather than nesting context.attempt inside it. See the stable-sync reference for sync-token boundaries.