periplus-data-extraction
Data extraction
Produce usable rows from source content. For a known small source, inspect it and proceed; do not require a separate plan or an unnecessary crawl. Use existing files or a more suitable tool when they already satisfy the request.
Define rows and fields
Identify the record unit: product, variant, offer, person, location or another explicit entity. Specify requested fields, types, units and repeated-value handling. Preserve original strings alongside normalized values where conversion could lose meaning. For prices, retain currency, billing period, tax basis and market when available; “from”, “contact us”, missing and zero are different values.
Inspect, extract, expand
Inspect source structure on contrasting examples before generalizing: repeated cards versus detail pages, an alternate layout, a missing price and a record with multiple values. Check whether the captured document actually contains the requested content. An empty client-rendered panel is an acquisition gap. Prefer an available structured source when suitable; treat JSON fields as JSON rather than HTML selectors.
Extract each field within its own record container. Keep capture identity with node identity, and test field alignment when optional elements are absent. Avoid pairing independent title and price arrays by position. Follow a detail-page link when the listing omits a required field. Expand paging URLs, linked records or bounded discovery as needed for the requested coverage.
Check source-stated totals against rendered records and extracted rows. Resolve duplicates at the chosen unit; do not collapse legitimate variants. Inspect failed acquisitions and SQL truncation separately from missing source fields. A page limit limits acquisition, and a result limit limits returned rows; neither guarantees full extraction. Report a partial result with its exact coverage when a remaining gap cannot be resolved within scope.
Validate the output
Compare extracted records with the source for different layouts and edge cases. Check null rates, repeated values, field shifts, malformed numbers, inconsistent units and key uniqueness. Route exceptions to explicit states such as unavailable in source, acquisition failed, parse failed or ambiguous; do not invent missing values. Preserve repeated values when flattening would change meaning.