bigquery-pipeline-audit
Audits Python + BigQuery pipelines for cost safety, idempotency, and production readiness with exact patch locations.
- Analyzes every BigQuery job trigger and external API call to identify cost exposure, loop-driven query multiplication, and missing
maximum_bytes_billedlimits - Enforces dry-run and execute modes with explicit prod confirmation, partition filter validation, and scan-size optimization
- Validates idempotent writes using MERGE, staging tables, or dedup logic; flags unsafe append patterns and duplicate-prone reruns
- Generates structured reports with PASS/FAIL verdicts per section, ranked patch list, and worst-case cost estimates in job count and bytes
BigQuery Pipeline Audit: Cost, Safety and Production Readiness
You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible.
Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites.
A) COST EXPOSURE: What will actually get billed?
Locate every BigQuery job trigger (client.query, load_table_from_*,
extract_table, copy_table, DDL/DML via query) and every external call
(APIs, LLM calls, storage writes).
For each, answer:
More from github/awesome-copilot
git-commit
Execute git commit with conventional commit message analysis, intelligent staging, and message generation. Use when user asks to commit changes, create a git commit, or mentions "/commit". Supports: (1) Auto-detecting type and scope from changes, (2) Generating conventional commit messages from diff, (3) Interactive commit with optional type/scope/description overrides, (4) Intelligent file staging for logical grouping
30.2Kgh-cli
GitHub CLI (gh) comprehensive reference for repositories, issues, pull requests, Actions, projects, releases, gists, codespaces, organizations, extensions, and all GitHub operations from the command line.
21.2Kprd
Generate high-quality Product Requirements Documents (PRDs) for software systems and AI-powered features. Includes executive summaries, user stories, technical specifications, and risk analysis.
17.4Kdocumentation-writer
Diátaxis Documentation Expert. An expert technical writer specializing in creating high-quality software documentation, guided by the principles and structure of the Diátaxis technical documentation authoring framework.
17.4Kexcalidraw-diagram-generator
Generate Excalidraw diagrams from natural language descriptions. Use when asked to "create a diagram", "make a flowchart", "visualize a process", "draw a system architecture", "create a mind map", or "generate an Excalidraw file". Supports flowcharts, relationship diagrams, mind maps, and system architecture diagrams. Outputs .excalidraw JSON files that can be opened directly in Excalidraw.
16.4Krefactor
Surgical code refactoring to improve maintainability without changing behavior. Covers extracting functions, renaming variables, breaking down god functions, improving type safety, eliminating code smells, and applying design patterns. Less drastic than repo-rebuilder; use for gradual improvements.
16.1K