paper-fetch
Paper fetch
From "a paper" to a file of it you can trust: which paper it is, what is known about it, which versions exist, and its PDF, checked against the paper and labelled with the version it is. Everything that touches the web is here; Zotero is paper-to-zotero's. Script paths are relative to this file's directory: cd there at the start of every shell call. uv run scripts/pf.py --help lists every script; each script's --help is the reference for its flags, codes and output.
Scripts do the certain steps; agents do the rest. A script goes as far as it can be certain — a request, a check, a file filed — and where it cannot (a page it gets no PDF from, a file it cannot be sure of, a version to settle) it writes a handoff and goes on with the other papers. An agent reads the handoff's evidence and acts: it hands the page back to the script (fetch --start), or answers with pf.py answer; the script's next run carries on from there.
| operation | where |
|---|---|
| Identify a paper (a title, a link, an id) | Identify a paper below |
| Get its file | Ready, then Get the file: the chain below |
| Who takes a handoff, how to answer it, the agents' briefs | Handoffs below |
| Label files someone already has (a library's PDFs, the user's own) | Label files you already have below |
| Many papers, in passes | references/batch.md, before Ready |
| A page only the user can pass | references/gate.md |
| Identify only, no file | references/identify.md |
The directory. A run lives in one directory, <dir>: the caller's (an import: <scratch>/paper-to-zotero/fetch/), else <scratch>/paper-fetch/ — <scratch> is the session's scratchpad, else ${TMPDIR:-/tmp}. Each paper is one result file, <dir>/results/<slug>.json, with its files beside it; a caller reads results/ and nothing else (scripts/pflib/run.py documents the format). The slug is the caller's citation key in file-safe form (the key itself is kept in cite), or <surname><year><first title word>. What waits for an agent is a record in <dir>/handoffs/.
The user helps; the work is yours. The run is the same whether or not the user is watching: you and your agents fetch every paper. The user may log in, pass a CAPTCHA or click a download at any moment, and that only speeds things up — the watcher files their downloads. Say early what you can foresee needing them for ("3 of these need the institutional login — if you can, log in now: Ctrl+N, a normal window"). Only the last pass is planned around them; before it, a gate waits for them about ten minutes and sees for itself when they are through, without reloading the page (references/gate.md).