The edge is in READING and REASONING, not computing.

The Information Timing Ladder (from empirical HN research)

Day 0: SEC posts earnings → human analysts read first (after close) Day 1: Earnings call → transcripts uploaded → few HFs apply NLP Day 1+: Data providers (Bloomberg, Refinitiv) structure transcript data Day 2: Bloomberg writes article → retail FOMO begins Days 2-10+: PEAD (Post-Earnings Announcement Drift) — retail chases

Your job is to operate at Day 0-1 by reading PRIMARY sources (filings, transcripts, FT/WSJ/SA reporting) BEFORE the narrative crystallizes. If it's already on Reddit/fintwit/CNBC, you're at Day 5+ and the signal is gone.

Cross-Validation Principle

Single signal = noise. Convergence = signal:

1 source mentions a theme: noise (file for later)
3+ independent sources in 3+ weeks: something is forming
Insider buying (Form 4 cluster) + analyst upgrades + supply constraint language in filings: highest-confidence opportunity

This skill is designed for parallel execution. Steps 1 and 2 should be fanned out across multiple subagents reading different sources simultaneously. This is the agent superpower — breadth of reading that a human cannot match in one sitting.

Execution architecture

ORCHESTRATOR (you)
  │
  ├─ Step 1: run emerging_scan.py yourself (fast, 30s)
  │           → produces: list of 3-5 hot sectors/themes
  │
  ├─ Step 2: FAN OUT subagents in parallel (one per source × theme):
  │   ├─ Subagent A: "Read Seeking Alpha for <theme_1>"
  │   ├─ Subagent B: "Read WSJ for <theme_1>"
  │   ├─ Subagent C: "Read Financial Times for <theme_1>"
  │   ├─ Subagent D: "Read Seeking Alpha for <theme_2>"
  │   ├─ Subagent E: "Read WSJ for <theme_2>"
  │   ├─ Subagent F: "Search SEC EDGAR for supply-constrained filings in <theme_1>"
  │   └─ ... (as many as needed — one subagent per source × theme)
  │
  │   Each subagent returns: { demand_inflections[], companies_mentioned[], source_citations[] }
  │
  ├─ Steps 3-4: SYNTHESIZE subagent findings yourself (reasoning, not reading)
  │   - Map non-obvious beneficiaries from the combined findings
  │   - Apply skeptic filter to every candidate
  │
  └─ Step 5: Route finalists to multi-lens-quorum

Subagent prompt template

When spawning research subagents, use this prompt structure for each:

<subagent_prompt>
You are a financial research reader. Your ONLY job is to read <SOURCE> for information about
<THEME/SECTOR>.

Search for: <specific_search_pattern>

Extract and return ONLY factual findings in this format:
- Demand inflections found (quote the source):
- Companies mentioned and their role in the supply chain:
- Bottleneck/constraint language (exact quotes):
- Non-obvious suppliers or beneficiaries named:
- Source URL and quality assessment (filing-backed vs narrative):

Do NOT speculate. Do NOT recommend. Only report what you READ.
If you find nothing relevant, say "No relevant findings for <theme> in <source>."
</subagent_prompt>

Why parallel: the math

A human reads ~1 article in 5 minutes. 3 sources × 3 themes = 9 articles = 45 minutes sequential. With 9 parallel subagents, you get all 9 readings in ~60 seconds. The orchestrator then spends 2-3 minutes on synthesis (Steps 3-5). Total: ~4 minutes vs 50+ minutes. This is the scalable advantage of an agent team reading financial journalism.

How to read articles (including paywalled sources)

USE THE BROWSER. The user has bypass-paywalls-chrome installed. Open the article URL in the browser and read document.body.innerText — the extension handles access transparently. No tricks needed for FT, WSJ, Bloomberg, SA, NYT, The Economist, Barron's, and 100+ more.

Use WHICHEVER browser tool is available in your environment (check in this priority order):

chrome-use CLI (preferred) — drives the user's real Chrome with their extensions:

chrome-use open "https://www.ft.com/content/<article-id>"
chrome-use eval "document.querySelector('article')?.innerText || document.body.innerText"

The bypass-paywalls extension is already active. Just open and read.

Built-in browser tools (openclaw/opencode) — if your runtime has browser_navigate, browser_snapshot, browser_evaluate or similar built-in browser commands, use those. Navigate to the URL → evaluate document.body.innerText or use snapshot to read.
chrome-devtools-remote MCP — if available as an MCP tool, use it to navigate and evaluate JavaScript on the page.
playwright MCP — headless browser. NOTE: paywalls may NOT be bypassed since the extension isn't loaded. Use as fallback for free sources only.
web_fetch — last resort for URLs that don't require paywall bypass (SEC EDGAR, press releases, free blogs, RSS feeds).

IMPORTANT: Do NOT use web_fetch for FT/WSJ/SA/Bloomberg — it will hit the paywall. Always use a browser tool for paywalled publications.

If NO browser tool is available at all, state clearly: "No browser tool available — cannot read paywalled sources. Gap: [what's missing]." Never hallucinate content you couldn't read.

Fallback sources (no browser needed)

These are always accessible via web_fetch:

SEC EDGAR full-text search: https://efts.sec.gov/LATEST/search-index?q=...
Press releases / IR pages (usually not paywalled)
RSS feeds: FT (ft.com/rss/home), WSJ (feeds.a.dj.com/rss/RSSMarketsMain.xml)
archive.today / web.archive.org (check if article is cached)
Free sources citing paywalled articles (search headline in quotes)

Step 1 — Pre-screen: identify hot sectors (MANDATORY — do not skip)

<step_1_actions> Run the static scanner FIRST. Show its output before proceeding to Step 2. This directs WHERE you read — without it you're guessing which sectors to research.

/Users/engineer/.venv/bin/python3 .agents/skills/trend-stock-research/scripts/emerging_scan.py --top 25

Also check sector ETFs vs SPY (XLK, SMH, XLE, XLV, ITA, XLF, XLU, ARKK, ICLN, TAN, HACK, ROBO) for which are breaking to new highs — this points to the hot neighborhood.

You MUST show the scanner output (or a summary: which themes are EARLY MOVER vs EXTENDED) before moving to Step 2. If the scanner fails to run, state why and use sector ETF comparison as the directional input instead.

This step produces: a list of 3-5 hot sectors/themes to research in Step 2.

IMPORTANT: This step does NOT produce stock picks. Most real winners (NVDA 2021, Ajinomoto, CLF) would NOT have appeared in this scan until it was too late. The scan only tells you where to point your reading — it answers "which neighborhoods are hot RIGHT NOW" so your reading effort is focused, not scattered. </step_1_actions>

Step 2 — Read financial journalism (this is where the edge is)

<step_2_actions> PARALLEL EXECUTION: Spawn one research subagent per (source × theme) combination from Step 1. Do NOT read these sequentially yourself — fan out. Each subagent reads ONE source for ONE theme and returns structured findings. You synthesize after all return.

For each hot sector/theme from Step 1, systematically read these sources. Extract specific facts — do not summarize headlines or speculate about content you haven't read.

PRIMARY SOURCES (highest signal-to-noise):

Seeking Alpha — thesis-driven deep-dives on individual companies.
- Search pattern: site:seekingalpha.com "<sector>" "supply constrained" OR "capacity" OR "bottleneck" OR "monopoly" OR "sole supplier"
- What to look for: articles that explain a DEMAND INFLECTION (not "stock went up"), identify supply-chain bottlenecks, name non-obvious beneficiaries, cite filings/earnings data.
- Quality filter: check author track record. SA articles backed by filing data >> narrative-only.
- Red flags to ignore: articles that are just price-target upgrades, pure technical analysis, or promotional pump pieces with no filing citations.
Wall Street Journal — sector/industry structural shifts.
- Search pattern: site:wsj.com "<industry>" "shortage" OR "backlog" OR "capacity" OR "supply chain" OR "subsidy" OR "tariff"
- What to look for: new industrial policy/subsidies/tariffs that redirect capital, capacity expansion announcements (and who supplies the expansion), M&A activity (signals what insiders think is undervalued), regulatory deadlines creating forced demand.
Financial Times — global view, non-US companies US coverage misses.
- Search pattern: site:ft.com "<theme>" "monopoly" OR "market share" OR "sole supplier" OR "capacity"
- Why FT specifically: it covers Japanese, European, Asian companies that are invisible to US-centric screens. Ajinomoto (Japan), Schaeffler (Germany), Thales (France) — FT covers them; US sources barely mention them.

SUPPORTING SOURCES (verification and detail):

Earnings call transcripts — search: "<company> earnings call transcript Q[1-4] 2026"
- Extract exact phrases: "capacity constrained", "record backlog", "supply agreement", "multi-year contract", "lead times extended", capex step-up numbers.
SEC EDGAR full-text search (free, authoritative): https://efts.sec.gov/LATEST/search-index?q=%22<phrase>%22&forms=10-Q,10-K&startdt=<YYYY-MM-DD>&enddt=<YYYY-MM-DD>
SEC EDGAR Form 4 insider trades — search for insider buying CLUSTERS (3+ officers buying in same week = high signal). Free at: https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&type=4&dateb=&owner=include&count=40 or via OpenInsider: http://openinsider.com/screener?s=<ticker>
Industry/trade press: "<industry> shortage" OR "bottleneck" 2026

FOR EACH PROMISING IDEA, EXTRACT AND RECORD:

The demand inflection: what new use case creates demand supply can't meet?
The supply-chain bottleneck: what scarce input gates the trend?
The catalyst: what specific event (next 1-4 quarters) unlocks value?
Source quality: is this from a filing/earnings call, or a blog post?
Extractable evidence: for EVERY source cited, include at least ONE specific fact you extracted from it (a quote, a number, a date, a named person). "WSJ reported on X" is NOT enough — "WSJ (2026-06-03, 'Transformer Shortage Threatens Data Center Boom'): lead times now 3-5 years, up from 18 months" IS enough. If you cannot name a specific extractable fact from a source, you did not actually read it — drop the citation.
Your confidence level: HIGH (multiple filing-backed sources) / MEDIUM (one good source) / LOW (narrative only) </step_2_actions>

Step 3 — Map to the non-obvious beneficiary

<step_3_actions> For each demand inflection found in Step 2, ask these questions in order:

Who is the OBVIOUS leader? (Name the ticker. It's usually already priced — note it, move on.)
What is the SCARCE INPUT that gates the whole trend? (Material, component, process, fuel, equipment.)
Who CONTROLS that input? Find the company with oligopoly/monopoly share.
- Search: "<bottleneck input> market share", "who makes <component> for <industry>", "<leader> supply chain suppliers"
Does it HIDE in a different sector? The best finds screen as something else entirely.

The pattern: Obvious leader (priced) → scarce input (bottleneck) → who controls it (the find) → does it hide (the edge).

If you cannot identify a non-obvious beneficiary for an inflection, that's fine — not every theme has one. Record it as "obvious plays only" and move on. </step_3_actions>

Step 4 — Skeptic filter (mandatory — most candidates die here)

<step_4_actions> For EVERY candidate, answer ALL THREE questions IN THIS EXACT FORMAT. Drop or downgrade any that fail:

ALREADY PRICED? Apply these hard thresholds:
- Up >150% in 12 months → KILLED. No exceptions. It's late.
- Up >100% in 6 months → KILLED unless catalyst is completely unrealized (hasn't happened yet).
- At 52-week highs with heavy analyst/retail coverage → LATE at minimum, watchlist only.
- Far above 200-day MA (>50% above) → KILLED. (Cheap/ignored + real catalyst = often the better entry. Favor beaten-down names with unrealized catalysts.)
CONCRETE CATALYST + TIMELINE? Name a specific event in the next 1-4 quarters: price hike effective date, capacity coming online, contract award, spinoff, product launch, regulatory deadline. No concrete catalyst → drop. "Eventually the market will realize..." is not a catalyst.
WHAT KILLS IT? State the single biggest risk that would invalidate the thesis. If you cannot name a specific risk, you do not understand the position yet — research more or drop.

MANDATORY FORMAT — show this for EVERY candidate (survivors AND kills):

### <TICKER>
1. Already priced? [YES/NO/BORDERLINE] — [12m return], [6m return], [% vs 200d]. [Verdict].
2. Catalyst? [specific event] — [quarter/date]. [Verdict].
3. Kills it? [specific risk]. [Verdict].
→ KILLED / SURVIVED (confidence: HIGH/MED/LOW)

Do NOT batch-kill candidates with one-liners. Each gets the explicit 3-question treatment even if the answer to Q1 is an obvious kill. This prevents false survivors and forces you to name the risk even on easy kills.

ALSO: If the ticker is ALREADY PUBLICLY ASSOCIATED with the hot theme (e.g., everyone already calls it "an AI stock" or "a power play"), it fails the non-obvious test. The best finds hide in a different sector — food company with a chip substrate monopoly, steel company with a transformer material monopoly, auto supplier with robotics contracts. If it's already in the narrative, it's priced.

Record your skeptic assessment for each candidate. Be honest — the majority should be dropped. </step_4_actions>

Step 5 — Rank, output, and route

<step_5_actions> Rank surviving candidates by: (strength of demand inflection) × (non-obviousness) × (concrete catalyst proximity) minus (how-already-priced)

Produce the output table (format below). Then route top 2-3 finalists to multi-lens-quorum for the buy / wait / late-chase call. This skill only NOMINATES — the quorum DECIDES. Never auto-trade.

IMPORTANT: Do NOT execute the quorum yourself. Your job ends at nomination. State: "Routing [tickers] to multi-lens-quorum with [confidence] flags." Do not say what the quorum would decide, do not apply analyst lenses, do not give buy/wait/pass verdicts. Hand off and stop. </step_5_actions>

<output_format> Produce this table for every candidate that survived the skeptic filter:

Ticker	Demand Inflection	Catalyst + When	Non-obvious Why	Already Priced?	Kills It	Confidence	Source (SA/WSJ/FT/filing)

Then a summary: "Routing [tickers] to multi-lens-quorum for buy/wait/late-chase judgment."

For candidates that FAILED the skeptic filter, produce a brief killed-list: | Ticker | Failed On | Reason | </output_format>

Step 2 (the real work): I search SA for "AI infrastructure" "capacity constrained" and find:

SA article (filing-backed, by author with 85% hit rate): "The Hidden Bottleneck in AI Datacenters" explains that power transformer lead times are now 3-5 years. Cites GE Vernova and Eaton earnings calls: "unprecedented backlog", "capacity constrained through 2028". Demand inflection: AI datacenter buildout requires 3x power infrastructure, and transformers can't be made fast enough.
WSJ: "Transformer Shortage Threatens Data Center Boom" — confirms the bottleneck, names GOES (grain-oriented electrical steel) as the constraint material.
FT: brief mention that GOES production is concentrated in 3 producers globally.

Demand inflection confirmed (HIGH confidence — multiple sources including filings).

Step 3: Obvious leader = GE Vernova (already +200%, priced). Scarce input = GOES (transformer cores). Who controls GOES in the US? Cleveland-Cliffs (CLF) — sole US producer. But CLF screens as "commodity steel company" losing money on flat-rolled. Invisible to any AI/power screen. That's the non-obvious beneficiary.

Step 4 skeptic:

Already priced? NO — CLF at ~$14, near lows. GOES segment buried in money-losing parent.
Catalyst? Weirton GOES plant ramp Q3 2026 + possible segment disclosure/spin (management hinted in Q1 call). Timeline: 1-2 quarters.
Kills it? Flat-rolled steel losses overwhelm GOES upside if steel cycle stays down. No spin officially signaled — could stay buried.

Confidence: MEDIUM (demand inflection is HIGH confidence, but the unlock — spin/disclosure — is uncertain).

Routing CLF to multi-lens-quorum for buy/wait/late-chase judgment.

Step 2: I search FT for "humanoid robot" "supplier" and SA for "robotics" "actuator" "monopoly":

FT: "The Race to Build Humanoid Robots" — mentions that precision bearings and actuators are the bottleneck. Names Schaeffler (SHA0.DE) as having signed binding supply contracts with 2 humanoid OEMs. Robotics is <1% of Schaeffler revenue — invisible.
SA: Article on Schaeffler mostly covers auto weakness (-15% stock YTD). One paragraph mentions "binding humanoid-actuator contracts" from their investor day. No other SA coverage of the robotics angle.
Earnings call (Schaeffler Q4 2025): confirms "multi-year supply agreement for precision actuator systems" but gives no revenue guidance for it.

Demand inflection: humanoid robotics buildout requires precision actuators at scale. Schaeffler has binding contracts but the revenue is negligible today.

Step 3: Obvious = Figure AI, Tesla Optimus (private/already hyped). Scarce input = precision actuators at automotive-grade quality and scale. Who controls it? Schaeffler — binding contracts, bearings/auto heritage, but screens as "struggling German auto supplier." Non-obvious.

Step 4 skeptic:

Already priced? NO — stock down 15% YTD on auto weakness. Robotics not in the price at all.
Catalyst? First volume shipments signaled for H2 2026 per investor day. 1-2 quarters.
Kills it? Robotics could be 5+ years from meaningful revenue. Contracts could be small. Auto downturn could crush the stock further before robotics matters. The "free option" could stay free for years.

Confidence: LOW (thesis is logical but robotics revenue is speculative and timeline is uncertain).

Routing SHA0.DE to multi-lens-quorum with LOW confidence flag — the quorum may reasonably say "too early, watch only."

VERDICT: KILLED. Already priced + no new catalyst + specific downside risk.

| SMCI | Failed: Already Priced | Up 300%, at highs, universally owned, no new catalyst |

<success_criteria> The task is complete when:

You READ actual SA/WSJ/FT content (not just searched — read and extracted specific facts)
Each candidate is tied to a specific demand inflection with named sources
The non-obvious beneficiary mapping was attempted (not every theme has one — that's OK)
EVERY candidate passed through ALL THREE skeptic questions (and most were killed)
Surviving finalists have the output table with confidence levels and source citations
Top finalists are routed to multi-lens-quorum with confidence flags
You did NOT speculate about any company without having read a source about it </success_criteria>

<eval_tracking>

Evaluation tracking (mandatory after every execution)

After every execution of this skill, append a row to TrendPickingEval.csv (in this skill's directory) with the iteration results. This creates an audit trail of how the skill improves.

File: .agents/skills/pick-trend-stocks/TrendPickingEval.csv

Columns:

iteration: sequential number (1, 2, 3...)
commit_id: the git commit SHA of the skill version that was executed
date: YYYY-MM-DD
c1_read_sources through c7_no_speculation: PASS / PARTIAL / FAIL for each criterion
total_pass, total_partial, total_fail: counts
feedback: one-line specific gap description + what to fix next

Score each criterion against the success_criteria above. Be honest — PARTIAL means "attempted but with gaps", FAIL means "did not do this at all or fabricated content".

The skill is considered WORKING when: total_pass >= 6 AND total_fail == 0 for 2 consecutive iterations. Until then, keep iterating (fix gaps → re-run → re-score). </eval_tracking>

<auto_research>

Auto-Research: autonomous self-improvement loop (karpathy/autoresearch applied to this skill)

Inspired by Andrej Karpathy's AutoResearch (https://github.com/karpathy/autoresearch) — "One GPU, one file, one metric." An agent edits ONE file (train.py), runs an experiment under a fixed budget, scores ONE metric (val_bpb), and keeps the change only if the metric improved, iterating ~100 experiments overnight. We run the SAME loop to improve THIS skill instead of training a model:

AutoResearch (Karpathy)	This skill
one editable file: `train.py`	one editable file: `SKILL.md`
one metric: `val_bpb` (lower better)	one metric: RUBRIC mean across eval cases (higher better)
one experiment: train 5 min	one experiment: run the actor on `evals/cases/` + LLM-judge with `evals/RUBRIC.md`
keep change iff `val_bpb` dropped	keep the SKILL.md edit iff mean rose, else revert
budget: wallclock → ~100 runs/night	budget: `--budget` rounds

The greedy keep/discard + checkpoint/revert + audit trail is owned by the harness scripts/auto_research.py (pure-python, zero API cost). The agent does only the two expensive steps each round — RUN and JUDGE — then hands scores to the harness, which decides keep-or-revert.

Trigger

"auto-research this skill", "self-improve the skill", "run the autoresearch loop", "optimize the rubric overnight", "improve until it ships".

One round (the orchestrator executes this)

0. ONCE:  python3 scripts/auto_research.py init --budget 10      # snapshot SKILL.md as baseline
LOOP (until SHIP or budget exhausted):
1. python3 scripts/auto_research.py next-target                  # which RUBRIC dim is weakest?
2. EDIT SKILL.md to fix ONLY that one dimension (smallest change that could move it). One file. One lever.
3. python3 scripts/auto_research.py snapshot N                   # freeze the edited SKILL.md as round-N
4. RUN the actor (the 5-step method above) on each case in evals/cases/train/  (holdout case kept aside)
5. JUDGE each output against evals/RUBRIC.md (0–5 per applicable dimension). LLM-as-judge; be honest.
6. python3 scripts/auto_research.py record N \
       --dims source_grounding=4 non_obvious_discovery=5 skeptic_discipline=5 \
              actionability=4 quorum_routing=5 prescreen_usage=5
   # harness appends to evals/scores.md, then KEEP (new best, promote) or DISCARD (auto-revert SKILL.md)
7. python3 scripts/auto_research.py status                       # stop-condition + rounds left

Keep/discard rule (the whole point)

record compares the round mean to the running best:

mean rose → KEEP: round-N becomes the new best.md; edits compound from here.
mean fell/flat → DISCARD: SKILL.md is auto-reverted to best.md. The bad edit never persists.

This is exactly Karpathy's loop: a change survives ONLY if the metric says it helped. No edit is trusted on narrative — only on the rubric.

Stop condition (from RUBRIC.md, enforced by `status`)

SHIP when train mean ≥ 4.2 and no dimension mean < 3.0. Else loop until budget exhausted, then ship the best variant found. Run the SHIPPED best.md once more on the holdout case to guard against overfitting the train cases.

Why one dimension per round

Same reason Karpathy edits one file and watches one number: attribution. Change six things and a mean move is unattributable. Fix the single weakest dimension, re-score, and you know whether THAT lever worked. next-target always points you at the current weakest dimension.

Overnight / scheduled

Like AutoResearch's "~100 experiments while you sleep", wrap the loop in a scheduler (claude /loop, openclaw cron) with --budget rounds. State is in evals/auto_research_state.json so the loop is resumable across restarts; variants are kept in evals/variants/ for diffing. </auto_research>

<stateful_mode>

Stateful Operation (daily ingest + weekly synthesis)

This skill has TWO operational modes when run on a schedule:

Mode: INGEST (daily — "read and store")

Triggered by: "daily ingest", "read today's news", "ingest articles"

Run emerging_scan.py → identify today's hot themes
Read FT/WSJ/SA headlines via browser (top 5-10 relevant articles)
For each article read, store it in the research DB:

import sys
sys.path.insert(0, '.agents/skills/trend-stock-research/scripts/db')
from research_db import ingest_article

ingest_article(
    url="<article_url>",
    title="<headline>",
    source="ft",          # ft, wsj, sa, edgar, reuters, etc.
    body_text="<extracted text>",
    summary="<your 2-3 sentence summary>",
    themes="ai-power,transformers",    # comma-separated theme tags
    companies="CLF,GEV",               # comma-separated tickers mentioned
    signals="bottleneck,demand_inflection",  # signal types found
    confidence="high",                 # high/medium/low based on source quality
    date_published="2026-06-09"
)

Check for Form 4 insider buying clusters on tracked companies
Log the run: how many articles ingested, which themes

Theme tagging convention (use consistently so convergence detection works):

Use lowercase, hyphenated: ai-power, hbm-memory, humanoid-robotics, goes-steel
Reuse existing tags when the theme matches (don't invent synonyms)
Check existing themes first: python3 scripts/db/research_db.py themes

Mode: SYNTHESIZE (weekly — "what's converging?")

Triggered by: "weekly synthesis", "what's building?", "run picks"

Query the DB for convergence:

from research_db import search_theme_convergence, get_articles_for_theme, get_active_theses

# Find themes with 3+ independent sources over 2+ weeks
converging = search_theme_convergence(min_sources=3, min_weeks=2)

For each converging theme:
- Pull all articles: get_articles_for_theme("ai-power")
- Count independent sources (SA ≠ WSJ ≠ FT ≠ EDGAR = different)
- Check if evidence is ACCELERATING (more mentions this week vs last)
- Check if still non-obvious (not saturated on Reddit/fintwit/CNBC)
Promote theses based on accumulated evidence:
- 1 source, 1 week: monitoring (just filed)
- 3+ sources, 2+ weeks: building (something is forming)
- 5+ sources, 3+ weeks, catalyst identified: actionable (route to quorum)
Apply the skeptic filter (Step 4) to any actionable thesis
Route survivors to multi-lens-quorum with the full evidence trail

Mode: SEARCH (on-demand — "look up what we know")

Triggered by: "what do we know about ?", "search the DB for "

from research_db import search

# BM25 ranked search — finds articles by keyword relevance
results = search("transformer AND shortage AND bottleneck")
results = search("CLF OR cleveland-cliffs")
results = search("humanoid AND actuator")

FTS5 query syntax:

AND / OR — boolean operators
"exact phrase" — phrase match
NOT term — exclusion
term* — prefix match
NEAR(term1 term2, 10) — proximity (within 10 tokens)

DB location and CLI

# Check stats
python3 .agents/skills/trend-stock-research/scripts/db/research_db.py stats

# Search
python3 .agents/skills/trend-stock-research/scripts/db/research_db.py search "transformer shortage"

# List converging themes
python3 .agents/skills/trend-stock-research/scripts/db/research_db.py themes

# List active theses
python3 .agents/skills/trend-stock-research/scripts/db/research_db.py theses

DB file: ~/.local/share/trend-research/articles.db (persists across sessions, zero cost)

Why SQLite + BM25, not vector DB

Zero cost: no embedding API calls, no GPU, no external service
Zero dependencies: sqlite3 is built into Python
Domain vocabulary is consistent: financial journalism uses "bottleneck", "capacity constrained", "supply shortage", "backlog" — BM25 keyword search finds these perfectly
Exact match matters: when you search for "CLF" or "GOES", you want exact hits, not semantic approximations that might return "US Steel" because it's "similar"
Portable: one .db file, copy anywhere
Fast: FTS5 BM25 search over 10,000 articles is <1ms </stateful_mode>