research_projects

The meta project: the processes that turn a subject into a folder we own — sources, claims, vectors, a graph, and instruments to look at them — so that questions get answered against a record, not against whatever was read last.

The object is a shared and independently evaluatable outside. Shared between Kieran and the model. Evaluatable from another origin, never from nowhere: a fetch is a noticing from a position and the position is recorded; a claim points at bytes and a span; anything derived is reproducible from sources/ plus the code that made it; opinions are frozen and dated before the evidence, so they can be scored.

Ancestors, on this box: bhookd (the pipeline shape: claims not truths, idempotent stages, local-first archive), riverbed/INTENT.md (weight is navigational salience, never probability-true), wink/kernel (immutable artifacts, competing parses as sets, lineage), xresearch (the X connector).

Layout

pipeline/            topic-agnostic stages; topic is ALWAYS a parameter (paths.py)
  src_openalex.py    connector: OpenAlex works + abstracts            (scholarship)
  src_ia.py          connector: Internet Archive OCR texts            (period primary sources)
  src_commons.py     connector: Wikimedia Commons images + metadata   (maps, photographs)
  src_ca.py          connector: Chronicling America newspaper pages   (loc.gov full text)
  src_commons_cat.py connector: walk a whole Commons category (+subcats) of images
  src_locpic.py      connector: Library of Congress Prints & Photographs (loc.gov collections)
  src_wikiimg.py     connector: every image on the Wikipedia pages already in the manifest
  src_wikipedia.py   connector: Wikipedia pages by title (wikitext + html, revision ids)
  src_web.py         connector: web pages by URL (trafilatura extraction)
  src_reddit.py      connector: a subreddit listing through Playwright Chromium (/usr/bin/python3), --images
  src_xmedia.py      connector: photos attached to hydrated X results, with xresearch's OCR
  xresearch (repo)   connector: X search; record the run JSON in the manifest by hand for now
  extract.py         sources/manifest.jsonl -> derived/chunks.jsonl   (span-addressed; per-connector caps)
  embed.py           chunks -> embeddings.npy   nomic-embed-text via ollama, cached by chunk id
  embed_images.py    image chunks -> image_embeddings.npy   nomic-embed-vision (same space); run with
                     pipeline/.venv/bin/python (uv venv, CPU torch) — project/gallery pick it up if present
  project.py         embeddings -> projection.json   PCA + statement-pair axes (recorded verbatim)
  claims.py          passages -> claims.jsonl   a local model proposes statement + verbatim quote + speaker;
                     code checks the quote against the passage (--connectors, --limit); no weights anywhere
  claims_verify.py   re-check quotes tolerant of OCR line breaks and spelling fixes (exact/whitespace/fuzzy)
  embed_claims.py    claim statements -> claim_embeddings.npy so claims rank by what they say
  dates.py           dates are claims: per source, who asserts each date (object / cataloguer / uploader / site /
                     platform / us / model), on what basis, and whether the object's own masthead agrees with the
                     catalogue; the room stacks its histograms by witness and never counts a copy as corroboration
  review.py          the survey: counts, noise, dates, per entry point tagged vs nearest, top sources, claims
  view.py            projection -> view.html    the rotator (colour = role, shape = connector)
  gallery.py         derived -> gallery.html (artifact, a 140-plate sample) or, with --mode local,
                     gallery-local.html (every plate, files under derived/plates/; `bin/serve tartaria` → http://localhost/gallery-local.html on the LAN)
  api.py             the evidence room's API + serves web/dist  (TOPIC=tartaria .venv/bin/uvicorn api:app --port 3790)
web/                 the evidence room (Vite + React): entry points → evidence, search, sources; `npm run build`
topics/<slug>/       one folder per subject — see topics/tartaria/README.md for the contract
  entrypoints.json   the person's entry points, verbatim, with tags (what was fetched under) and a prompt (nearest evidence)

Run

cd pipeline
python3 src_commons.py --topic tartaria --query "Tartary map" --n 12      # any connector, any query
python3 extract.py --topic tartaria && python3 embed.py --topic tartaria \
  && python3 project.py --topic tartaria && python3 view.py --topic tartaria \
  && python3 gallery.py --topic tartaria --rotator <rotator url>

Every stage is a pure function of what is on disk; rerun anything, any time.

The evidence room

http://localhost on the LAN (WSL address; port-proxy from Windows if needed). The control topic (erie-canal) runs the same app on :3792. Entry points on the left in the person's own words; each opens the pictures and passages the record holds on it, when they are from, and which other entry points share evidence. Search asks ollama for the nearest passages and pictures. Every item opens to its source, its fetch position, and its neighbours.

Rules

• No stage may know a topic's name except as a parameter.
• sources/ is append-only. Re-fetching writes a new entry with a new position; nothing is edited.
• A frozen beliefs file is never edited. A later freeze is a new file.
• Prose is the last rendering, never the first. The instrument shows shape.