instruments / wink / EXP1-nursery/RESULTS.md · raw

EXP1 Nursery: results. A research program Kieran shapes and a model (the "seat") writes down. "I" in these files is the seat unless a line is quoted as his. Snapshot of 2026-10-15 from the private repo; local addresses and paths scrubbed, nothing else changed.

EXP1 — Nursery results

parses evaluated: 72 json failures: 0 anonymized evaluated: 12

Recovery (top-1 across all six ground truths, structure-only WL)

overall: 0.40
scheme=eventseq_v0: 0.31 (wl_vs_truth mean 0.377)
scheme=relgraph_v0: 0.50 (wl_vs_truth mean 0.391)
instrument=mistral: 0.36 (wl_vs_truth mean 0.347)
instrument=qwen: 0.44 (wl_vs_truth mean 0.421)
rendering=events: 0.46 (wl_vs_truth mean 0.446)
rendering=prose: 0.29 (wl_vs_truth mean 0.315)
rendering=pseudocode: 0.46 (wl_vs_truth mean 0.390)

P1 — designed near-miss family (cycle3/feedback3/inversion)

family parses landing top-1 INSIDE the family: 0.36
of those, wrong member (confused sibling): 5/13
P1 verdict: NOT SUPPORTED (structure-only comparison cannot separate the family)

P2 — sequence type vs branching shapes (tree, diamond)

relgraph: top1 0.83, wl_vs_truth 0.347
eventseq: top1 0.00, wl_vs_truth 0.053
destroyed structures per parse (GED): relgraph 3.8, eventseq 2.7
P2 verdict: SUPPORTED

P3 — label destruction (prose, relgraph, both instruments)

prose top-1: 0.33 anonymized top-1: 0.42 drop: -0.08
P3 verdict: SUPPORTED (structure survives noun loss)

P4 — pseudocode vs prose

pseudocode top-1 0.46 vs prose 0.29
P4 verdict: SUPPORTED

Confusion (rows=true shape, cols=top-1; relgraph parses only)

true\top1	chain	cycle3	diamond	tree
chain	5	0	0	1
cycle3	2	3	1	0
diamond	0	0	4	2
feedback3	2	0	4	0
inversion	1	4	0	1
tree	0	0	0	6

lineage spot-check: 10/10 parses walk back to artifacts

back to wink