Kin Retrieval Proof
Citable within scopePre-registered symmetric protocol v2, n=26
Kin ties a lexical grep/find/ls baseline on file localization, leads on symbol, span, and line localization, uses the same tokens, and is bit-reproducible in this measured run. The lexical baseline has the higher file-F1 point estimate; the pre-registered significance rule still returns a tie.
File localization
TIE
Pooled micro-F1 is 0.4148 for Kin vs 0.5325 for lexical. The separate paired decision uses mean per-task F1: 0.5257 vs 0.5963, delta -0.0706, 95% CI [-0.1595, +0.0090], McNemar p = 0.125.
Structural localization
.318 vs .289
Kin leads on symbol F1. It also leads on span F1 (.326 vs .297) and line F1 (.307 vs .266).
Cost
1.00x tokens
Token use is at parity, not a savings claim. Kin used 0.95x tool calls in this run.
Determinism
26 / 26
Kin was bit-identical across all tasks and passes. The lexical baseline was bit-identical on 3 of 26 tasks.
Pooled micro-F1 and mean per-task F1 are different aggregations. The pooled values describe magnitude; the paired per-task values govern the pre-registered statistical verdict.
Method
- Suite: 26 Multi-SWE-Bench Go tasks from the gh CLI repository.
- Symmetry: same model, seed, temperature, context limit, agent loop, and tool-call cap. Only the retrieval surface changed.
- Runtime: qwen3.6-35b-a3b at q8_0, temperature 0, seed 0, context 32768, max 24 tool calls, parallelism 1.
- Decision rule: a task-paired 10,000-sample bootstrap over per-task file F1 and an exact McNemar test over file hits. A win requires the confidence interval to exclude zero and p < 0.05.
- Kin gate: the Kin arm had to be bit-identical across three complete passes or the result was not citable.
What this does not prove
- It is not an end-to-end patch-passes-tests or live-agent success result.
- It does not prove lower token cost; the measured result is token parity.
- It does not establish superiority on the file axis; grep has the higher point estimate and the statistical verdict is a tie.
- It does not generalize beyond this suite, language, model, or runtime without another governed run.
Provenance and Artifacts
The run pins Kin at 2508da69, kin-bench at2c9875a, and scorer version3.0.0-gold-denominator. The compact governing verdict and paired decision are published below with SHA-256 hashes.
Disclosure: the full benchmark harness repository and raw run directory are not public today. This page publishes the governing compact artifacts and their exact hashes, but does not claim full third-party reproducibility yet. The complete evidence bundle is available for investor diligence while the public harness boundary is prepared.