Hand-picked coding tasks, run locally via pi + llama-swap, each model repeated to expose flukes. Each run is kept separate.
Every model ranked by its mean score across all tasks it was run on. Click a column header to re-sort; use the buttons above to filter local vs cloud.
| # | Model | Type | Avg score | Pass rate | Tasks |
|---|---|---|---|---|---|
| gemma-4-26b-a4b Local | Local | 100% | 24/25 | 5 | |
| gemma-4-31b@or Cloud | Cloud | 99% | 14/15 | 3 | |
| gemma-4-31b Local | Local | 95% | 27/30 | 6 | |
| gemma-4-26b-a4b@or Cloud | Cloud | 92% | 13/15 | 3 | |
| qwen3.6-27b Local | Local | 90% | 24/30 | 6 | |
| qwen3.6-35b-a3b Local | Local | 88% | 24/30 | 6 | |
| ornith-1.0-35b Local | Local | 87% | 25/30 | 6 | |
| qwen3.6-35b-a3b@or Cloud | Cloud | 87% | 13/15 | 3 | |
| qwen3.6-27b@or Cloud | Cloud | 80% | 12/15 | 3 | |
| north-mini-code-1.0 Local | Local | 80% | 20/30 | 6 | |
| bielik-11b-v3 Local | Local | 27% | 5/30 | 6 |
Each cell is the newest run that covered that model × task; click it for that run's code, transcript and verify output.
| Run | Profile | Models | Pass | When (UTC) |
|---|---|---|---|---|
| 2026-07-04T205859Z-session-minimal | minimal | 1 | ungraded | 2026-07-04T20:58:59Z |
| 2026-07-02T101616Z-9f192f9-minimal | minimal | 7 | 74/85 | 2026-07-02T101616Z |
| 2026-06-30T130601Z-3ea440b-minimal | minimal | 4 | 70/80 | 2026-06-30T130601Z |
| 2026-06-30T073559Z-99ec315-minimal | fixed | 2 | 7/9 | 2026-06-30T073559Z-99ec315-minimal |
| 2026-06-29T230457Z-99ec315-minimal | fixed | 4 | 16/16 | 2026-06-29T230457Z-99ec315-minimal |
| 2026-06-29T212443Z-9dd5ff9-minimal | minimal | 1 | 14/15 | 2026-06-29T212443Z |
| 2026-06-29T114508Z-e0d7de7-minimal | minimal | 6 | 25/30 | 2026-06-29T114508Z |
| 2026-06-29T100926Z-fa9f1f9-minimal | minimal | 2 | 6/20 | 2026-06-29T100926Z |
| 2026-06-29T073206Z-8fd0e80-minimal | minimal | 4 | 37/40 | 2026-06-29T073206Z-8fd0e80-minimal |
Lessons that are not tied to a single run — failure modes, config gotchas — with the raw evidence attached.