Koottam (കൂട്ടം) · Malayalam for "a gathering" · a measured AI experiment
Is a crowd of AI models smarter than its best member?
Four open-weight models, each on a different server, answer the same 616 security questions and vote. Then the crowd's agreed answers train one small model that runs on a laptop. Nothing is assumed. Each claim is scored on held-out questions.
1. The council beats its best member
Weighted council 93.5% vs the best member's 94.3%. It beats three of four members, but not the best.
2. A small model trained on the council beats its untrained self
Gemma 3 1B went from 60.4% to 66.7% on the same 616 questions (p = 0.008). Almost all of the gain is prompt-injection detection.
How it works
Every server speaks the same OpenAI-compatible API, so one async client drives all of them. One member per provider is deliberate: free tiers cap requests per day, per account.
Results: 616 held-out questions
500 questions from the published CyberMetric-500 security benchmark, plus 116 prompt-injection examples (is this text an attack on an AI assistant?). No test question, or near-copy of one, is in training.
| System | Security (500) | Injection (116) | Overall |
|---|---|---|---|
| gpt-oss-120b (Cerebras) | 93.6% | 75.9% | 90.3% |
| Qwen 3.8 27B (Groq) | 94.2% | 81.0% | 91.7% |
| Gemma 4 31B (Google), best member | 96.2% | 86.2% | 94.3% |
| Gemma 4 E2B (laptop) | 84.4% | 81.0% | 83.8% |
| Council, equal votes | 95.2% | 82.8% | 92.9% |
| Council, weighted votes | 95.6% | 84.5% | 93.5% |
| Student, base: Gemma 3 1B | 63.0% | 49.1% | 60.4% |
| Student, fine-tuned on the council | 64.4% | 76.7% | 66.7% |
Members and council scored 2026-09-27 and 2026-09-28, students 2026-10-02. Every system answered all 616 questions.
What the data showed
The right answer was almost always in the room
On 608 of 616 questions at least one member was right. Knowledge wasn't the bottleneck. Choosing whom to believe was.
Correlated mistakes outvote the best member
gpt-oss and Qwen gave the same wrong answer 32 times. On 9 questions the best model was the only one right, and equal votes simply outvoted it. A council needs members that fail differently. Diversity of model family is necessary, not sufficient.
Weighting votes closes part of the gap
Each vote counts log(p(k−1)/(1−p)), the optimal rule for independent voters
(Nitzan–Paroush). The weights were learned from training questions only; tuning them on the test set would only
memorise the answer key. Result: 92.9% → 93.5%, with no tie-breaks needed.
The training key is noisy
On ~7% of training questions, three or more models agreed against the answer key. Spot checks found real key errors. The training filter keeps an example only when the council and the key agree, so the student never learns a disputed answer.
What the student showed
Gemma 3 1B, LoRA-trained for 9.6 minutes on a free T4 with 2,094 council-agreed answers, against the same model untrained. Both are exported and served identically, so the weights are the only difference.
Measured paired, not eyeballed
Both students answered the same 616 questions. Only the flips count: the student got 123 right that the base missed, and missed 84 the base got right. If they were equally good, that split would happen by chance 0.8% of the time (exact McNemar test). The gain is +6.3 points, 95% interval +1.8 to +11.0.
The base was crying wolf
Untrained, it called 113 of 116 texts a prompt injection, so its 49.1% on injection was about what always answering "attack" earns. Trained, it recognises 52 of 56 safe texts: injection rose to 76.7%. Security knowledge barely moved (+1.4 points, p = 0.61). The fine-tune changed behaviour, not knowledge. In detection terms, precision (how many flags were real) went from 50% to 90%, while recall (how many attacks were caught) went from 95% to 62%.
The filter taught it to lean "safe"
The agreement filter kept all 78 safe injection examples but only 37 of 50 attacks, because models disagree more about attacks. Trained on 68% "safe", the student now misses 23 of 60 real attacks: 62% recall. An agreement filter is not neutral about labels. Run 2 rebalanced it, and the lean went away: attacks caught rose from 37 to 51 of 60 (recall 85%, precision 94%) and injection reached 89.7%.
Run 2: balanced lessons fixed it
With as many attack lessons as safe ones (153 each), the student kept recognising safe texts (53 of 56) and now catches 51 of 60 attacks. Injection rose from 76.7% to 89.7% (18 gained, 3 lost, p = 0.0015). Security knowledge stayed flat (64.4% → 63.8%, p = 0.81), which shows the size of run-to-run noise. Against the base, claim 2 still holds: 60.4% → 68.7%, p = 0.0002. Caveats: overall the gain over run 1 is +1.9 points and not significant (p = 0.25); the run changed the mix and the amount of injection data together; and it is one training run.
It inherits the council's blind spots
On security questions all four members got right, the student improved (69.5% → 73.3%). Where two or fewer were right, it got worse (37.1% → 25.7%). Distillation copies the mistakes too.
Engineering
Four free tiers, four sets of limits
Per-model request pacing (fractional req/min) plus an in-flight cap, and retry on 429/5xx with
Retry-After and backoff. Total model spend: $0.
Resumable by design
Every answer is cached, keyed by the model's fingerprint (id + settings). Runs survive rate limits, sleep and shutdown; scoring the council costs no extra calls.
No test leakage
Exact and near-duplicate (≥80% word overlap) removal between train and test. This caught a typo'd copy of a test question hiding in the training pool.
Built to be checked
Python 3.11, pydantic v2, mypy strict, ruff, 27 offline tests. gitleaks runs pre-commit and in CI over the full history.
Roadmap
- Done: data, train/test split with leak checks
- Done: every member scored alone
- Done: council scored, equal and weighted votes
- Done: training set from the council (2,094 examples from 2,500 questions)
- Done: LoRA fine-tune of Gemma 3 1B on a free GPU (9.6 min), with base and student exported and served identically (same GGUF path, same chat template)
- Done: claim 2 scored: 60.4% → 66.7%, supported; the gain is injection detection
- Done: run 2 rebalanced the injection lessons (as many attacks as safe texts): attacks caught 37 → 51 of 60, injection 76.7% → 89.7%, cyber flat. Every pass mark was written down before training and all five were met.
Scope, deliberately: defensive security and prompt-injection detection only. No offensive tooling in the training data.