What an LLM council is — and one you can watch run.
An LLM council is a setup where one question goes to several large language models at once, each answers independently, the models critique each other’s answers, and a designated model synthesizes the final response.
The word “council” is a metaphor, and a good one: a room of experts who each give an opinion, argue, and hand you a considered verdict instead of a single off-the-cuff take. What follows is how the pattern works, what it does and doesn’t fix, and — further down — the measured verdicts from real councils run on this site, which is the part an explainer can’t give you.
How an LLM council works
Nearly every implementation — Karpathy’s repo, the hosted products, the workflow-builder templates — runs the same three stages. Only the vocabulary differs.
Independent answers
also called: fan-out, first opinions, round zero
The question goes to every member of the council at the same time, and each one answers without seeing the others. This is the load-bearing constraint, not an implementation detail: if the models can read each other first they anchor, and the agreement you measure afterwards is an artifact of the ordering rather than a fact about the question. Independent draws are what make convergence mean anything.
Blind peer review
also called: critique round, deliberation, ranking
Each model is shown the other answers with the names stripped out — Response A, Response B — and asked to evaluate them for accuracy and insight. The anonymization is what stops a model flattering its own output or deferring to a sibling from the same lab. This is the step that turns a pile of parallel answers into something closer to a deliberation, and it is where you occasionally watch a model change its mind under pressure.
Synthesis
also called: chairman, judge, arbiter
One designated model reads every answer and every critique, then writes the response you actually read. Note that this is not a vote. A vote returns the median opinion; a synthesis returns a considered one, and it is free to blend three answers, follow the minority, or overrule the whole room. The distinction matters most exactly when the council splits — which is the case the pattern exists for.
On Polymind the council members are panelists, the review stage is a critique round you choose the depth of, and the synthesizer is the judge. The glossary maps the rest of the vocabulary between implementations.
What a council actually returns
These are councils run on this site, on questions somebody actually asked. The verdict on each line is what Polymind measured: how much the panelists’ final answers converged, or — on runs where the judge counted how many panelists it actually agreed with — that count. Open one and you get every panelist’s answer side by side, the dissenting model named, and the synthesis that came out of it.
- Partial overlap7 models
strongest character out of all comics
- Answers diverge7 models
what is the proven best way to get better SEO/GEO
- 6 models
iphone vs google pixel, who takes it? for tech lover and heavily using phone user.
Browse every published council run → Or see which models the judges pick most often across all of them, on the leaderboard (methodology).
Why a council beats one chatbot — and when it doesn’t
The honest case is not that a council is fancier. It’s that a single chatbot hides three things you would want to know, and a council surfaces all three.
The first is a confidence signal. One model answers in one fluent register whether it is certain or guessing. A council gives you convergence or scatter, which is a real read on how settled the question is. The second is blind spots: every model is the product of a specific training mix and a specific company’s idea of helpful, and those choices leak into answers. Ask several and the idiosyncrasies cancel instead of compounding. The third, and the most valuable, is the disagreement itself. Two frontier models answering the same question differently is a flashing light over a soft spot — exactly where you would want to slow down before acting.
This is also why a council is a genuine hallucination check. A model inventing a citation does it with full confidence and no tell. Two independent models producing the same fabrication is rare; one inventing while the others decline is common. The fabrication doesn’t stop happening — it stops being invisible.
A council is not free, and it is the wrong tool for a reformat, a throwaway draft, or a question you could answer yourself. One model is faster and cheaper and that is the right call. It earns its keep on questions where being wrong is expensive: a contract clause, a library choice you will live with for two years, a factual claim you are about to publish.
Karpathy’s llm-council, and the hosted version
The term spread in late 2025 when Andrej Karpathy published a small local web app called llm-council. You clone it, install the Python and JavaScript toolchains, supply an OpenRouter key, and run it on your own machine; the default council is four models and you change the lineup by editing a config file. Karpathy is direct about what it is — in his own words it is “99% vibe coded” and provided as-is for other people’s inspiration. That is the right framing for a weekend project meant to spread an idea, and it worked: the idea spread.
Whether you want the repo or a hosted council comes down to what you are optimising for. The repo wins the first two rows outright.
| What you might want | Run the repo | Hosted (Polymind) |
|---|---|---|
| Prompts stay on your hardware | Yes — nothing leaves your machine except the model calls you configure | No — questions are processed server-side |
| Change the prompts, add local models | Yes, it is the point of the repo | No — the pipeline is fixed |
| Setup before the first answer | Two toolchains, an OpenRouter key you fund, a config file, two local servers | Type a question |
| Runs follow you across devices | No — conversations are JSON files on one machine | Yes, saved to your account |
| A link you can send someone | No | Yes, per run |
| Score kept across runs | No — every run is its own island | Yes — every completed run feeds a public leaderboard |
That is the whole trade: the repo gives you control and asks for setup; a hosted council gives you setup-free convenience and a shared scoreboard, and asks you to trust a service with the keys. If your actual goal is to hack on the council, use the repo.
Common questions
- What is an LLM council?
- An LLM council is a setup where one question goes to several large language models at once, each answers independently, the models critique each other’s answers, and a designated model synthesizes the final response. The premise is that models trained on different data by different labs have different blind spots, so mistakes one would state with total confidence are caught by another — and where they all agree independently, that agreement is itself evidence.
- Who came up with the term LLM council?
- The pattern predates the name, but the name spread in late 2025 when Andrej Karpathy published a small open-source project called llm-council, which crossed nineteen thousand GitHub stars. Perplexity shipped a feature called Model Council in early 2026. You will also see the same shape called an AI council, a model council, an AI advisory board, or simply multi-model AI.
- Is an LLM council better than asking ChatGPT once?
- Not automatically, and not for everything. A single model is faster and cheaper, and for a reformat or a throwaway draft that is the right call. What a council adds is a confidence signal: one chatbot answers in the same fluent register whether it is certain or guessing, while a council tells you whether independent models converged. That signal is worth its cost on questions where being wrong is expensive — a contract clause, a library you will live with for two years, a factual claim you are about to publish.
- How many models should sit on a council?
- Enough that agreement is not a coin flip. At two models an agreement ratio carries almost no information; three is the practical floor, and most implementations run three to seven. Beyond seven the marginal model mostly adds cost and latency rather than a genuinely new blind spot, because the frontier labs’ training mixes overlap more than the branding suggests.
- Does a council stop AI hallucinations?
- It does not stop them happening; it stops them being invisible. A model inventing a citation or an API that does not exist does so with full confidence and no tell. Two independent models producing the same fabrication is rare; one inventing while the others decline is common. The council converts a silent error into a visible disagreement, which is the difference between acting on it and checking it.
- Can I run an LLM council myself?
- Yes. Karpathy’s llm-council repo runs locally: clone it, install the Python and JavaScript toolchains, supply an OpenRouter key, edit a config file to pick the lineup, and keep two local servers running. That is the right tool if your goal is privacy, local models, or hacking on the prompts. A hosted council is the shorter path if your goal is to use the pattern rather than study it.
Run one on your own hardest question
Ask your usual chatbot, then run the same prompt past a council and watch where the panel agrees, where it splits, and what the judge does with the disagreement. Nothing to install and no API key to manage — you can try it without an account.
Run a council