Research note 01
Why models are average, and where greatness lives.
Generative models have become very good at being correct. This note sets out why the same training makes them average, why taste is a skill that can be measured, and how expert judgment becomes a signal that models and agents can learn from.
Abstract
Next-token prediction and preference tuning both pull a model toward the centre of its training distribution. For arithmetic the centre is the right answer. For design, writing and film the centre is mediocre, because distinctive work is by definition improbable. Code and mathematics escaped this through verifiable rewards; subjective domains have no verifier. We argue that taste is pattern recognition, that it can be decomposed and measured, and that the missing piece is a record of expert judgment: what was rejected, what was kept, and why. We describe a three-tier cascade that routes each judgment to the cheapest signal able to carry it.
§ 1
Models are brilliant statistical machines. Which is exactly why they’re average.
01 / Pretraining
Likelihood ≠ quality.
Next-token prediction learns a probability-weighted blend of everything. For 2+2 the average is optimal. For design, the average is mediocre, and distinctive work is by definition low-probability.
02 / Preference tuning
Aggregated preferences erase the tails.
Reward models average annotators of different tastes and learn what nobody dislikes, not what anyone loves. RLHF-tuned models produce measurably less diverse outputs than their base models.
03 / Rewards
No verifier, no flywheel.
Math and code got good on verifiable rewards. There is no function that returns 1 for good design and 0 for bad. So subjective domains never got to ride that flywheel.
The training objective rewards the likely. Creativity, that feeling of something “hitting different”, comes from doing improbable things. A model that is rewarded for the middle will not find the edges on its own, however large it gets [1] [2].
§ 2
Slop isn’t new. Infinite slop is.
Mediocre work has always existed. What changed is the cost of producing it, which is now close to nothing. One prompt returns a dashboard, a short story, a landing page, a game or a slide. Your timeline is full of it, and it all looks the same.
We have a short window to reject the monotony before it becomes the sad default.
§ 3
Taste is a skill. It’s pattern recognition.
Taste is the bar of quality in the absence of correctness.
Exposure
Seeing many things.
Observation
Noticing patterns.
Point of view
Deciding what you like.
Down-selection
Deciding what you don’t.
If it’s pattern recognition, it can be decomposed. If it can be decomposed, it can be measured. If it can be measured, it can be improved.
We’re making the unverifiable verifiable, starting with generative media.
§ 4
What we’re betting on
- 1
Millions, then billions, of people will create. Most won’t be experts or have the patience to develop taste. We have to make it easy for them to create great things.
- 2
Agents will create on behalf of people. They’ll need the right tools for context, inspiration and judgment.
- 3
Creatives and tastemakers will matter more than ever. The key is propagating their taste beyond themselves, and letting them share in the upside.
The cure for AI slop isn’t better generation. It’s better judgment.
§ 5
Models learn from the vote. Quality lives in the reasons.
Most preference data is a click. Two options are shown, one is picked, and the record keeps the winner. It is cheap, it scales, and it carries no criteria and no reasoning. Average enough of those clicks and you learn what nobody dislikes.
{ "winner": "D" }
no criteria · no reasoning
An expert’s record is a different object. It holds the option that was kept, the ones that were killed, and the reason for each, alongside rubric scores, the revision path, and the brief it was judged against. The reason is what a model can learn from.
§ 6
Turning judgment into signal
We turn something like design into the shape of a problem like code: measurable, decomposable, visualisable. Every judgment routes to the cheapest signal that can carry it.
Verifiable checks
Programmatic, deterministic. Cheap at scale.
↓ what code can’t judge
Reference rewards & learned models
Trained on tastemaker data. Judges against a reference.
↓ what models can’t judge yet
Human tastemaker judges
The tails: where experts genuinely disagree.
- Text critiques linked to the exact code component
- Individual preference vectors preserved, not averaged into consensus
- Quality over quantity in every dataset
Every T3 judgment becomes tomorrow’s T2 model.
One platform: capture judgment, encode it, serve it
Where tastemakers design, grade model outputs and review each other’s work.
Whose judgment predicts what works, per domain, on one underlying representation.
The same judgment, delivered to labs for training and to agents at inference.
§ 7
Two verdicts, never one.
Every piece of AI creative work faces two questions. We keep them separate. When they disagree, that disagreement is the signal.
The audience
Can you tell it’s AI? Will it perform?
Passing as human is the floor.
The expert
Would you ship it?
Being worth shipping is the bar.
§ 8
Annotation vendors sell labels. We operationalise judgment.
Generalist data vendors optimise for scale and agreement, which is exactly what produces the average. Brand tools extract assets but can’t judge. Canopy sits where expert judgment meets domain depth, at training and at inference.
§ 9
Mission as a moat
The best tastemakers don’t join a labelling vendor. They join a mission. That network, and the data representation we build on top of it, compounds.
Judgment is non-stationary. A style loses value as it becomes common. It’s a never-ending problem. That’s why it’s a great one.
Two engines. One taste graph.
Programs with model and agent labs fund the research. The API for agents and apps compounds it. Both draw on the same record of judgment.
Judgment is the new bottleneck.
§ 10
Starting with design. Not stopping there.
Generative media first. Then every subjective domain.
Now
Canvas and Brand API
The creative canvas is live through MCP, the platform and the API. The Brand API is the first tool we ship for agents.
Creativity canvas + API
Brand API
Next
Judges inside the loop
Online judges an agent can call while it works, and lab data and evals for vision and design.
Online judges for agent loops
Lab data and evals
Then
All of generative media
Once the floor is solved, taste becomes a personalisation problem.
Studio · Art · Video
3D · Spatial
§ 11
References
- Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., Raileanu, R. (2024). Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR 2024.
- Padmakumar, V., He, H. (2024). Does Writing with Language Models Reduce Content Diversity? ICLR 2024.