Why language models flatten a population, and what fixes it

A few years ago, one of our co-founders' wife couldn't settle on a colour for their hallway. So he fed a language model everything he knew about her and asked it to pick.

It said sage green. She was sure she wanted blue. They painted it blue. Five days later she hated it, and the hall has been sage green ever since.

That is a small story with a large idea inside it. Given enough information about one person, a model could predict what that person would actually want. The obvious next question is whether the same thing works for a whole population, and that question is why Cambium AI exists.

Michael Birdsall set out what we have learned since, in a webinar with the Association of Public Data Users. The full recording is below. This is the written version.

The obvious approach does not work

The natural first attempt is to take public census microdata, tell a language model which person it is, and ask it to answer a survey. We tried it. It fails, and it fails in a specific way.

Language models are very good at being an average human. They are poor at everything else. We measured this by giving personality tests to models. The real difference between men and women on agreeableness is about 0.6 standard deviations. Tell a model it is a woman and the difference it produces is two. It is not playing a person. It is playing a stereotype, and it exaggerates by more than three times.

Survey data makes this worse rather than better. Any survey pulls towards the middle, which means the people at the edges of a population are already thinned out before a model touches them. Those are usually the people you needed to hear from. They are the ones who never come to the meeting.

What actually helps is more data, not a better model

The fix turned out to be the same thing that made the hallway work. A model told only that it is a woman carries a bias about women. That same model carries a bias about someone who studied law, and another about someone who runs marathons. Layer enough real detail on, and those biases start cancelling each other out. One attribute buys a stereotype. Enough of them start behaving like someone real.

Doing that at population scale is harder than it sounds, because the combinations have to stay honest. If the relationships between attributes drift, you end up with a character built on a character built on a character, describing someone who does not exist anywhere. Most methods start to struggle once you are holding six or seven of those relationships at once. We are currently holding 96, across more than 200 public datasets, and any data about people can be joined onto a synthetic individual as long as those relationships hold.

Two things worth knowing if you are trying this yourself

First, synthetic personas converge. Let them run long enough and they drift towards the same shared opinion, and the disagreement that made them a population disappears. We do not fully understand why yet. Our working approach is to build them into households and communities first, so there is some tension between them rather than a single conversation with no friction.

Second, and less expected: the newer models are worse at this. GPT-3.5 and 4.1 produced a wider range of answers than GPT-5.4 and 5.5. Our suspicion is that the better models get at coding, the more they behave as though every question has one right answer. That instinct is useful in software and actively harmful when the task is to represent a range of real people.

None of this counts without validation

This technology is new and it needs checking. For us, validity means predicting something that has not happened yet and then testing that prediction against real data when it arrives. Our population is trained back to 2007, so when new public data lands we can compare what we said would happen against what did. Elections are useful for the same reason: there is a held-out answer nobody has seen yet.

Longer term this should not be self-assessed. Language models have independent groups that set the benchmarks, and synthetic populations will need the same.

Watch the session

The full 45 minutes, including a long Q&A with Amy O'Hara, is on APDU's YouTube channel: Perspectives on AI: Cambium AI's Simulated Population.

If you would like to see how the method works in more detail, that is written up on our methodology page.

If you have a dataset you would like joined onto public data, tell us about it. And if validation or privacy frameworks for synthetic populations are your area, we would genuinely like to hear from you.