The pitch for so-called digital twins, or synthetic data, is an appealing one. Instead of recruiting, screening and paying hundreds of real survey respondents, you build AI models of individuals and put your questions to them instead. Once set up, it is obviously much faster and cheaper to deliver than traditional surveys. And interest is high: start-ups in this space have raised millions of dollars to deliver on this promise.

The majority of the focus for digital twins is on market research, but a growing number of companies now also use this for their thought leadership research. Worryingly, it seems that not everyone discloses the use of these digital twins. According to a couple of sources, at least one major global producer of thought leadership is making use of synthetic data in their studies without actually mentioning it in the published results.  

Last year I wrote about a wider issue with synthetic respondents: namely, how do you credibly announce this in your press release? Not disclosing the use of these digital twins wasn’t one of the options I’d considered, but it’s clearly a choice that some are making. Nevertheless, we now have some strong evidence, courtesy of a major new study from Columbia Business School, about the effectiveness and accuracy of synthetic samples (or lack thereof).

Not the results you’re looking for 

The new study from Columbia study (with the encouraging title: "Digital twins are funhouse mirrors"), is likely the most detailed study of digital twins to date. The researchers ran 19 pre-registered studies, covering 164 outcomes, with a representative panel of around 2,000 American adults. Each participant had a digital twin built from their own answers to more than 500 earlier questions, spanning personality, cognitive ability, economic preferences and behavioural experiments. The team then compared what each twin said with what its human counterpart actually said, including on studies the underlying model could not have seen before.

In short, it’s a best-case setup: rich data on every individual, a strong base model and carefully tuned settings. Most commercial synthetic panels have far less to work with.

Unfortunately, the results are not great. The correlation between digital twin and human answers averaged around 0.2. That’s an exceptionally weak correlation. It would be decent odds for a lottery entry, but it’s pretty awful in the realm of credible research. The twins' headline accuracy of roughly 75% looks respectable until you learn that picking answers at random scored about 63%, and that a generic persona built only from demographics (age, income, education and so on) performed almost as well as the full twin. The hundreds of detailed personal data points used to enrich the digital twins and to make them more distinctive and detailed unfortunately did little to nothing to improve their ability to answer similarly to their human twin. 

In addition, the twins' answers were also consistently narrower than those of real people, with less variation in around 94% of outcomes. As one of the lead researchers put it, the twin is "only the approximation of the real thing".

The paper sets out five ways in which twins systematically distort human behaviour: insufficient individuation, stereotyping, representation bias, ideological biases and hyper-rationality. 

The researchers suggest it may be more realistic to think of twins not as clones, but as "hyper-rational, quasi-omniscient versions of humans". That finding echoes a recent HBR piece, which found that synthetic respondents make the rational choice far more often than real people do.

Why this matters for thought leadership

These findings come from a general consumer panel, not more senior B2B audiences. But if anything, the problems are likely to be greater for thought leadership research, for four reasons.

First, credibility. Announcing that your findings come from 500 AI personas in your press release is unlikely to bolster your study’s credibility. You could choose to go the path of not disclosing your use of synthetic data, but that is clearly misleading. 

Second, data scarcity. The Columbia twins were built from a total of 500 data points for each person. Simulating the niche audiences that corporate research often targets (say, treasurers in the retail sector, or supply chain leaders in automotive) means having access to significantly less real-world data to use to enrich your twins. 

Third, distinctiveness. Good thought leadership depends on surfacing the unexpected finding, not confirming the consensus. Under-dispersed responses that cluster around the mean are the opposite of what you need to establish a genuine point of view.

Fourth, bias. The twins have a clear pro-technology bias, among other biases. So a study on AI adoption, for example, run on synthetic respondents that are systematically more enthusiastic about technology, would provide an overly distorted view of reality.

It’s not all bad, though 

At the same time, it’s not right to conclude that digital twins have no use at all. The Columbia researchers ran a field test with a global bank, building twins of around 330,000 customers based on real account information and spending data. Used alongside the bank's existing models to personalise how credit card offers were framed, the data helped lift card sign-ups by 6% and payroll transfers by 25%. There are caveats (the card result was only marginally significant, and some of the lift may reflect greater message variety), but nonetheless it was a decent result. 

The researchers are also candid that part of the problem is simply that human behaviour is hard to predict. It seems that twins work best on narrow tasks, fed with relevant behavioural data. Predicting whether a customer takes up an offer, based on their spending history, is a very different exercise from asking a simulated executive what they think about the future of their industry.

All this said, there is potentially still room for twins within the thought leadership workflow, provided their output is not reported as real findings. For example, the following use cases remain valid: 

1. Pressure-testing a questionnaire before fieldwork, to spot confusing questions or predictable answers.
2. Scenario planning and storytelling, providing it’s clearly labelled as simulation.
3. Exploring the "why" behind real survey results, as a prompt for further analysis rather than as evidence.