Lots of platforms quote a single headline accuracy figure: 91%. 95%. "Within three points." But if you ask what that number is a percentage of, the conversation sometimes has a way of mysteriously ending.
A mean can be close while the shape of the answers is wrong, and the shape is where business decisions are made. One fix is a statistic almost nobody quotes: Jensen-Shannon divergence. It compares the entire distribution of synthetic answers to the entire distribution of human answers on the same question, option by option. Zero means identical shapes. By our rule of thumb, under about 0.05 means the shapes are close. (And we often get a lot closer than that!) It is cheap to compute and brutal on outputs that collapse to the middle, which is why you rarely see it in a sales deck.
1. What is underneath? Which human respondents, from which studies, on which population, how recent? Which forms of data (survey, transcript, behavioral, transaction, and so on)? Ingested in raw form or only in summary deck form? If the answer is "the model plus a persona description," you are buying the internet's opinion of your customer. The Columbia mega-study found a persona given only demographics scored the same on accuracy as a twin built from 500 of the person's own answers. [1] Descriptions and tethered digital twins built from individual respondents -mostly disappointing: Columbia's correlated with their humans at about 0.20. [1] (Individual humans are just too complex to model; modeling emergent interactions between groups of synthetic individuals is even harder!) In our parallel tests, respondent data clustered–in real-time–into clusters holds up much better. [4] There’s a reason organizations typically seek to understand people at the segment level in order to engage them and persuade them to act or think differently.
2. Is the validation automated, or was it conducted manually once? Validate against each dataset you generate with a platform. Standardized validation means every synthetic study passes through the same tests before release and the client sees the results. Consistent improvement becomes possible.
3. Do you test the distribution, or only the mean? Jensen-Shannon (or similar) divergence per question, or an equivalent. Then the spread: is the synthetic standard deviation as wide as the human one? Then the tails: do minority answers survive, or does everything regress to the modal consumer? If so, your outputs will be category average and hardly unique to your brand or business challenges.
4. Does the structure hold? Real respondents' answers hang together. Someone who says price is the barrier does not (usually!) also say they never look at price. Ask for a correlation-structure test across question pairs (we use Cramér's V) and a "top-two box" match rate on ranked items: does the synthetic sample pick the same winning concept, claim or package the humans did? This decides whether the data can be productively cross-tabbed, which is necessary to shape strategy, innovation, or messaging for particular customer segments.
5. What is the denominator? Humans disagree with themselves. In Columbia's own panel, people re-asked the same questions matched their earlier answers 81.72% of the time. [5] A synthetic accuracy figure without the human test-retest figure beside it has no ceiling and no meaning. Ask for both, and ask how two human samples differ on your own tracker before you judge a synthetic gap of five points.
6. Did it predict, or remember? If the model saw the answers it is being graded against, the grade is worthless. Ask for the exact-match check and the privacy test on the same run. A vendor who cannot tell you the exact-match score cannot tell you whether the output was predicted or recalled.
7. Can you show me the misses, show me where the humans come in, and show any improvement over time? Every validated study has questions it got wrong. Ours get labeled: "lead" where the synthetic data can carry a decision, "support" where it belongs beside other evidence, "explore" where it is for ideation only. Then ask how the human study launches, whether it uses the same questionnaire on the same platform, and whether the human result feeds the next synthetic run. Finally ask about other ways synthetic data outputs can improve over time: with additional client data ingestion, with improved weighting, with improved targeting, and so on.
Columbia Business School's study of digital twins found the twins' answers had less spread than the humans' on 154 of 164 outcomes while often landing near the right average. [1] Pew Research Center's test of AI survey-takers found the model turning attitudes held by a plurality into near-unanimity: 97% of Hispanic adults following the World Cup, against an actual 43%. [2] A preprint this month found that personas built from descriptions of people recovered only 53 to 67% of the real variation between them. [3] None of those failures show up in a mean. [4] The number matters less than the process. If a vendor cannot show you a distribution comparison, they haven't looked.
We show clients automated comparisons per question in real-time, available within a simple to use dashboard, each coded green (use with confidence), yellow (directional only), or red. If the answer is yellow or red, a human study with AI probing can be launched at the touch of a button to gather more foundational data. Or we can always ingest more client or syndicated data to fill gaps.
Here’s an example:
Read this before making a digital twin, synthetic data, or simulation decision. Why a single headline accuracy figure hides what matters, the statistic to ask for instead, and seven questions every buyer should put to a vendor.
Three of us from Panoplai made ESOMAR's 2026 Insight250. Adam Bai on what that says about an industry in flux, why human data is about to get more valuable, and why we build the way we build.
Panoplai's Chief Strategy & Client Officer, Adam Bai, joined Karen Lynch on Episode 173 of the Greenbook Podcast — "Shaping the Future of Insights." It's a wide-ranging conversation about where the insights industry is headed, why Panoplai built what it built, and what it actually means to be "too early" in this space. Here are the highlights.
%20(200%20x%20108%20px).png)
