I built an AI research engine.
Then I stopped believing its numbers.
AskSociety puts a business decision in front of a digital collective of psychologically distinct residents who answer independently and blind. Building the engine was only half the job. The harder half was proving what the product was allowed to claim, then removing what did not survive the evidence.
A research product is selling the right to believe the output.
AI can generate a polished research report remarkably quickly. That is exactly what makes the problem difficult. A weak answer can still look considered. A mistyped audience can still produce a coherent report. A model can correctly identify a requirement it cannot honor and then quietly behave as though it honored it anyway.
The failure is not obvious because the output still looks rigorous. AskSociety therefore had a harder requirement than simply producing answers: I needed to establish where the system was reliable, where it was not, and how those boundaries would become part of the product itself.
546 answers.
Not one ended in zero.
Six instrument designs had tried to make 0–100 scores more useful. Instead of building a seventh, I measured the answers themselves.
Real survey respondents tend to pile onto round numbers. This distribution did the opposite because non-round numbers read as more considered. The scale was not merely compressed. The magnitude itself was being generated.
So I stopped trying to improve the score and removed it from every report surface.Change the product instead of defending the output.
The written responses contained far more information than the residents' self-reported numbers. The product shifted toward the prose: themes, objections, quotes, reasons behind the split and qualitative evidence. The 0–100 scores remained available in the transcript for audit, but stopped being presented as a customer-facing truth.
Where continuous placement was still useful, one blind reader scored the written answers against a fixed rubric and pairwise comparisons supplied finer ordering. The larger principle became structural: model output could inform the system, but deterministic checks and external evidence would decide what the system was permitted to say.
The direction was right.
The magnitude was not.
I benchmarked the system against Gallup's published question on attitudes toward AI, asked cold without retrieved context. AskSociety reproduced the broad harm-leaning direction and the large ambivalent middle, but substantially underrepresented the optimistic minority.
That failure mattered more than a flattering internal validation because it exposed the difference between getting the shape of an answer right and being calibrated to the real population.
Broad harm-leaning plurality, large middle, small optimistic tail.
Enough error that magnitude could not be presented as public fact.
The miss replicated across 955 residents.
The limitation went into the customer-facing methodology instead of disappearing into internal notes.
A working research product with checks around the AI, not just AI inside it.
AskSociety became a deployed product and managed service spanning audience construction, research design, study execution, qualitative analysis, client delivery and operator review.
Digital collective
Seventy-two archetype master profiles become psychologically distinct residents with reproducible casting and persistent identities.
Evidence pipeline
Qualitative reads, confidence intervals, data-discovered segment cuts and direction checks designed to catch fluent but unsupported conclusions.
Audience compiler
Plain-English audience descriptions are checked and returned as approved, approved with gaps, or rejected rather than silently broadened.
Managed delivery
A client request surface, operator review layer, explicit publish step, exports and a written read designed around accountable delivery.
I used Claude Code extensively for implementation and multiple models inside the research system. I retained responsibility for product architecture, research design, validation criteria, consequential decisions and what ultimately shipped. The leverage came from AI. The judgment could not be delegated.
The compiler correctly refused a claim.
Then the model quietly put it back.
The first real client study through the compiled audience path exposed a failure that code review had missed. The audience request included experience hiring an outside research firm. The compiler correctly identified that the system could not guarantee that requirement, but the generated resident context still realized the clause as though it were true.
Of 144 residents, 34 narrated having hired an agency. None said they had never done so. The report could have looked completely credible while being built on a false assumption.
Move the check out of model discretion. Two deterministic guards now enforce the boundary.
Build the experiment before building the expensive answer.
The obvious audience-engine plan was to build the full compiler, richer person model and evaluation system, then discover whether the new architecture actually improved anything. I inverted the order and used controlled experiment arms to isolate what was producing the effect before committing to the hardest build.
Classic-shaped prompt
BaselineClassic-shaped prompt
Isolates who was selectedRicher role context
Measures the combined systemA product that became more credible by making fewer claims.
The outcome was not a perfect simulation of public opinion. It was something more useful: a working decision-research system with explicit boundaries, validated strengths, known weaknesses and a mechanism for refusing what it cannot responsibly support.
A real business paid $500 for a decision and received three studies inside a day.
Passed the sealed semantic-fidelity stage with zero silent misses.
The pre-registered business-audience claim survived the test.
The interval crossed zero, so the consumer claim was withdrawn exactly as pre-committed.
Most companies do not need to build something as complex as AskSociety. They do face the same problem.
The hard part of bringing AI into a meaningful workflow is rarely getting it to produce something. The hard part is deciding what deserves to be trusted, where human judgment still matters, and what checks need to exist before the system becomes part of the business.
AskSociety became a case study in exactly that discipline: build fast, measure what the system actually does, narrow the promise when the evidence demands it, and turn the limitations into product behavior instead of hiding them.