BigCat Research

AI-assisted research: what it really speeds up

Short answer: AI-assisted research is fastest at the parts of a project that are about handling text: drafting, summarising, scanning sources and sorting open-ended answers into categories. Controlled studies show large time savings on that kind of work. It is weakest where a project needs facts that must be true, sources that must exist, or the opinions of real people. Those steps still need a researcher, and published tests of synthetic respondents (survey answers generated by a language model) do not yet support using them in place of a real sample.

Disclosure: BigCat Research uses an AI working layer of its own (BigAgent) and sells desk research, market research and consumer insight research. We have tried to separate what the evidence shows from what vendors, including us, would like to claim. Every factual claim below links to a source we opened; the plant-based milk example is illustrative.

What does the evidence say AI speeds up?

The best-known controlled test is an experiment by MIT economists Shakked Noy and Whitney Zhang, published in Science in 2023. College-educated professionals in marketing, consulting, data analysis, HR and similar jobs were given writing tasks typical of their work. Those with access to ChatGPT finished 40 percent faster, and independent evaluators rated their output 18 percent higher in quality.1 The working-paper version adds a detail that matters for research teams: the tool mostly substituted for effort, shifting work towards idea generation and editing and away from rough drafting.2

Two caveats came with that result. The tasks did not require precise factual accuracy or knowledge of a specific company's context, and the researchers did not hire fact-checkers to test accuracy. Noy said the speed gains would likely be smaller in real work because people need time to check facts.1

The second strong finding is about classifying text. A 2023 study in PNAS gave ChatGPT 6,183 tweets and news articles to label for relevance, stance, topic and framing. Working zero-shot, without examples, it was on average about 25 percentage points more accurate than crowd workers on Amazon Mechanical Turk, and each label cost under $0.003, roughly thirty times less.3 For a research team, this is close to coding open-ended survey answers or sorting customer reviews into themes.

That study has limits too. Accuracy was measured against labels made by trained annotators (political science students), so a human standard still sat underneath. It tested one model (gpt-3.5-turbo), and the authors list performance across multiple languages as an open question.3 Anyone coding Turkish verbatims should treat that as a reason to check a sample by hand.

Which parts of a research project does AI speed up, and which still need people?

Research stepWhat AI speeds upWhat the evidence warnsWhat still needs a person
Drafting briefs, guides, report textFirst drafts, rewording, structure; 40% less time on professional writing tasks in one experiment1Tasks in that test did not require factual accuracy; real gains are likely smaller once checking is included1Deciding what the report argues; checking every number
Coding open-ended answers and reviewsLabelling at scale; above crowd-worker accuracy in one study3Benchmarked against trained human coders; one model; multilingual performance open3Building the code frame; hand-checking a sample
Desk research and literature scansFinding and summarising material quicklyFabricated references: 55% of GPT-3.5 and 18% of GPT-4 citations in one test did not exist4Opening each source; judging date, purpose and reliability
Early hypotheses about consumersRough signals on familiar products, e.g. willingness-to-pay estimates5Results move with prompt wording; less variation than real people6Testing the hypotheses with real respondents
Replacing survey respondentsNot supported for statistical inference in a direct test against a national election survey6Answers changed over a three-month period with the same prompt6A real sample, recruited and weighted

Where does AI-assisted research go wrong?

The most practical risk for desk research is invented sources. William Walters and Esther Wilder asked ChatGPT-3.5 and ChatGPT-4 to write short literature reviews on 42 topics and checked all 636 references in the 84 resulting papers. Of the GPT-3.5 citations, 55% were fabricated; of the GPT-4 citations, 18% were. Among the references that did exist, 43% of GPT-3.5's and 24% of GPT-4's contained substantive errors.4 The study tested 2023 models, but it shows why "the model cited it" is not the same as "the source says it".

Türkiye's data protection authority makes the same point in its November 2025 guide on generative AI. It describes hallucinations as outputs that do not match reality yet often look plausible and convincing, gives the example of a lawyer meeting court decisions that do not exist, and says the accuracy of outputs needs to be checked.7 Where outputs contain personal data, the guide calls for regular monitoring and human oversight mechanisms where needed.7

A plain example: you ask a chatbot for the market size of plant-based milk in Türkiye. It returns a figure, a growth rate and a report title. All three can be wrong together, and the report may not exist. The only fix is to open the source and find the number on the page.

Can AI replace survey respondents?

This is where the most money is at stake, and where the evidence is most mixed.

The case for: a 2023 study in Political Analysis conditioned GPT-3 on thousands of demographic backstories from real US survey participants and compared the resulting "silicon samples" with the human ones. The authors argue that a model with sufficient "algorithmic fidelity" can reproduce the patterns of attitudes found in human subgroups and could be a useful research tool.8 In marketing, a Harvard Business School working paper by James Brand, Ayelet Israeli and Donald Ngwe prompted GPT-3 to act as a shopper. Its willingness-to-pay estimate for fluoride in toothpaste was $3.40, against $3.27 from a human consumer study.5

The case against: a 2024 test in the same journal assigned ChatGPT personas based on American National Election Study respondents and asked for feeling-thermometer ratings of 11 groups. Average scores tracked the survey closely, but the synthetic answers varied less than real ones, regression results often differed significantly from the real data, small prompt changes shifted the answers, and the same prompt gave different results over three months. The authors conclude that this kind of sampling "is not reliable for statistical inference".6 The HBS study's own limits point the same way: the authors say better prompt phrasing could improve results, some answers departed from economic theory (for wine and yogurt), and the authors presented the method as a way to screen product attributes before studies with people, not a replacement for talking to consumers.5

Our reading: synthetic answers can help you decide what to ask real people. They are weak evidence for what real people think, especially for a new product, a niche segment or a market the model has seen little text about. The ICC/ESOMAR International Code draws the same line in its definitions: an "individual/person" means a human being, as distinct from a synthetic or digitally created persona.9

What should you ask an agency about AI use?

Professional rules now expect disclosure. The 2025 ICC/ESOMAR International Code says the client must be told when AI or other emerging technologies are used in compiling datasets, analysis, reporting or interpretation, including synthetic data and synthetic personas, and that the extent of human oversight must be stated.9 Research data used with AI must stay confidential, with access limited to a secure and controlled environment.9 When findings are published, researchers and clients must disclose whether AI or synthetic data played a significant role, and how much human oversight was involved.9

In Türkiye there is a data transfer question as well. The KVKK guide says that when a data controller in Türkiye uses a generative AI service run by a provider abroad and personal data is transferred, the transfer must follow Article 9 of Law No. 6698 and the related regulation.7 Transcripts and open-ended answers can contain names, places and health details, so treat this as more than a technicality.

Questions worth putting in a brief:

  1. At which named steps of my project will AI be used: drafting, coding, desk research, analysis?
  2. Who checks the output, and how? For coding, is there a hand-coded sample to compare against?
  3. Are any figures in the report taken from AI output without a source a person has opened?
  4. Will any synthetic respondents or personas be used, and will their results be reported separately from real respondents?
  5. Which tools process my data, where are they hosted, and can my data be used to train an external model?

Our guide to briefing a research agency has a fuller question table and a scorecard you can adapt.

How we use it at BigCat, and where we stop

BigAgent is the working layer we use for desk market research, source scanning, competitor and category monitoring, open-ended text analysis, qualitative synthesis and report preparation; method choice, source reliability, interpretation and recommendations stay with BigCat researchers.10 On the BigAgent page we describe its role as clarifying where fieldwork should focus and which questions need to be tested, and BigAgent does not replace the researcher.10

If you are deciding whether desk work is enough or you need fieldwork, see our BIG 101 notes on when desk research is enough and whether AI replaces or strengthens research. For projects built on what real customers think, the guide to commissioning consumer insight research is the next step. More method guides are in our guides, and the BigAgent page describes the workflow.

FAQ

1. What is AI-assisted research? It is research in which a person designs and checks the work while language models handle text-heavy steps such as drafting, summarising, scanning sources and coding open-ended answers. Controlled studies show large time savings on writing tasks and strong performance on labelling text, with human checking still underneath.13

2. Can AI replace a survey? Not on current evidence. A 2024 Political Analysis study found that ChatGPT-generated survey answers varied less than real ones, shifted with prompt wording and changed over three months, and concluded they were not reliable for statistical inference.6 Synthetic answers can help shape hypotheses, which should then be tested with real respondents.5

3. Is AI good for desk research? It is fast at finding and summarising material, but it can invent sources. In one test, 55% of GPT-3.5 and 18% of GPT-4 citations were fabricated.4 Every source in a desk research report should be opened and read by a person.

4. Must an agency tell me if it uses AI? Under the 2025 ICC/ESOMAR International Code, the client must be informed when AI is used in compiling datasets, analysis, reporting or interpretation, including synthetic data and personas, and the extent of human oversight must be stated.9

5. Is it safe to put interview transcripts into a chatbot? Only with care. Transcripts often hold personal data. Türkiye's KVKK says transfers abroad through generative AI providers must comply with Article 9 of Law No. 6698, and outputs containing personal data should be monitored with human oversight where needed.7 Ask which tool is used, where it is hosted and whether your data trains it.

Sources

Author: BigCat Research team. Last reviewed: 15 October 2026.

Footnotes

  1. MIT News, "Study finds ChatGPT boosts worker productivity for some writing tasks", 14 July 2023, reporting Noy, S. and Zhang, W., Science (2023). https://news.mit.edu/2023/study-finds-chatgpt-boosts-worker-productivity-writing-0714. Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4 ↩5

  2. Noy, S. and Zhang, W., "Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence", MIT working paper (not peer reviewed), 2 March 2023, abstract. https://economics.mit.edu/sites/default/files/inline-files/Noy_Zhang_1.pdf. Accessed 15 October 2026. ↩

  3. Gilardi, F., Alizadeh, M. and Kubli, M., "ChatGPT outperforms crowd workers for text-annotation tasks", PNAS 120(30): e2305016120, 18 July 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10372638/. Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4 ↩5

  4. Walters, W. H. and Wilder, E. I., "Fabrication and errors in the bibliographic citations generated by ChatGPT", Scientific Reports 13: 14045, 7 September 2023. https://www.nature.com/articles/s41598-023-41032-5 (open copy: https://pmc.ncbi.nlm.nih.gov/articles/PMC10484980/). Abstract. Accessed 15 October 2026. ↩ ↩2 ↩3

  5. Senz, K., "Can AI Predict Whether Shoppers Would Pick Crest or Colgate?", HBS Working Knowledge, 30 May 2023, on Brand, J., Israeli, A. and Ngwe, D., "Using GPT for Market Research", HBS Working Paper 23-062 (SSRN 4395751). https://www.library.hbs.edu/working-knowledge/can-ai-predict-whether-shoppers-would-pick-crest-over-colgate. Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4

  6. Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B. and Larson, J. M., "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models", Political Analysis 32(4): 401–416, published online 17 May 2024. https://www.cambridge.org/core/journals/political-analysis/article/synthetic-replacements-for-human-survey-data-the-perils-of-large-language-models/B92267DC26195C7F36E63EA04A47D2FE. Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4 ↩5

  7. Kişisel Verileri Koruma Kurumu (KVKK), Üretken Yapay Zekâ ve Kişisel Verilerin Korunması Rehberi (15 Soruda), Publication No. 113, November 2025. https://www.kvkk.gov.tr/SharedFolderServer/CMSFiles/MTY5MjNmNmIwZWY3YTE.pdf. Sections on risks (hallucinations), accuracy of outputs and transfer of personal data abroad. Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4

  8. Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C. and Wingate, D., "Out of One, Many: Using Language Models to Simulate Human Samples", Political Analysis 31(3): 337–351, published online 21 February 2023. https://www.cambridge.org/core/journals/political-analysis/article/out-of-one-many-using-language-models-to-simulate-human-samples/035D7C8A55B237942FB6DBAD7CAA4E49. Abstract. Accessed 15 October 2026. ↩

  9. ICC/ESOMAR, ICC/ESOMAR International Code on Market, Opinion and Social Research and Data Analytics, 2025. https://community.esomar.org/uploads/public/knowledge-and-standards/codes-and-guidelines/ICCESOMAR-International-Code_English.pdf. Definitions; Article 7(e)–(f); Article 9(b). Accessed 15 October 2026. ↩ ↩2 ↩3 ↩4 ↩5

  10. BigCat Research, BigAgent. https://bigcatresearch.com/en/bigagent/. Accessed 15 October 2026. ↩ ↩2

Related guides