TL;DR
In a randomized trial at Boston Consulting Group, consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. When the chatbot was removed, researchers found no evidence that the ChatGPT group performed better on a technical knowledge test. The authors describe the task gains as temporary and say a finding about overconfidence in the tool was exploratory.
A randomized trial involving Boston Consulting Group consultants found that access to ChatGPT improved their performance on data science tasks, but researchers found no evidence of a measurable gain on a technical test after the chatbot was taken away. Published in Nature Human Behaviour on October 26, 2026, the study examines whether generative AI helps workers learn skills they can later use independently.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The trial assigned consultants to use either ChatGPT or resources including Google and Stack Overflow. The ChatGPT group received 15 to 20 minutes of prompting instruction and used GPT-4. Participants completed two 90-minute tasks drawn from coding, statistics and prediction. According to the authors, 986 consultants were allocated to the study; 487 submitted work on both tasks and formed the main analysis sample. Most participants reported no coding experience.
Compared with the control group, consultants with ChatGPT scored 49 percentage points higher on coding, 20 points higher on statistics and 17 points higher on prediction. The scores still averaged below those of 44 BCG data scientists who completed the tasks without AI. After the work, participants answered five technical questions without ChatGPT. The reported differences between the ChatGPT and control groups ranged from −7.7 to +4.8 percentage points; none was statistically significant, and the joint test had p = 0.531.
The researchers also reported that the ChatGPT group became more confident that the tool could solve problems it could not actually solve. They describe this finding as exploratory. The study was preregistered before data collection, and the authors report that the tasks were designed so ChatGPT could assist but could not independently produce correct answers to every question.
Performance Gains Without Lasting Recall
The findings distinguish doing more with AI from learning to do more without it. In this trial, consultants could make substantial progress on technical work while the chatbot was available, but the post-task test did not show a measurable advantage over the group trained to use conventional online resources.
For employers, that difference affects how they evaluate AI training. A worker who completes a task with a chatbot may still need support when the tool is unavailable or when its answer is wrong. The results suggest that task performance alone may not establish whether employees have acquired durable skills. They also leave open how companies should pair AI access with instruction, independent checks and human review.
The study does not establish that AI cannot teach skills. It tests a particular short training, a specific set of tasks and a five-question assessment in one consulting firm. Its results offer evidence about those conditions, rather than a verdict on every workplace, tool or training approach.
Products Worth Considering
Bose QuietComfort Wireless Bluetooth Headphones – Active Over‑Ear Noise Cancelling with Mic, USB‑C Charging, Deep Bass, up to 24 h Playtime,
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Black
Samsung Galaxy Tab A11+ Kids Edition, 128 GB Android Tablet, Royal Blue
As an affiliate, we earn on qualifying purchases.
Inside the BCG Experiment
The trial took place at BCG in late March and early April 2024 and was preregistered on March 13 that year. Participants completed two of three tasks: cleaning and combining datasets in Python; reviewing a statistical model and interpreting residual plots; or building a model related to the predictability of international football matches.
The task scores were compared with the average performance of 44 BCG data scientists, who worked without AI. That benchmark indicates how close consultants came to the expert group; it does not mean the consultants matched or exceeded them. The study’s main analysis included participants who submitted work on both tasks, a smaller group than the 986 initially allocated.
The paper, by Wiles, Krayer and co-authors, appeared in Nature Human Behaviour in October 2026. Its conclusions concern the trial’s design and measures, including the short prompting session and the technical test given after ChatGPT access ended.
“The authors “find no evidence” that the ChatGPT group did better on the technical questions answered without the tool.”
— Wiles, Krayer and co-authors
How Far the Results Extend
The study does not show whether longer or different training could produce lasting learning. It also does not establish how results would vary across occupations, experience levels, AI systems or tasks beyond the three tested. The unaided assessment consisted of five technical questions, so it measures a limited slice of possible knowledge and skill.
The reported post-task differences were not statistically significant, but that result does not prove the groups were identical. The authors’ separate observation about increased confidence in ChatGPT’s abilities is explicitly exploratory; the material provided does not establish how large or durable that effect is.
Testing Longer-Term Skill Retention
The published results leave the next research question open: whether workers can retain skills after more sustained instruction and repeated practice with AI. Follow-up studies could test performance after longer delays, use broader assessments and compare different ways of teaching workers to check AI output. The material available does not identify a scheduled follow-up study or a next milestone.
For employers applying these findings, the immediate evidence supports tracking both AI-assisted output and unaided performance. The BCG trial does not specify a company policy, but it shows why those measures may answer different questions about what workers can do and what they have learned.
Key Questions
What did the BCG study find?
Consultants using ChatGPT scored higher than a control group on coding, statistics and prediction tasks. After ChatGPT was removed, researchers found no statistically significant group difference on a five-question technical test.
How much higher did the ChatGPT group score?
The reported advantages over the control group were 49 percentage points on coding, 20 points on statistics and 17 points on prediction. The ChatGPT group still averaged below 44 BCG data scientists who completed the tasks without AI.
Did participants retain more data science knowledge?
The authors reported no evidence that the ChatGPT group did better than the control group on the technical questions answered without ChatGPT. The test included five questions, so the result does not settle every measure of skill retention.
When and where was the trial conducted?
The experiment took place at Boston Consulting Group in late March and early April 2024. The paper was published in Nature Human Behaviour on October 26, 2026.
Did the study show that ChatGPT makes workers overconfident?
The authors reported that the ChatGPT group became more confident that the tool could solve problems it could not solve. They label this analysis exploratory, so it should not be treated as a definitive conclusion about workers generally.
Source: Thorsten Meyer AI
