For coding specifically (there are many studies out there by now, but I know of these offhand):
https://www.mdpi.com/2076-3417/14/10/4115 -- probably the earliest one of its kind, finds over-reliance degrades critical skills but supplementary use is mostly harmless.
https://arxiv.org/html/2601.20245v2 -- Anthropic's study, same as above except supplementary use (like clarifying concepts) can actually be beneficial.
https://scale.stanford.edu/ai/repository/ai-meets-classroom-... -- "Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort." Interestingly, they found simply disabling copy-paste on the chatbot interface resulted in better outcomes!
(Multiple studies find that the outcome depends on how AI is used. Surprisingly, incorrect guidance / unreliability / hallucinations appear to be a bigger problem than over-reliance! That could also explain poor performance in some cases.)
My intuition, supported by these studies, is that as long as students are willing to do the hard cognitive work -- for which there is no substitute, really -- having LLM assistance is a boon. Which makes sense, it's comparable to having a tutor explain difficult concepts. This is why in my mind the real problem is that the incentives to use AI as a crutch are just too strong.
I’ve noticed you are posting a lot of studies around, some of which have been peer reviewed and some not, some argue against your point, and some show mixed results.
Are you a researcher in the pedagogical sciences? Regardless, you have to admit that the original claim has very little evidence behind it despite being testable. And also the caveat you tag onto the end is a pretty massive caveat, and from the sources you provided it seems that students which use in the way which you claim has been shown to be effective, that those students are in a minority anyway.
I'm not a researcher in the official sense, my interest is that of a parent whose kids are interested in programming and will be graduating into a world upended by AI, and how I can best prepare them for it. I always look to empirical evidence whenever there is a conflict of opinions, and there certainly are many opinions here!
I initially banned them from using LLMs for homework or coding assignments, because as above, my intuition is that you learn best by doing, and you won't learn anything if LLMs do everything for you.
On the other hand, I personally have learned insane amounts of a new subject matter simply by pair programming and conversing with an LLM. I could not even "cheat" and let the LLM do everything because the problem I tackled is not really addressed anywhere! This forced me to experiment a lot, which helped me learn very quickly.
This led me to wonder what "disciplined" use of LLMs can do for learning... which is how I came across a whole bunch of these studies.
I think your concern is really about disciplined use of LLMs, rather than the overall effect of LLMs on learning. And I would agree: students will just be too tempted to use them to cheat. However, I think those who have the discipline to use them judiciously can supercharge their learning like never before, but only as long they do the hard work of "building the muscles" without AI.
https://www.mdpi.com/2076-3417/14/10/4115 -- probably the earliest one of its kind, finds over-reliance degrades critical skills but supplementary use is mostly harmless.
https://arxiv.org/html/2601.20245v2 -- Anthropic's study, same as above except supplementary use (like clarifying concepts) can actually be beneficial.
https://scale.stanford.edu/ai/repository/ai-meets-classroom-... -- "Students who use LLMs as personal tutors by conversing about the topic and asking for explanations benefit from usage. However, learning is impaired for students who excessively rely on LLMs to solve practice exercises for them and thus do not invest sufficient own mental effort." Interestingly, they found simply disabling copy-paste on the chatbot interface resulted in better outcomes!
Beyond coding, I recently came across this new meta-study; largely positive findings (which it admits may be biased) but does highlight evidence of negative effects of over-reliance: https://www.sciencedirect.com/science/article/pii/S2666920X2...
(Multiple studies find that the outcome depends on how AI is used. Surprisingly, incorrect guidance / unreliability / hallucinations appear to be a bigger problem than over-reliance! That could also explain poor performance in some cases.)
My intuition, supported by these studies, is that as long as students are willing to do the hard cognitive work -- for which there is no substitute, really -- having LLM assistance is a boon. Which makes sense, it's comparable to having a tutor explain difficult concepts. This is why in my mind the real problem is that the incentives to use AI as a crutch are just too strong.