> Why do only the poor quality LLM code generation users make PRs to open source projects, and never the engineers that know how to really use it correctly?
It could be that the engineers who are extremely productive with LLMs are landing PRs that look indistinguishable from good, hand-written PRs.
If so, where are all the new features in the open-source projects I use? Why hasn't GIMP replicated Photoshop? Why hasn't CUDA been fully reverse-engineered as an open source toolchain? These are unreasonable expectations, but only in response to unreasonable claims of productivity. What before took ten years should now only take one, right?
It seems likely that the gains from generating tons of code are being offset by the debt incurred to understanding what you're doing. We see lots of greenfield projects one-shotted with GPT or GLM or whatnot, but very little on the side of projects with long-term maintenance goals. This is telling, to me, that the _effective_ gains are much lower than perceived (it's lots of fun to see the thing crank out code at breakneck pace, probably contributing to this). Still quite nice, and very useful, but not a totally new paradigm.
They don't though, very few people were anti AI in the early days. Open source projects were largely initially very interested in LLM usage, and generally had a pro AI contribution policy. This changed when it did not help their lives
Eg if you check out Godot, initially they were cool with LLM code, and then had to clamp down on it because it was not leading to anything good
Scaling AI use in highly-collaborative environments, which open source projects are, is very challenging. I totally understand the apprehensiveness around it.
How is it challenging for open source projects? You just write a PR, and try to get it to meet the maintainer's quality bar. That's exactly how all code is written in any environment - you do code review and refine a PR until its good enough to be merged
With AI and how a lot of people use it, the quantity of written PRs goes up, but the quality goes down, and the maintainers of the project, who are often volunteers, now have to deal with this in addition to the standard work they've been doing before.
Just my experience, but the relatively few fully AI-generated merge requests we've received that were "big features" were all really bad.
For instance, we had one person submit a merge request to convert the entire GIMP codebase into Python, and it was just tons of files with stubcode implementations.
Another person submitted a fix for our OpenEXR plug-in to load YCbCr images, and when I reviewed it, they'd "imported" a non-existent version of the library that had an imaginary "OpenYCbCrOpenEXR ()" function. :)
Why are Amazon and Walmarts mobile apps still janky crap that costs them money? Why does Meta still have four redundant, broken sign-up flows before you can give them money to advertise?
Wherever these “10000x” engineering gains are… I’m wondering why they’re only for shit nobody sees or touches.
Those websites are janky despite having all the (non AI) solutions available already. They suck for reasons independent of the quality of LLMs. I have not worked inside a megacorp but my understanding is they suck because of what are essentially political roadblocks within the organisation.
Ok, let’s assume that’s the case and we’d never see those improve under any circumstances.
Where are all the killer new things? All we see are glorified hobby projects (guilty as charged) and an unimpressive pace of equally unimpressive feature additions to all the same old stuff.
Businesses with resources out there should tackling huge things we all want, that they never could have undertaken. But I’m not seeing it. Everything is just… more or less the same. And this tooling didn’t arrive last Wednesday.
Or did we get magical AI at a time when we’re also completely out of ideas? That seems unlikely.
This shit just isn’t adding up. We’re not seeinganything that isn’t more “here’s some more ai tools and services… they’re basically the same as the ones you saw last month.”
I agree they are over-hyped. Maybe the problem is that all the things LLMs are great for are essentially quality of life improvements. There are a dozen killer apps worth of functionality I get out of LLMs - near instant proofreader, one-shot code bugfixer, excellent cursory research tool, bibtext generator, etc. All of these things extend what I can do a bit but none of them instantly result in me being able to generate category killing apps. They still make stupid mistakes, they don't reduce my workload, but they reduce cognitive load on mundane tasks.
If that were true, we'd expect to see massively accelerated velocity of open source projects by these engineers. They should be creating new open source projects at a truly astounding rate, with new tooling springing up every day that dwarfs the existing open source space as their productivity completely eclipses traditional development
Instead, software is plodding along exactly the same as it did prior to LLM code generation, and there's no evidence of superprogrammers making superprojects in 1/10th of the time. With a 10x productivity gain, what used to take a year should take a month
I think what you're missing is that there are new open source projects being created at a very fast rate.
But that doesn’t mean they stick.
And the core issue isn’t AI, it’s that… a successful, highly adopted project requires time for people to know it exists, adopt it, and also time by the maintainer growing community and ensuring reliability. The people factor is more important. I see tons of new projects, but I’m not going to pick a brand new project that has high odds of being abandoned. I’m going to pick the one that consistently maintains it and has some adoption already.
I have seen engineers create very successful internal projects fairly quickly. And yes, a project that would have taken a year taking a month. And offering a lot of extra bells and whistles that you just wouldn’t have time for.
But these are greenfield internal projects, and the bar is much, much lower for those.
I have seen multiple internal incidents root-caused by an agent faster than the humans responding. Just because it can go up and down rabbit holes a lot faster.
The problem is that AI solves one bottleneck, but not others. One team member produces a huge amount of new PRs. (Like 12 solid enhancements and big fixes in a couple hours.) Now I have less time because I’m reviewing that. And we’re all context switching a lot more.
On top of that, I find AI workflows continue to be deeply immature, even though certain models are very effective and very good at troubleshooting.
The story around testing is not really improving for example. AI can write tests, but are they good? I don’t think we have much actually ensuring product quality and reliability automatically. Unit tests are not enough.
Collaboration is very poor too. Coworkers agent creates PR, and now I’m reviewing it, and now he sends my comments back to his agents… really clunky workflow especially since I’m ACTUALLY just prompting his agent.
Plus, certain models (Opus) are getting much worse at writing. I will not use Opus any more because the writing style is so horrible.
The constant change means approaches that worked well a couple months ago don’t work well today. And there are no real experts, because no one’s been doing this for long. And half the posts and learning out there are outdated, or straight up blogspam.
This makes it hard for people to learn and get better, despite the fact that models like Sol 5.6 are effective troubleshooters, and write decent code.
Github's Octoverse report kind of gives you that evidence, no? Pull requests landed in 2025 were up 30% over 2024. This year it's going to be much higher.
It could be that the engineers who are extremely productive with LLMs are landing PRs that look indistinguishable from good, hand-written PRs.