I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users, project docs, business plan docs. A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff. They're able to generate this stuff way faster than we can review. It used to be that it would take a significant part of a day for a project manager to come up with a planning doc - now they can generate one in a few minutes and send it out for review. It's just really tiring.
The only way to even start to counter that is to make it a firm company policy that if you use an LLM to hallucinate any documents you absolutely must thoroughly review them yourself before you send them to anybody else, and that you are still responsible for the quality of LLM-generated content.
Getting an LLM to vomit out a bunch of documents and sending them straight to another colleague is absolutely unacceptable behaviour.
This is going to just run up against the insanity that is tokenmaxxing every moment of the day. When people are incentivized (upon pain of firing) to get the LLM to vomit out as much as possible, they're hardly going to stop and ponder if schlepping the slop over the wall is acceptable if the alternative is a pink slip.
We employees need to remember that most software projects fail. So the work we produce should have lower value than we give it.
We also need to be motivated to stay in our jobs.
Most developers like their projects and value their work. But the chances are that it's for nothing.
Many developers know they work on bad products (gambling industry, military, surveillance, whatever) and so it's here that they focus on their technologies, tools and frameworks rather than the work they produce.
"Agentic engineering" for example.
Id be curious to see what and how Googlers are doing with their 20% time.
If tokenmaxxing wins in your company, your company is going to lose. There's an external reality out there, outside your company, and your company has to produce things that actually work out there. Hallucinated AI slop does not help you do that. It leads you to unworkable plans, and if the plans produce, they produce unsellable products.
If you're an employee in that situation, push back if you can. If you can't, put your resume on the street. (But that may not work, these days. If it doesn't, all I can say is ride it out as best you can, and try to maintain both your job and your sanity. How? I don't know.)
Don't know if you are serious, but why become part of the problem?
Why not just review a single document quickly, find an error which invalidates the document, and send it back saying "Policy paper 1 mentions X as being on the business plan for Y, it's not on the plan, please can you fix."
Unless you can write a good-sounding reason why it's on them to review a LLM output before sending it to you, they will outsources this reviewing to you, and it's a lot of reviewing.
Original comment stated that it was 10 documents, all LLM-generated.
In my experience, it does take a lot time and effort to find contradictions between 10 documents. Even with good documentation, it's hard to build a mental map for that amount of information.
I always laugh because I've been practicing prompting every single day for the last few years, if they want to start a prompt fight, brother let me at 'em.
I find myself telling co-workers that if they couldn't be bothered writeing the email/document, why should i have to be bothered reading the document/email.
this is just spam, people sending unsolcited data at you and expecting you to swoollow and process that data.
its just rude and unreasonable, not to mention an unconscious (hopefully) act of sabotage.
Seems like for such requests it's necessary to get some proof of work: require a meeting where for every artifact they sent you to review, they briefly explain the gist and point out the motivation for creating the artifact.
> I'm getting so many requests to review LLM-generated documents
That's the other nightmare of AI slop. So easy to generate endless content. Who will review?
Just today the boss request I review slides for a presentation. But it's all AI slop, generated from querying tickets and docs and who knows what. It's mostly sort of correct but also plenty misleading and incorrect. So now I have to fact check all this slop which will take hours (even with my AI assistance) and rewrite most of it.
If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.
> If AI didn't exist, he would've had to do the research to generate the content and it would be 99% correct and I could just give a few notes of feedback in 5 minutes. But with the asymmetric AI workload, he can generate it in 5 minutes and I get to spend 3 hours correcting.
Maybe, depending on the boss. Some would have spent five minutes describing what they wanted, and someone else would have spent three hours creating the deck.
I'm fine with that. The company doesn't have infinite people so as long as someone spends 3 hours generating and I spend 5-15 minutes reviewing, that's fine.
Problem with AI is that generation is so many orders of magnitude faster than reviewing so it's basically infinite monkeys on infinite typewriters.
> A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff
You just discovered the unlock to massive AI-driven productivity increases: outsource the hard stuff to others, or just don't do it at all. Keep the easy tasks that generate a big volume of output for yourself.
I got this problem with my own employees, LLM are fine, but lazy slop is not permitted. Current idea is to have a clear "best practice" template for most of the research/specs/problem definition they submit and it reduced the slop to a manageable level.
But this might work in a smaller company where the management is reading and is strict about these things.
Wait, what? I thought everyone agrees that modern models post September 2025 (or whenever Opus or whatever 5.6789 was released) do not hallucinate, make things up, contradict themselves and can review their own output into perfection regardless of task, goal or context???? /s
In general I think from the coding side they're more robust now. However, people generating docs are maybe not as experienced with how to prompt in ways that avoid having the LLM tell you what you want to hear. I think this is still a pitfall that can easily be fallen into. Those of us who are doing LLM-assisted coding for the last couple of years are more aware of this now. Those who are planning/management folks are still kind of susceptible depending on how much experience they've had dealing with LLMs.
They do, just less. To the degree of being usable, as long as there are guardrails and they're used responsibly. For example, if there's code being output, there should be type checking and compilation, as well as code tests that prove that it works or that it doesn't - seeing how abysmal code coverage is in most of the projects I've seem, for whatever reason people thought that they didn't really need it much. They were wrong.
This also implies you need SOTA models on max reasoning.
> make things up
Same as above. Ideally you'd give them some way to verify their claims, like web search or browsing and referencing docs, Jira tickets etc., basically improve the signal to noise ratio.
> contradict themselves
They do so way less than before, as long as the above is true.
> can review their own output into perfection
They are pretty good at reviewing things, especially if you make them do adversarial review! It will never be perfect, but can be close in quality to human output (e.g. the code they produce, when used properly and with intent, is better than the code I've seen many developers write and ship before LLMs were a thing).
This also more or less scales with how much compute you give them - three parallel review agents will turn one output artifact into something good with higher confidence than two, and definitely better than with no review. There's a cost vs quality balance and it seems that all those xhigh and max reasoning modes are still geared way too much towards cost, instead of quality. So you have to make up for that shortcoming yourself.
> regardless of task, goal or context????
Garbage in, garbage out. I won't be an asshole and say that you're holding it wrong, nor will I say that anyone should listen to the claims marketing AI (absolutely delusional takes, meant to attract investors), but we're slowly getting to a better position in regards to LLMs, year by year.
It's just a shame that the peak of inflated expectations hit while the technology still hasn't fully plateaued and reached whatever its ceiling is.
I probably also shouldn't ignore the fact that some people will not care about any of it and send AI generated slop verbatim and to an outside observer there's no way to easily tell apart the difference between the two, unless you make a technical report contain exact references to where the data is sourced from, for example (and then either verify the references yourself, or make another agent do it).
Yeah, my bad, though I’ve also heard those arguments more or less said genuinely - on one hand people hold LLMs to some unreasonably high standard, expecting to one shot apps before being deemed good, and on the other just outputting slop with no regard for the quality.
I'm getting so many requests to review LLM-generated documents - planning docs, docs intended for end-users, project docs, business plan docs. A team member sent me a zip file with about 30 LLM generated documents in it the other day and asked if I could review them right away. And a lot of it was just repetition and/or stuff that was just out of left field, made-up, hallucinated stuff. They're able to generate this stuff way faster than we can review. It used to be that it would take a significant part of a day for a project manager to come up with a planning doc - now they can generate one in a few minutes and send it out for review. It's just really tiring.