If that's the case then this seems like a symptom of a problem I've been thinking about for a while...is it safe to build systems where it's impossible for someone to understand how it works? What happens when it breaks down? Who will be responsible?

You just described human society in the large, and the answer appears to be "We muddle on."

I think this phenomenon has become a lot more important in the last few decades...I think it was fully possible for one person to understand how a car worked, how household appliances worked, etc, until quite recently

Probably depends on how you slice the question / define "fully understand." "How it works" in terms of being able to tell yourself a story that makes sense? Sure. Car burns gas, gas makes engine go, engine turns wheels. But what about the details you'd need to actually make a car (which include but are not limited to thermodynamics and metallurgy)? Probably not.

One difference is that there are parts of the system that, most likely, not a single person understands nor, even with weeks of effort, could understand. At least with cars, you can be reasonably sure somebody understands or can reverse engineer any specific component. With certain software that is a lot less possible.

It would be like if our society couldn’t fully function without Roman concrete, but no one knew how to make it anymore. Which is sort of what happened during medieval times: people continued to use Roman roads and Roman bridges. But if anything degraded, there was no hope of repairing it.

That's unfortunate, because their simpler system worked better.

I don't think so. A few years ago, Google worked much better when you knew how to phrase queries. I often helped family members to find something online, just by rephrasing their original query.

Today, it doesn't matter how you build your query, Google returns good results in any case. That also means that you can't search for specific info by phrasing queries differently but for the vast majority of people it makes life much easier.

You have a good point here. However, it would be great if both directions would be maintained. One, with "good enough" results for the common crowd, and another one, with a scalpel sharp ability to dissect it.

If you can do that at Google's scale I'm sure they'd be happy to hear your proposal. I really doubt Google's engineer wanted to give up the precision (they'll use it for their daily job as well), I just think they couldn't justify the additional resources.

Don't make it sound like a lack of ability. They could keep giving time slices to the old code if they wanted to. And it wouldn't that that many resources, "justified" or not.

Keeping old code running within Google is surprisingly hard...

Most big services (eg. Google Docs) have entire teams of people who run them (SRE's). Without those people to keep the system going, it would probably break within a few weeks.

I suspect this might be a case of “it is worse in my specific use case, but better for the overall user base”.

I think it's more of a "we could support both cases, but the ROI on supporting the power user is not worth it".

We will see more frequently and in more areas since AI has become so hot in MBA buzzword bingo. It's fashionable, so businesses want, or even need, to say "yes we use AI."

Even Google falls into the trap that a worse solution with more fashionable tech gets deployed.