We're already hitting some logistical limits. Even if transformers don't have an inherent capability plateau, we only have so many GPUs and so much power to improve them, and we're finding it very difficult to expand that infrastructure. Something like 6 GW of new DCs have been announced over the past 2 years of which less than 1 GW has actually been turned on and started serving, and the deliverable dates for the rest just keep slipping. (Not to mention that the DCs are all talking as if the chips in them will last 6 years, which is turning out to be a stretch.)
Markov chains are very useful and imo so are LLMs, the issue I believe stems from calling it "inteligence" which creates some expectation which is often not met
