Apple's biggest bottleneck for real-world inference is prefill processing. They need a better GPGPU architecture, which is what I'm expecting M7 to reveal.
Agreed, seems like the M5 has already made steps in that direction, with 4x prompt processing / prefill performance vs. M4. [1] And token generation also got a 10% boost. The data below is only for Pro & Max but I think the base M5 got the same relative boosts vs. M4 base.
Chip BW (GB/s) GPU Cores Q4_0 Prompt Q4_0 Gen
M4 Pro (20c) 273 20 439.78 50.74
M4 Max (40c) 546 40 885.68 83.06
M5 Pro (20c) 307 20 ~1500 to 1700 ~56
M5 Max (40c) 614 40 ~3000 to 3500 ~92
The M5 GPU added better matmul and dot product support in hardware (IIRC) that really boosted performance of those kernels over previous GPUs. Current transformer models rely heavily on matmuls. Prefill (processing the current context) is mostly limited by raw GPU processing power where token generation (predicting the next token from the current context) is mostly limited by memory bandwidth.
While an older M-series might run a matmul or dot product kernel just fine the M5 can run them much faster.
The Apple chips are quick when the model is "warm" on your second and subsequent requests. It's also quick if you don't have a harness, and feed "Hi, how are you?" as a prompt. However, a lot of common usage is to have a harness with lots of instructions and context. Loading that context is slow. So, the first response may take ~1 minute to return the first token (varies widely by hardware, model, etc.).
Look at the "Q4_0 Prompt" column. That's the tokens-per-second processing the initial prompt/system prompt. This is where the "neural accelerators inside each GPU core" is seen most prominently.
I believe that the CPUs are actually limited by ram bandwidth more than the neural engine right when it comes to LLM processing?
Maybe the M7 introduces something new to get around the current ram bandwidth problems on the non-Ultra chips.