AIUI you can only connect two 3090s at a time with NV link? So you’d only have 48GB of fast combined memory? That’s not tremendously interesting as you’re still limited to the smaller models which, while impressive in their own right, are still IMO too limited to use as your only model.
* Maximum NVLink bandwidth between two 3090s: ~56 GB/s one way
* Maximum PCIe v4 bandwidth between two 3090s: 31.5 GB/s one way
* M5 Ultra memory bandwidth: 1.2 TB/s
I know that memory bandwidth is only one factor influencing LLM performance, but this seems like a major problem if your goal is to run larger models that won't fit on one 3090 - and even those fitting on two 3090s with NVLink will be pretty restricted.
