The vanguard chiplets even steps down to 30M… but they do claim to have a “Mead” design [1] designed to run GPT-3 in a single chip.
3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely).
I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear.
> unless they figure out a way to keep it analog all the way (really cool but unlikely)
Very unlikely for the connection to the cache RAM, seemingly impossible if we want to get an articulate output :)
> they do claim to have a “Mead” design [1] designed to run GPT-3 [ - i.e. to hold a 175b NN - ] in a single chip
Careful: that is the /intention/, but the chip is just the storage (and CiM) for the NN and other parts are missing - explained in the Vanguard, not explained in the Mead.
> Whether it works out or not will sadly be a matter of economics more than physics I fear
Yes, but:
-- what is not enabled today may often be tomorrow through advances, esp. in the economy of production;
-- a very great point about this product seems to be that the chips can be rewritten, the NN is not etched and static (cpr. Taalas): that makes the practical implications extremely relevant, the demand would be "screaming mob" like;
-- we have to go in that direction (of NNs in CiM) anyway, so it's just a matter of time, effort after effort we will get there.
This might end up being a strong use case for FeFET FRAM, nonvolatile, high density, and sram fast. In an alternative universe where fram infra was as widespread as flash, memory is fast, cheap, and nonvolatile. It’s just that flash built the infrastructure and climbing that ladder is nearly impossible.
I wouldn't really say SRAM-fast. Maybe DRAM-fast? At least in write speed. But the thing is, you don't need high write speed for this application. And read speed should be the same between flash and FeFETs.
Btw, I've seen some roadmaps in which foundries were heading towards integrating ferroelectrics and floating gates to accelerate switching. The two technologies are not mutually exclusive really, so I think we might see something in the relatively near future. The crazy memory market may help!
Temperature is a relatively trivial issue because it can either be compensated or the chip simply kept at a constant temperature (they’re low power chips anyway, throttling could be skipped to simplify the design). The ADCs and DACs are indeed the main thing though. The whole analog compute game is about making those power efficient and scaled enough that the math still works out in your favor. The demos do work though, this company is far from the only one; see [1] or [2] for example.
Flash NAND can routinely be bought with 4 bits per cell, perhaps even 5 soon (QLC and PLC drives). Since it has been proven to store 4 digital bits at production scale, I’m willing to bet an analog architecture running an LLM should be able to yield the performance analog of a 4-8 bit quantized model. Where in that 4-8 range is pretty crucial, but it depends on the specific design.
Joke’s on us, all of their pages are LLM pages! LLM generated, that is.
Btw, can guarantee that they are not ready to demonstrate that yet. They’re using 2D FLASH with 30M weights per die [1], so to get to 1T they will need… 33,333 dies. Interesting scaling problem to say the least
But they also declare having a "Mead" technology that stores at least 175b NNs in a single chip through 3D stacking - see https://www.mythic.ai/mead and other posts in this page.
A confusing thing is that the goal is tackled through a number of proposals... Why Vanguard if they have Mead? If Mead, how to get the memory integration that are explicit on Vanguard?
I have bent over backwards trying to enforce brevity with deepseek v4 flash to the point where I think I broke some things trying to do prompt injection in my Hermes setup and was still unsuccessful.
Meanwhile Sol blows me away and I want that to be my default for everything now.
In general though I seem to have the most success with a "<=10w" requirement in my prompts.
What I don't see listed and would be a good comparison is the STS models. OpenAI's live model is an absolute joy to talk with.
That one is more specific, but "vomit" captures the feeling of Opus 5's writing very well for me. I don't know if it's the watermarking, but every single language idiosyncrasy that Opus 4.x (x > 5) had has been pushed up to 11 on Opus 5. Plus we got nouns verbing and seams seaming.
It's really unusable for anything other than code. And I have to remove its incomprehensible comments 50% of the time before committing anyway. After interacting with it, "slop vomit" is truly the most fitting description. I have to admit I have lost my temper and spontaneously referred to its output as vomit more than once. Seems like I'm not the only one.
It can also go the other way. It outputs a paragraph so full of telegraphic technobabble that it’s just unreadable, inventing new jargon every other sentence, using terminology invented in its thinking stream and never explained. And when it combines the two issues and it outputs 4 pages of technobabble, it gets just exhausting to read. Paradoxically, given that 2 years ago everyone was using LLMs to summarize web pages, their ability to condense text (including their own thought stream) is still lackluster.
I dream of some post-processing diffusion optimizer just condensing all of their slop away. Until then, I just tell it to _avoid_ telegraphic and newspaper-title speech and to output one or two paragraphs of prose only.
That's such a fantastic video. I never totally grasped why hybrids were so much more efficient, because my naive assumptions about how they worked were so simplistic. The real-time graphs he showed were excellent for making his points.
TC is filled with these sorts of videos. If you have time to burn then they are basically all this quality. His interests are also just wildly all over the place. From Christmas lights to dishwashers to coffee machines you just don't know what the next video will be.
One thing I really don't like about this video, is that he was specifically describing the advantages of that specific van and drivetrain, that being Toyota Sienna, and he's not wrong at that, but then he made grandiose general statement like "Hybrids are simple" and "minivans are great".
No, quite a few other hybrids are just an electrical motor tacked on to an existing ICE drivetrain, and they are actually more complicated than a regular ICE car, with horrible reliability and repair cost. And other non-hybrid minivans are either seriously underpowered, or require a seriously big engine that eats fuel.
The complaints that people have against hybrids and minivans are valid, if they're not talking about Toyota hybrids or this specific Toyota minivan.
This is intentional repetition. More specifically, anaphora [1]. You may not personally like it, but it is a rhetorical device used to emphasize a point. This one also comes with a nice progression: storms, storms, heat, heat, farmland, groceries.
This substack article also comes with additional graphs, a much better story flow (data is progressively introduced and explained before reaching the final plot), and was posted 2 days before the OP. I agree with GP that it is significantly superior to the OP (which is likely AI slop). Thanks for posting it.
3D NAND flash can indeed routinely store hundreds of GB per die, so that’s proven. The question is about all the peripheral circuitry needed. Each attention block would need its own KV cache (i.e. SRAM or DRAM somewhere), plus DAC/ADC inputs and outputs, unless they figure out a way to keep it analog all the way (really cool but unlikely).
I think this field is very interesting, at least from a technology point of view. Whether it works out or not will sadly be a matter of economics more than physics I fear.
[1] https://www.mythic.ai/mead