The same ones that terminate TLS for millions. Most are garbage but at least they keep the key offline. Also you can scale horizontally or only use the HSM as a CA to sign short lived host keys.
You could also use things like Nitro enclaves which have all the same specs as a regular EC2 instance.
Take Utimaco Security Server and you'll get 10k+ RSA signatures per seconds. Yes, you'll definitely need dozens of them, but you'll probably want several for high availability and low latency anyway.
> For SSH? Only by having the same private key in all of them, which means it's still around somewhere.
End-to-End encrypted key transfer between HSMs is well established. Ops for such a setup is definitely going to be a pain and lots of manual (and thus expensive) work but it is doable. The banking industry has been operating like that since forever – with symmetric cryptography only. Imagine two people being sent letters with XOR halves of a transport key and physically meeting at an HSM and entering the halves on a PIN pad (not a hexadecimal but a decimal PIN pad, mind you, where you need to press shift for A-F). From a modern perspective it's totally bonkers, but it works.
If I was tasked with building something for large scale companies like GitHub, I would probably pass up on HSMs and use commodity measured boot on a minimal Linux for my trusted key servers. Outside of these key servers the SSH key would only be stored split up with Shamir-Secret-Sharing with few trusted employees which will only restore the key on ephemeral offline systems. Is that overkill? Very much depends on your threat model. Investing in the security of their build chain runners and data-at-rest integrity might have higher pay off. But then again, GitHub has become such a big player that we should also hold them to very high standards. And the setup can be re-used for all their secret management, e.g. TLS certificate private keys.
Thanks! That's hugely helpful - everything I could find myself seemed to be an order of magnitude or so slower than that (and they support ecdsa, too, so I can't even object on the basis of algorithmic support). With hindsight my reply was somewhat flippant - really I just wanted to push back on the idea that this was a problem that could be solved by simply sprinkling HSMs on it rather than something that requires a meaningful infastructural effort. Github is a sufficiently core piece of infrastructure that I agree more should be expected, and I hope this does serve as encouragement for them to do that.
