So... uh... why was their private key in a form that could be stored in a Git repo? In the old days, we used to put private keys in hardware co-processors that would never reveal an un-encrypted private key. I guess we don't do that anymore.
Imagine the amount of ssh connections that github must handle every second, I don't think it's so easy to hold the private key in a HSM while maintaining the necessary performance and availability.
HSM is just a computer. May be with crypto accelerators. You can set up OpenBSD boxes, put them into safes and disable any incoming connections other than your custom HSM protocol and that would work better than storing private keys on web frontends.
Yes, it makes things hard and unconventional to set up. But GitHub is not some small website.
That's not all of what an HSM is. Or should be. The beefier ones come with rf shielding to prevent bad guys from trying to reduce the key search space by listening to EM energy coming out of the box. And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
>And active key zeroation if it thinks you're trying to drill through the epoxy surrounding the crypto boundary.
Or passive! Probably it wasn't a real product but I recall reading about one that derived its key from the field generated by randomly arranged magnetic particles in the resin, or something like that. The point was to make it impossible to disturb the resin without altering the key.
Is it technically possible (and/or wise) to duplicate the key across multiple HSMs running in parallel? I'm guessing if you have a super massive CDN like Apple or Meta this is a necessary thing.
That sounds like an engineering problem. I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously and they definitely operate on a larger scale than github, although most of their traffic is HTTPS not SSH.
> I have my political opinions on Meta as much as the next person, but every bit of evidence I've seen so far is they take this kind of security thing seriously
Loading it from HSM to memory/keychain is probably fine too. It's certainly odd it found its way to a repo and makes you wonder how that could have even possibly happened. And what that indicates about their security practices in general.
Github is host to a large percent of US tech IP. Pretty concerning if you extrapolate.
The whole point of a HSM is to never reveal the private key, but instead to have the HSM perform the necessary crypto operation for each request. This wouldn't scale to the amount of ssh traffic I would imagine github sees.
I think it's completely reasonable that they have the ssh private key in some sort of configuration management repository, because they need to be able to deploy that key to all their public facing ssh servers. You would hope they would have more than 1 ssh server instance world wide for availability and resiliency.
> The whole point of a HSM is to never reveal the private key
That's not "the whole point". You can have methods to copy a key between HSMs, methods that can scale just fine while being much more secure than a file in a repository.
Or you could just buy HSMs that can do the number of private key operations per second that you require.
I've never seen the inside of GH's network, but I would be surprised to discover they're not distributing the load of SSH termination across a fleet of machines. Just put a HSM on each of the machines that terminates a SSH connection.
HSMs don't handle the entire load of the the connection, only the private key operations. At the 10k per sec sites I worked at, we attached some beefy HSMs to the few (less than 40) machines that terminated TLS.
