Not now obviously, but think 50-100 years down the line. Today there is a very real threat of a lone hacker (or a small team) breaking into and damaging some piece of infrastructure and committing an act of terrorism that kills real people. What happens when real AI starts to be developed and frameworks start appearing, where kids learning to program can download Intelligence Software to play around with and make AI apps. Then the threat becomes more realistic.
Assuming Moore's law holds for quite a while, and that there are not any other natural limits that will get in the way of super-intelligent AI. My hunch is that Maciej is right that there are physical limitations we don't see yet because of the pace of progress over the last 50 years.
It's totally possible that we're about to run into a natural limit. But it's not so overwhelmingly likely that we shouldn't have a modest investment in preparing for what happens if we don't hit any natural limits for a while yet. It could be an existential threat that we're dealing with here.
I don't disagree, but the tricky part is where do we place the modest investment? The type of AI research happening now is too fundamental and low level to place any safeguards. We are so far away from even beginning to approach anything that could be described as motivation.
I don't know anyone who thinks that current AI research is in need of safeguards, we're all in agreement that current AI is too far away to be of any SAI-type danger. There is fundamental AI safety research that has a reasonable chance of applying regardless of what techniques we use to get to SAI. Fortunately, the funding situation for Friendly AI has improved significantly this year.
