Allowing you to do whatever you want is not the same as allowing everyone to do whatever they want. You can't understand free software from an egocentric view of the world. A non-free derivative will ultimately restrict someone's freedom, hence, considering it restricting freedom is perfectly applicable.
We could debate forever on the semantics of which is more free, but I don't think having the freedom to restrict the freedoms of others is going to encourage freedom.
It's not freedom in any sense of the word. The FSF is at best a fiefdom, but in all reality a dictatorship. Stallmans utopian future is more 1984 than any of the actors he accuses. This whole thing is bullshit.
Your freedom ends where it conflicts with the freedoms of others. At that point, clearly something needs curtailing and the question is what/how. The GPL is one resolution.
My freedom to swing my fist stops at the tip of your nose; my freedom to modify legalese surrounding GPL software stops when it would deprive others of their "four essential freedoms" (in the words of the FSF).
0. The freedom to run the program, for any purpose.
1. The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this.
2. The freedom to redistribute copies so you can help your neighbor.
3. The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes.
If you don't believe that these are important and should be respected, then you shouldn't be licensing things under the GPL. If you believe these are more important than the restriction imposed, then releasing under a permissive license is not itself "restricting freedom", but all it is doing (compared to the GPL) is enabling "restricting freedom". When the only thing a change does is make a bad thing possible, it's still better not to make that change.
I find it somewhat interesting we have no viral license that protects freedom 0 and 1 in derivative works but allows adding additional restrictions that conflict with 2 and 3 (or don't, which would make it GPL compatible...).
That's a common misconception amongst software developers, but in fact its not your freedom that's being defended, its the users of software. So let's rephrase that:
Interesting that preventing users from doing something with a bit of software (e.g. selling non-free derivatives) is considered restricting freedom.
Kind of self evident when stated from the users point of view really.
I'd say more than 99.9% of them are 'using' GCC or clang, rather than 'developing' it, even if they are 'developers' using it to develop other software. So they're probably quite happy to have their rights as users privileged over their rights as developers in this particular case, particularly if it stops other developers from ruining things for them as users.
I compile my C source code with it, or other people's. I don't sell the programs that I write, or get paid for writing them - for those reasons I don't consider myself to be a software developer, most of my peers think I am wasting my time.