Allowing you to do whatever you want is not the same as allowing everyone to do whatever they want. You can't understand free software from an egocentric view of the world. A non-free derivative will ultimately restrict someone's freedom, hence, considering it restricting freedom is perfectly applicable.
We could debate forever on the semantics of which is more free, but I don't think having the freedom to restrict the freedoms of others is going to encourage freedom.
It's not freedom in any sense of the word. The FSF is at best a fiefdom, but in all reality a dictatorship. Stallmans utopian future is more 1984 than any of the actors he accuses. This whole thing is bullshit.
We could debate forever on the semantics of which is more free, but I don't think having the freedom to restrict the freedoms of others is going to encourage freedom.