Free speech also refers to a closely-held cultural value in the US, that is to say the respect for private citizens to hold and espouse beliefs which others find distasteful or offensive.

Saying "free speech only refers to the government censoring you" (at least in the context of America) is like saying that American society and culture consist _solely_ of the official bureaucratic proceedings of its constituent governmental bodies. In other words, its intellectually dishonest.