The following is based on something that I wrote on my Facebook wall a couple of years ago - and still seems pretty much right to me.
I'm sick of people who think that freedom of speech is only about freedom from state censorship. That is a very narrow and legalistic view of it (often distorted by thinking that freedom of speech is just what is in the First Amendment to the US Constitution). Yes, resistance to state censorship is at the core of the concept, and perhaps it's the only part that can, in practice, be protected by constitutional arrangements, legal provisions, etc. But the deeper foundations for the idea also apply to attempts to suppress/deter/punish unwanted views through the power of popular opinion and feeling. If you are trying to deter/suppress certain views by organising with others to punish people for expressing them - even if you're trying to do this in informal ways that do not involve the exercise of government power - then you have not grasped the essence of what the concept is all about. If you attempt, e.g. through collective boycott action, to pressure a body such as an employer to enforce restrictions on what can be said on topics of general interest, then you don't get the essential idea of free speech.
On one hand, it's the ability to speak our minds and express ourselves relatively fearlessly, subject only to the risk being criticised (or of losing the friendship of individuals who deeply disagree, since no one has to engage socially with people with whom they are deeply out of sympathy), and even if what we end up saying is abhorrent, ugly, or plain wrong.
It's also about the availability of unpopular views for consideration, even if they are abhorrent, ugly, or wrong.
Please go back and re-read Mill's On Liberty if you think the liberal concern with freedom of speech and individual liberty is only about restricting the coercive power of the state. If you're motivated to use whatever power you can collectively organise to punish, deter, or suppress certain disliked viewpoints, you can't legitimately claim to be a free speech advocate merely because you're not supporting state censorship.