Confused!

Adlai Stevenson once said "A free society is a place where it's safe to be unpopular". The quote has confused me to no-end, and I don't understand how to interpret this. How does one's safety get challenged if he or she is not popular?? I, for one, like to maintain a low profile, and remain submerged below the surface as much as I can. And I am safe. I mean, to be quite open, I don't find any reason to be unsafe. But again, this does not make me feel I am living in a free society. That, I feel, is an entirely different domain altogether. Maybe I am getting things wrong. Someone please try to help me understand this. Ok, I won't think anymore and rather concentrate on something else... something a bit more uncomplicated.