I can only agree. This hobby is purely for your own enjoyment. If any your friends truly believe you are weird for doing it, well they don't deserve to be your friend. My friends and my family (especially my son) fully support what I do. It keeps me from wandering down to the pub too much and it is a great way to while away some quiet time at home.
Now to answer the main question of the topic, I believe that it is a socially acceptable thing to do, however I think, to some degree anyway, that it is modellers who are afraid to own up to doing it that help perpetuate this myth.