When I mention my love of trains to people, they usually have never heard of anyone like me. And if they have, they don't have any negative preconceptions about railfans. They just say, "Oh, I have a another friend who loves trains too!" So I'm not afraid to tell people about my hobby. As mentioned above, it's an important part of me, and it runs deeper than most people realize. I like sharing this part of me with people, because it makes me different. Like the original poster, I have various other interests (cycling, hiking, skiing, RC vehicles) and a healthy social life. But railfanning is a really unique thing that I find other people are really curious about. In fact, sometimes they really get into it! No, they don't rush out to buy scanners and cameras, but they are always wowed when I bring them down to the tracks at night to watch a train go by.
One time I ran into a lady friend of mine and a guy friend of hers by coincidence when I was down by the tracks. They stayed with me for a few minutes until a freight train came through, then went on their way. Some weeks later, I get a call from this lady friend. The guy she was with that night wanted to know when trains were scheduled to go by, because he wanted to bring a girl down to the tracks to see one!!! Man, he was stealing my moves!

I'm telling you guys, I think we're onto something. A lot of people are railfans and they don't even know it!