Crap, what do I do now?

Anybody who knows me, knows that I am not one for being touched just because. And that goes doubly for people I do not know. So the other day when I was walking home from the store and a man in front of the old folks apartments on the corner stepped out into my path I was a little off put, but it gets worse. This man has waved or whatever from a distance a few times. Once he asked me what I was listening to. A few months ago I commented to Mike that he kind of gives me the heeeby jeebies, but that seemed to be the scope of it. I was wrong. So, he is now in between home and me, and very in my personal space. He's making polite conversation, but he's so close that I can smell his breath, and I am frightened. I tell him I need to get home to my son and his father. He reached up and grabbed my face with this lecherous look on his face and said "you want a ride?", in that voice that makes your skin crawl. On one hand, as I type this I kind of feel stupid. Like people will think it's no big deal. But I don't care. Honestly even if there weren't things in my past that might condition me to be more wary of things, I still think this would have creeped me out.
Anyway here we are three days or so later, and I just got back from sprinting past the building on my way home from the store. I could force Mike to go to the store, but it's my one little bit of outside exercise I get a day. And I'm a little pissed that I am now scared to walk down the street in broad day light. Now I want to move all the sooner, and I have debated on changed my route to a much longer one to go to the store. Am I wrong for being upset/scared?