If you didn't hear what someone said, you can ask:
What was that?
What did you say?
But asking "What did you say to me?" has a completely different meaning. You say this when someone insults you, challenges you, or says something rude:
A: Shut up.
B: Excuse me? What did you just say to me?
You should say this in an angry tone of voice, and stress the word "what".