If you didn't hear what someone said, you can ask:

What was that?

What did you say?

But asking "What did you say to me?" has a completely different meaning. You say this when someone insults you, challenges you, or says something rude:

A: Shut up.

B: Excuse me? What did you just say to me?

You should say this in an angry tone of voice, and stress the word "what".