When someone praises or compliments you, sometimes you accept the praise. You think that you deserve the compliment.
Other times, you don't think that the person's compliment is really deserved. You think that they're just saying nice things to you to be polite, or that their opinion of you is higher than it should be. In these situations, you can respond to the praise or compliment with "That's nice of you to say."
A: You have a great singing voice!
B: What? That's very nice of you to say, but I don't think so at all!
Sometimes people respond to compliments this way, even when they completely agree with the compliment, in order to seem polite and modest.
This phrase can stand on its own:
That's nice of you to say.
Or "say" can have an object:
It's nice of you to say that.
Or it can be in this form:
That's not true, but it's nice of you to say.