This means "You don't really think that; you're only trying to be nice to me." Say this when someone compliments you, but you don't believe them or if you want to seem humble:
A: You're one of the smartest people I've ever worked with.
B: Whatever! You're just saying that.
A longer version of "You're just saying that" is:
You're just saying that to be nice.
You can also use this phrase in negative form and in questions:
A: You look great!
B: Really? You're not just saying that?