"Thanks" is a more casual way to say "thank you".
You say "thanks anyway" when:
- you've asked someone for help or information
- they tried to help you but it didn't work
For example, if you ask someone for directions but they've never heard of the place that you're looking for, you can say:
OK, thanks anyway.
"Thanks anyway" usually sounds polite.