This phrase means that a person didn't understand something or didn't know what it was. In the example above, the speaker didn't know whether the guy was serious or joking, whether he really didn't care about her or whether he was trying to hide his true feelings.
Other feelings that you might have when you "don't know what to make of" something are:
- You're not sure whether something is good or bad.
- You can't decide what something is.
- You don't know why a person said something the way that they said it.