In this case, "tell" doesn't mean "say". It means "decide" or "figure out". The phrase "I can't tell ___" is used to talk about something you're not sure about:
I can't tell whether it's finished or not. I can't tell whether she's being shy or just doesn't want to talk to me.
You can use "can't tell" with any question word like "how", "what", "who", and so on:
I couldn't tell what they were saying.
For the life of me, I can not tell how he did it.