“I can never tell whether he's joking.”

You have an uncle who has a strange sense of humor.

He tells jokes, but his face always stays serious. You get confused by this. Now you're talking about your uncle's personality with your mother, and you say:

I can never tell whether he's joking.

I can't tell whether (clause)

In this case, "tell" doesn't mean "say". It means "decide" or "figure out". The phrase "I can't tell ___" is used to talk about something you're not sure about:

I can't tell whether it's finished or not. I can't tell whether she's being shy or just doesn't want to talk to me.

The examples above use "whether", but you can also use other question words like "how", "what", "who", and so on:

I couldn't tell what they were saying. For the life of me, I can not tell how he did it.

(someone) is joking

When someone "is joking", it means that they're saying something that's not true, in order to be funny:

A: You're quitting? Are you serious? B: No, I was just joking.

You probably also know that "jokes" are little funny stories or sayings that you tell people. When you're doing this, it's called "telling" a joke:

He always tries to tell jokes, but he usually messes them up.