“I saw you checking out that lady in there.”

When you were at a restaurant with your dad, an attractive woman was sitting at a table nearby. You noticed that your father was looking at her a lot. After you leave the restaurant, you want to tease him for looking at the woman. You say:

I saw you checking out that lady in there.

see (someone)(doing something)

This means to see someone when they're doing something:

I just saw Judy talking to Tom in the hallway.

check (someone) out

To "check (someone) out" means to look at a person's whole body to see how attractive they are. You can check yourself out in a mirror, or check someone else out. You can also "check out" a certain part of a person's body:

Were you checking out my ass?

This is very casual English, of course.

You can also "check out" an object, an e-mail message, a video, and so on:

Check out these shoes I bought. Nice, huh?

But this just means to look at something, and doesn't have a negative sexual meaning.