This is an example of understatement. If you're talking about someone and say:
She's not exactly poor.
...it means that she actually has a lot of money. Instead of simply saying "She's rich", you might use this expression in order to sound witty and intelligent.
Here's another example: if your husband is overweight, but made fun of you for being a little heavy, you can say:
You're not exactly in the best shape yourself, you know.
"Not exactly" can be followed either by an adjective, or by a noun:
A: You're dumb.
B: You're not exactly a rocket scientist yourself.