This is an expression that you use when you want to say that something (like a "person") doesn't have very much of a quality or characteristic (like "niceness"):
not that nice of a person
The meaning of this is similar to:
not a very nice person
But "not that ___ of a ___" is more casual.