A person who is "petty" is mean to other people for small and unimportant reasons. Here are some examples of "petty" behavior:

• a teacher gives a student a failing grade on a test because the student wrote it in ink instead of pencil
• one of your children complains because his brother got a slightly larger piece of cake than he did
• a neighbor calls the police to complain about the volume when you listen to music on a weekend afternoon

"Pettiness" is the noun form of "petty". Use "pettiness" when you want to comment on this quality:

Pettiness is one of the least attractive qualities a man can have.

I'm tired of his pettiness and jealousy.