This is an expression for talking about things that you don't usually do. It means "I'm not the kind of person who ___." For example, if someone offers you a drink and you like to drink, you can say:
Of course! I'm not one to turn down a drink!
Here are some other versions of this phrase:
I'm not usually one to argue with strangers, but that guy got on my nerves.
He's never been one to back down from a fight.