This is an expression for talking about things that you don't usually do. It means "I'm not the kind of person who ___." For example, if someone offers you a drink and you like to drink, you can say:

Of course! I'm not one to turn down a drink!

Here are some other versions of this phrase:

I'm not usually one to argue with strangers, but that guy got on my nerves.

He's never been one to back down from a fight.