When a lot of people can't do something, say "it's hard to ___":

It's hard to find a job in this economy.

But if you have trouble with something on your own, say "hard for me to":

It's hard for me to understand what they're saying when they talk so fast.