When you say that you "can't afford" to do something, it means that it's important not to do it. For example:

This is a really important meeting. I can't afford to be late.

We can't afford to take the risk.

You can also reverse the phrase:

At these prices, you can't afford not to buy!

This means "You have to buy."