When you say that you "can't afford" to do something, it means that it's important not to do it. For example:
This is a really important meeting. I can't afford to be late.
We can't afford to take the risk.
You can also reverse the phrase:
At these prices, you can't afford not to buy!
This means "You have to buy."