"Not ___ to begin with" means something like "not even ___".

Imagine that you were supposed to have the day off from work, but your boss asked you to come in at the last minute. While you're at work, a coworker asks you to cover her shift and work later. You might say:

No way! I'm not even supposed to be here today to begin with.