(not something) to begin with
"Not ___ to begin with" means something like "not even ___".
Imagine that you were supposed to have the day off from work, but your boss asked you to come in at the last minute. While you're at work, a coworker asks you to cover her shift and work later. You might say:
No way! I'm not even supposed to be here today to begin with.