"Going to work" means leaving your house to go to your workplace.

It's a lot more common to say "go to work" than "go to my work", "go to my job", "go to the office", or other choices.