When you're talking about doing something at the office that you work at, you can call it "the office" (not "my office" or  "my company's office"). "The office" refers not only to the location, but also the people who work at your office. For example:

 I only go into the office two or three days a week. The rest of the time, I can work from home.

I just got a call from the office. I'm sorry, but I'm going to have to cut this meeting short.