When you see the phrase "the four of us", you should be able to tell that:
There were 4 people in the group that went to the restaurant, including the writer.
The writer already mentioned how many people were in the group in an earlier sentence.
You can tell #2 because this phrase uses "the". English speakers use "the" when they think their audience will know what they're talking about.