This is a set phrase which people say when they're leading a group of people and want to get started with the main topic of discussion. Here are some other situations in which someone might say "Let's get down to business":

• a teacher has been taking attendance and making announcements, and now wants to start his lesson.
• a boss has called one of her employees into her office for a yearly performance review, but started off by asking about a problem that they've been dealing with that day. Now she wants to start the performance review.

You can see that "Let's get down to business" can be used not only in business situations, but in any situation where you have one important thing that a group of people has come together to talk about.