If you're a woman, "the opposite sex" is men; if you're a man, "the opposite sex" is women. You usually use this phrase when you're talking about how men and women relate to each other.