"Dragging someone down" means making them unhappy, angry, frustrated, unsuccessful, or other bad things.
People who are in a bad mood or have bad personality traits themselves sometimes "drag down" others. For example:
His constant complaining is starting to drag down the entire team.
Another way to use this phrase is in the expression "drag ___ down to (your/his/her/my) level".
You can't let them drag you down to their level.