This is something that you say when you're really angry with a person. The message that it communicates is something close to "I hate you" and "Go away."
You'll mostly hear "Go to hell" from family members, spouses, girlfriends, and so on. It's more common for women to say than men, and more common for the lower-status person in a relationship than the higher-status person. For example, a teenager is more likely to tell his parents "Go to hell!" than the parents are to tell this to their children.
"Hell" is a horrible place that bad people go when they die in Western religion and mythology. So it's quite insulting to tell someone to "go to hell".