This is a phrase to use when you want to ask someone out on a date.
Of course, you don't just come out with this at the beginning of your conversation. You have to make small talk with them and maybe tell them some interesting stories about yourself first. Then you can make the invitation using this phrase.
If you want to invite someone to hang out with you, but you don't specifically want to make it a date, you have to use a more indirect method of asking. For example:
Have you seen that new movie Walking the Tracks? I was thinking of going to see it. You want to go with me?
Me and some friends of mine are going out to this bar on Fifth Street tonight. You're welcome to join us if you'd like.