You're telling a story about something that happened in the past. You want to talk about someone's experiences before that time. In that case, you use "had ___" or "had never ___":

We went to a Korean restaurant downtown. I think he'd been there before, but I'd never even heard of it.

Remember that these phrases mean that something happened before a certain point in the past that you've mentioned. If you just want to talk about someone's past experiences, use "have ___" or "haven't ___":

We're going to a Korean restaurant downtown. I think he's been there before, but I've never even heard of it.