The best way I understand it is the self is the little mental agent that you think you can control. It's connected to the illusion of free will, for me. I stopped believing in free will for years but the illusion was still there (or, more accurately, I thought it was). But as soon as that was gone I stopped trying to control my thoughts and ever since it's felt like living my life is like watching some sort of movie. Truly feeling like there is no free will, rather than merely not believing in it is, ironically, really freeing.
In my experience, the experience of the self and the experience of free will is the same thing. Or rather, the delusion that you experience it. There is no self to have a sense of. That's the point. You think you feel something that you don't. You might not call it that. But once the belief has gone you realize you never had it in the first place, you just thought you did.
You won't really know what I'm talking about until you lose it, as far as I'm concerned. Once the Buddha reaches enlightenment he realizes that it feels like he had enlightenment all along except before he didn't know it and didn't see it. You gain enlightenment only when you stop craving it.
You'll only really know what it is when you've experienced the alternative. I strongly think that losing your sense of self is basically this:
https://en.wikipedia.org/wiki/Choiceless_awareness
Which, believe me, is much much deeper than merely intellectually recognizing that there is no free will. It's when the phenomenology matches the intellect. That's the key.
And, by the way, I was just as confused about what Sam Harris was rattling on about as you are, until it happened. And boy is it great. Living without anxiety, basically. And not caring one iota about the past or future apart from when it helps you get through the day (I still pay my bills before their due date, I just don't worry about it anymore).
And, from what I see of Sam Harris ... he is still struggling a lot when it comes to egolessness. Just look at his relationship to Twitter. He's an intermediate at best!