Guilt is the gift that keeps on giving.
I've felt guilty a lot -- sometimes for good reason, sometimes not.
Guilt is a critical piece of child development: guilt and fear. Sounds medieval? Well, I don't mean it that way. When children are raised, they are usually disciplined regularly (I don't mean cruelly beaten, or anything like that.) When children are punished for doing something their parents don't want them to do (like throwing a tantrum on the bus, in the supermarket, at church, whatever) the parents will spank the child, speak harshly, isolate the child for a while (solitary confinement in a "quiet space") and the like.
To the child, the punishment is frightening, because they fear the parent will withdraw their love and care. (We were always threatened with being sent to bed without supper, or being forced to get out of the car and walk home). The fear of punishment is learned, and internalized.
Guilt comes along with internalized morals: internalized fear. The child knows what he is supposed to do and supposed to not do, but sometimes does it anyway. Because the fear of punishment is internalized, he feels fear (called guilt) when he does something wrong.
Sounds awful, but this is what enables us to go through life obeying laws, not cheating, not stealing, not being unspeakably nasty to other people because they got in your way, and so on. In fact, it helps us be nice to other people, it helps us to be moral individuals.
Some people who have a brain disorder that interferes with the fear/guilt/obedience mechanism are classified as "psychopaths" or "sociopaths" to varying degrees. They don't feel guilt, and they can't feel responsible for what they do -- though cognitively they are perfectly capable of understanding guilt and responsibility. But they don't feel it, and feeling it is where the rubber hits the road.
So, a little guilt is a good and necessary thing, a lot of guilt is destructive.
When one feels guilty a lot, (like if one is religious, is gay, and can't square the religious condemnation of what one can not help but feel), then something has to give. Usually, and it's the best outcome, people decide their morals are too rigid, are incorrect, or just plain wrong. The alternative is to tear one's self to shreds feeling guilty for what one IS and can not change.
My solution to the Christian/gay/guilt problem was to decide that the church was wrong. Fortunately there were several religious organizations handy to facilitate this conclusion.
So, you can beat guilt. It doesn't mean you have to throw the moral bath water and the baby of principles out the back door. What you have to do is, maybe change the bath water and teach the baby better principles that come closer to matching your personal situation.
Good luck.
"In the name of mumble mumble mumble I hereby absolve you of all your sins. Go in peace."