When I read the Tanach, there’s a lot of wiping out that God does. When I examine my own thoughts, I think, “It’s only a matter of time before He gets around to me.” I don’t act on those thoughts most of the time but it sure bothers me that I have them.
Yes, we all have thoughts that are immoral, inappropriate, or just not right, sometimes. This is part of our challenges as human beings. We are judged more on our actions than our thoughts because actions are a truer reflection of our real desires. We have many thoughts and feelings that are contradictory and each thought or feeling only reflects one aspect of our personality.
Many a parent feels like throttling a two year old with a tantrum, but the vast majority of parents control that feeling. Many a spouse has thoughts of attraction to people other than their spouse. This is human and natural. Only acting on those thoughts is adultery.
The fact that it bothers you that you have some thoughts shows that those thoughts are coming from only part of you, not all of you. And further, those thoughts may not even be coming from you at all, I mean the real you, they may be coming from what we call the “yetzer hara” the part of us that is not the soul but an element within the human being to give us moral challenges. We are designed by God in a way that gives us moral challenges. Your thoughts are most likely and example of challenges you face and are popping into your mind precisely to see what you will do with them. You may not be judged on those thoughts at all, merely on the actions that will or will not follow from the thoughts.
On a side note, the stories in Tanach show how patient God is. Even when our thoughts or actions are deserving of consequences, He delays punishment for a long time. He sends messages all that time to try to alert us to our failings. And even when there is a punishment, it is meant as a lesson and a message, not just to give a slap.