Clifton, I also removed myself from the conversation when people got irritated with my stutter. Obviously, I did NOT want to stutter and I did not know how to cure it AT THE TIME, so it is as stupid as getting irritated with somebody for moving slowly in front of you because they are in a wheelchair.
I had a friend who would start singing the word I was stuttering on (random, I know) and everybody would start laughing because she made it SO much less awkward as it took the attention off of my stuttering and moved it on to her being a clown. I then calmed down and would continue without stuttering. I never feared stuttering around her, and what do you know, I actually stuttered much less around her.