Sunflowerfield and all, I suspect that people that think a reply that says "no" is more rude, i.e. rejects, are the people that internalize rejection as a personal affront rather than a generalized "not interested". Hard not to do.
The essence of self-esteem means that you validate your own worth as a person. If you truly respect who you are, external circumstances or judgement won’t determine how you view yourself. Just being aware of this and stopping habitual responses to perceived rejection often make a huge difference. It is hard, but self awareness makes all the difference.
Personally I fail often ....lol! I do get back on the horse when I fall off with no self recrimination, and an apology to the other if necessary.
Unless the approach is really crass/ rude/offensive/obnoxious I do try to respond... but that is my personal decision. As I stated before, no one owes anyone anything here. I've learned that people are too varied and humans have so many differences, that to get upset over someone not being happy with who I am is beneficial or realistic. That changes in a close or intimate relationship of course, but THAT is a different tale....