If the worry is over the opinions of ppl who aren't involved in your life or care about you, they're not real. They will always have a negative opinion because that's who they are. Why fuss over them? They're NPC background dialogue to tune out in search of a good side quest.
