If you're not wearing flip-flops, a tank top and shorts that fall down, how are people going to see your glorious ass-crack and 100 tattoos? Those tattoos cost money, you know, much more than proper clothes. It's only fair to show them off.
This is never going to be a classy sport. The best you can do is be the best person you can and be an example for others by your actions. Be a gentleman, follow room- and game etiquette, dress as well as it is practical and affordable to you. Your clothes do not change the game, however. It's a reward of its own. The people who think they can change this sport are due for many and serious disappointments. I do think there is some hope for change at the highest level of tournament competition, as far as dresscode and overall behaviour is concerned, maybe some sponsorship could be possible. It's a remote possibility, but it does exist. Not even 3 cushion or snooker are inherently played by classy people, there are plenty of riff raff in those sports as well, but they have managed to create classy "cultures" at the top of the games, which have positively influenced the games overall. That's the best we can hope for in pool.
But don't think for a second that you can play pool in an average poolroom and not occationally wonder where your opponent or person on the table next to you did time or if he has a knife on him. It goes with the territory, unless you join a private club.