This really depends on your exact location. In most of the “western” world, it would be legal. But just because it’s legal, doesn’t mean it will go unpunished. You could still be arrested and then have some charges thrown at you trying to make one stick. Disturbing the peace would probably be one of the charges that could stick. And of course other things such as resisting arrest. Whether or not it goes that far would depend on why, where, and how you were out in your underwear and if someone has it out for you or not.

Businesses and even certain public places can obviously refuse you entry.

If it’s legal to be in a swimming suit, it’s legal to be in underwear because you can argue it’s your swim suit.