It’s bad because of the sheer number of them. A single cat can kill over a million animals over its lifetime I believe. A single pair of cats can produce around a thousand cats over the course of a few years, so that adds up. I love cats but they’re absolutely an ecological disaster if left unchecked.
Neutered or not they will still kill for fun, that’s what cats do. And they have lots of fun. Indoor cats have a longer lifespan and are generally healthier than outdoor cats. Also, with the proper stimulation and environment, they’re happier indoors. You don’t need a mansion, you need high shelves, scratching posts, hiding places, etc. The amount of space is less important than the quality.