It’s difficult to know that for sure, which is why (e.g.) the US government wants to make sure that there is domestic chip manufacture with a completely controlled supply chain to make hardware for classified communications. It can help to consider the difference between targeted surveillance (spending millions to tap the President’s phone, to get big juicy national secrets) and dragnet surveillance (tapping everybody’s phone so that you can have dirt on Joe Schmoe if he does something interesting later, even if he is of no particular interest right now). Hardware backdoors would be used mostly for targeted surveillance.

Stuff like VPN’s and encrypted apps can be of considerable help against dragnet surveillance, which is what the civil privacy community mostly cares about. If you think you might be a subject of targeted surveillance, you have to be much more paranoid. Not just hardware backdoors in your computer, but suspicious white vans on your street, microphones in your flower pots, FBI agents under your bed, the whole bit.

There are some countermeasures you can take against hardware backdoors (electromagnetically isolate a computer from the network and transfer data from it by floppy disc or similar) but basically you’re in a different world if you’re dealing with this.

You mght like the book “Security Engineering” by Ross Anderson (older editions free online and still very good: https://www.cl.cam.ac.uk/archive/rja14/book.html and scroll down). It goes into this stuff, has lots of good overviews even if you gloss over the technical parts, and will generally help you see clearly in the topic.