The public key is public and there is a single vulnerable private key. Someone can identify you with the use of your public key and if someone gets access to your private key (maybe a solen device) they can decrypt logged messages that used that key. This means they can still get access even if you deleted the messages off your device.