I think the issue is that many Lemmy users will think more carefully about what they comment than what they up/downvote, as a comment appears connected to your username but a vote doesn’t. You might decide against commenting on something you disagree with because you don’t want to get in a fight, instead just downvoting it, but if people then know if was you who downvoted can still pick the fight.

Basically the issue is you’re revealing a lot more information than you might initially have realised if you’d have known votes were public all along. Maybe a disgruntled person uses that to dox you, or maybe a corpo feeds all that information into their fancy computer system to work out who you might be, who knows.