Navigate to a problematic movie or episode -> 3 dots -> Get info -> right column -> Color space
Usually SDR is bt709 and HDR is bt2020. In older codecs (h264) 10 or 12 bit color depth can also cause issues.
I’m not sure how tone mapping works on plex since I don’t have a pass but with Jellyfin you need to setup OpenCL which is run on the GPU so your guess that a hardware change can break it is plausible.
That’s a very old video format, which is definitely not HDR.
Some GPUs support the MPEG4 codec maybe try updating the driver. Did you try turning on or off Hardware acceleration for the transcodig?