I have tested out Navidrome, and scanning there works as intended. Plex has some scanning issues that makes it add secondary album entries randomly. I have now switched over to FLACs, and the issue is still there (well, the format wasn’t really an issue).
This is an issue specific to how Plex handles music scans and imports.