Yes, this is expected behaviour and a result of unfortunate naming for the “Unmatch” option.
When you manually unmatch something it actually just matches it to the personal media agent which differs from something that has never been matched before.
There is some discussion around this internally on ways to improve this.
Is there any way I can filter out these “matched to personal media agent” somehow? Or alternatively, is there a way to truly remove a match so it becomes unmatched?