I just tested this and cannot reproduce your experience.
Are you sure you picked the Smart Collection option in the Save As menu of the Advanced Filter (the save-as button has options to create regular playlists and collections as well as smart playlists and collections… so it’ll be an easy error to make)?
If so… on what version of PMS and what version of Plex Web are you experiencing this (e.g. hosted or bundled)?
