Inspect the icon of this playlist. If it has a little cog wheel overlaid on it, it is a “smart” playlist.
Smart playlists are built by a filter condition. Any newly added file which fits this condition is automatically added to the playlist.
If the playlist was created in plexamp, you might not be able to edit the filters with the web app (yet).