There’s no ‘select all’ button, but you can select everything by selecting the first item, then scrolling to the bottom of the library, and Shift+click the last item. This selects everything in between. This is very much like Windows File Manager.
There is a caveat to this: after selecting the first, you have to scroll to the end slowly enough so that Plex will load all the tiles, then Shift+click the last item. This problem is reported here.