Adding plex to your group (or creating a new group entirely for this purpose and adding your user and plex) and setting the desired group ownership permissions are probably the simplest way of accomplishing this. Aside from the wide open permissions you mentioned, of course.
However, I like ACLs for problems like these. They allow you to specify very targeted permissions for a specific user, while leaving intact the traditional Unix permissions. Something like:
setfacl -R -d -m u:plex:rwx /home/me/media
This will recursively set the ACLs for the files/directories in the specified path to allow user plex to read, write, and execute. This may not be appropriate in all circumstances; find with -type and -exec would be helpful here (and removing the from the setfacl command).