I wrote a script which finds most, if not all, partially downloaded or corrupted art, banners, covers, etc. The bad files can also be deleted automaticly and no more partial images are shown yay! Unfortunately it seems plex does not automatically redownload the deleted images. I still have to manually select an alternate image, or force refresh section metadata to get the complete images again. Still a time consuming process though having a list of which shows/movies need manual intervention instead of having to manually look at every image really helps.
How can I trigger the redownload of the deleted art images by their agent for individual shows/movies via http? I haven’t been able to find it in the documtation.
Of course preventing partial downloads to begin with is the ultimate solution, but at least this fixes it when itygoes wrong.
Thanks
I wrote a script which finds most, if not all, partially downloaded or corrupted art, banners, covers, etc. The bad files can also be deleted automaticly and no more partial images are shown yay! Unfortunately it seems plex does not automatically redownload the deleted images. I still have to manually select an alternate image, or force refresh section metadata to get the complete images again. Still a time consuming process though having a list of which shows/movies need manual intervention instead of having to manually look at every image really helps.
How can I trigger the redownload of the deleted art images by their agent for individual shows/movies via http? I haven't been able to find it in the documtation.
Of course preventing partial downloads to begin with is the ultimate solution, but at least this fixes it when itygoes wrong.
Thanks
wow - thank you. i was thinking of doing something similar for missing thumbnails. Now that you have written yours it is probably very easy for you to adapt
Last I looked at the database, an ID for thumbnail and art existed but then when you look it up the target objects did not exist !
Could you have a look?
Edit: Oh you have done thumbs as well !
I wrote a script which finds most, if not all, partially downloaded or corrupted art, banners, covers, etc. The bad files can also be deleted automaticly and no more partial images are shown yay! Unfortunately it seems plex does not automatically redownload the deleted images. I still have to manually select an alternate image, or force refresh section metadata to get the complete images again. Still a time consuming process though having a list of which shows/movies need manual intervention instead of having to manually look at every image really helps.
How can I trigger the redownload of the deleted art images by their agent for individual shows/movies via http? I haven't been able to find it in the documtation.
Of course preventing partial downloads to begin with is the ultimate solution, but at least this fixes it when itygoes wrong.
Thanks
Analyse....And care to share the script?
happy to wait until you have a perfect and packaged script !
happy to wait until you have a perfect and packaged script !
sa2000....This is the Dev forums, not the regular forums!
This is about asking Q's about development, as well as about sharing knowledge, so we all both stand stronger, as well as participate in making the perfect solutions....
As such, the sooner you share your development efforts, the stronger we all stand, and more people might also participate towards coding.....
The OP had a brilliant idea here, and nobody is going to steal that away from him!
If he post his work on github, then maybe others, with greater insight, might join him and making his solution even better!
I've seen that myself in my little project for finding unmatched items....
I had an idea, and have been blessed with the company of another forum user, that made my project sky rocket.
To see my little endeavor, check my signature
Best Regards
Tommy
I probably should have worded that more cautiously. The script is very dumb and very simple because it just looks for an obvious sign of corruption. It just loops though all files in all sub folders of a selected directory, checks if the file is an image, then checks if the bottom middle and bottom right pixles of the image are both gray (128, 128, 128) and if they are then it flags it, and optionally deletes it. It also checks for some errors when loading the file. For me at least one of these condition was true for all partially downloaded images and only partially download images, but might not be true for everyone, I don’t know. The script is very slow but it at least completes overnight for my library and what do I care if its slow when I’m sleeping. By looking in the directory of the flagged file its easy to see what show it was associated with and use the web interface to take care of the currently manual part of the fix process. I think I can make the script print the show name too so you won’t have to look at the directory, but it will slow the script down more since the only way I can i figure to do this so far involves a lot of requests and loops though lists without making it complicate or using the database directly. I might try just for fun though.
I think if I find what the web interface sends to PMS when manualy unmatching and rematching media I will be able to work out how to automatically redownload, I was just hoping someone already worked it out. That method will of course only work if your media matches automatically and manual searching for a match is not required, thankfully automatic matching works for my media. There must be a better way.
I’ll post what I have so far when I get home but don’t expect miracles. I only spent a few minutes on it before I got stuck on the redownloading part and stopped. I’d rather see nothing then a partial image but that’s just me.
sa2000....This is the Dev forums, not the regular forums!
Tommy
I keep on missing that... It is because the Forums 'New Content' page gives the forum and does not display which grouping the forum is in when listing new new threads !
It just said General and I did not look closely when opening the thread to read the post !
Would be good if they can give the grouping / forum section as well on that view
I probably should have worded that more cautiously. The script is very dumb and very simple because it just looks for an obvious sign of corruption. It just loops though all files in all sub folders of a selected directory, checks if the file is an image, then checks if the bottom middle and bottom right pixles of the image are both gray (128, 128, 128) and if they are then it flags it, and optionally deletes it. It also checks for some errors when loading the file. For me at least one of these condition was true for all partially downloaded images and only partially download images, but might not be true for everyone, I don't know. The script is very slow but it at least completes overnight for my library and what do I care if its slow when I'm sleeping. By looking in the directory of the flagged file its easy to see what show it was associated with and use the web interface to take care of the currently manual part of the fix process. I think I can make the script print the show name too so you won't have to look at the directory, but it will slow the script down more since the only way I can i figure to do this so far involves a lot of requests and loops though lists without making it complicate or using the database directly. I might try just for fun though.
I think if I find what the web interface sends to PMS when manualy unmatching and rematching media I will be able to work out how to automatically redownload, I was just hoping someone already worked it out. That method will of course only work if your media matches automatically and manual searching for a match is not required, thankfully automatic matching works for my media. There must be a better way.
I'll post what I have so far when I get home but don't expect miracles. I only spent a few minutes on it before I got stuck on the redownloading part and stopped. I'd rather see nothing then a partial image but that's just me.
well....does indeed sound cool so far, but when that's said.....haven't got the problem myself, and was merely stepping in here to make sure, that you didn't create something beautiful, and wanted to hide it for the world....
It's a cool idea, and deserves the fame :wub:
/T
Darn work keeps getting in the way. Attached goes a first proof of concept release for detection of bad images. It appears to work for me, but probably wont work for you. It also includes 2 examples of types of corruption found for test purposes.
This does not force plex to redownload the images, manual intervetion is still required so its probably not worth running for many users.
Here's how to run it:
1. Have python 2.6 installed
2. Open FindCorruptV1.py Edit the SearchDirectory point to your Plex metadata directory.
3. If running on a posix machine, set the SymlinkDirectory to point to a diretory where you would like links to the corrrupt images to be placed.
4. If not running on a posix machine then comment out (#) or delete the lines starting with os.symlink...
5. Run the application from command line, optionally direct the output to a file for later review (eg python FindCorruptV1.py | tee results.txt)
6. If running on a posix machine review the images in the symlink diretory. If it has worked correctly the directory should show all the corrupt posters, art, banners downloaded by Plex agents. If not running on a posix machine, then vist the directories printed at the command prompt and verify the corrupt images are there.
7. Optionally try and Delete the corrupt images. If your feeling very brave, then you can uncoment the os.unlink lines and run the script again to have the script delete the images for you.
8. Repeat this process with the SearchDirectory pointing at the photo transcoder cache. Untill this is done cached images may still show up in clients.
9. Unmatch and rematch the shows
Some notes:
1. The 128,128,128 check applies to images in the phototranscoder cache directory mostly, the phototranscoder appears to fill the missing data from the origional in the metadatadirectory with gray.
2. The IOERROR - image file truncated will be the one thrown for most if not all problems in the metadata directory. Images may be corrupted in other ways, this can be expanded to find other forms of corruption, but I havent seen any other yet.
3. The _combined and _stored directories may contain links to images stoed in other directories so duplicates will show up. I didnt filter them out because you have to remove the links too.
All the usual disclaimers apply -> Dont run somthing you don't understand. This could result in any kind of unexpected errors including resulting in you having to rebuild your plex setup from scratch. Always take a backup, etc. etc.
I'm cleaning version two of this which also finds the plex web addresses associcated with the corrupt files, saving some typing / clicking at least. That seems to be the first step in automating the redownload of the deleted corrupted images. I'll post that when I get a bit further.
Attached is a proof of concept script which generates a list of web addresses for plex entries which have corrupted thumb, cover, poster, art, banner data. This is based upon py-plex 0.0.2 with a couple of a couple of extensions, big credit to that libraries author. Like my other posts, this is a hack job with very little time spent on it and probably wont work for you! I put it out there anyway in case it can help.
The next step will be to replicate the POST data sent to PMS by the web client in order to unmatch/rematch the movie/show so that the corrupted images which have been deleted are redownloaded. After that I will consider cleanup if there is interest in it.
Limitations:
Only works for a movie section.
Requires manual input of several constants which should be automatically generated (ex section number).
No MS windows.
If the movie hash isnt in the selected section it will crash, if it cant find the hash try a different section or if you have alot of time then comment out that if line and it will scan all (time consuming).
Its very slow for large sections - start with a small section!
How to run:
1. Change Search Directory to address of the movie metadata directory on your computer
2. Update the symplink directory if desired.
3. Update the server address to your server
4. Enter a single movie section name which contains at corrupted art
5. Update web address to match your server and section number.
6. Run python FindCorruptV2.py
What to expect:
1. The application will generate the metada sha hashes of all the items in your selected section. Each line look something like this:
35 b49b2249d847fa7939b59ae0048c4c47439dfa3a 0.264355182648 seconds
The first number is just a counter
The second number is the hash used to match with the folder of the corrupted image as named in the file system
Third is the name.
Fourth number is the time it took to retrieve the XML and generate the hash. This may be long the first time and short subsequent times if request has triggered PMS to analyze the media.
2. It will scan the search directory you have input for corrupted images. See previous post for information on how it recoginzes corrupt images.
3. It will read print results which look somthing like this:
/home/user/Plex/Library/Application Support/Plex Media Server/Metadata/Movies/b/49b2249d847fa7939b59ae0048c4c47439dfa3a.bundle/Contents/_combined/posters/com.plexapp.agents.movieposterdb_3886e3cbaaa8347f0412fa647301df3f604966e0
http://192.168.1.11:32400/web/index.html#!/servers/f4ebd76ca4384d3d2/sections/19/details/58254
Blade Runner
The first entry is the file name thats corrupted, you can navigate to it and see if it really is and delete it if you want to.
The second is a link to the web interface address, depending on the on the terminal applicaiton you are using this may nicely be converted into a link you can click and open a browser!
The third is the items name, just to double check.
If he post his work on github, then maybe others, with greater insight, might join him and making his solution even better!
I've seen that myself in my little project for finding unmatched items....
I had an idea, and have been blessed with the company of another forum user, that made my project sky rocket.
To see my little endeavor, check my signature
Best Regards
Tommy
Off topic:
I get the hint. I'm quite embarassed to even be putting this out there in this condition, but I'm not sure I will have time to make them presentable and no point it keeping it to myslef just becaus it is ugly. If I can find time to make them presentable I will put them on GitHub.
I looked at your project. Unfortunatly it seems many people are still experiancing the the corrupt images issue I'm addressing in this thread. If it continues (I think bad proxy was the last I heard), then perhaps this cleaner can be made into a plugin like your project is so its more friendly to users.
Have Fun
Attached is a proof of concept script
You forgot to attach it ;)
Doh. Here you go.
I'm dropping this app altogether in favor of integrating image corruption identification directly into the metadata agents.
Starting with themoviedb then thetvdb bundles I'm modifying the the agents behavior to:
1: Validate the image after right after it has been downlaoded, retry at least once, then dont the corrupted image if it still fails. Hopefully this will mean the problem wont visable in the first place, just an entry in the log file.
2: Validate the already stored images when refeshing metadata so that users which already have bad images on their disk should get cleared out during the first refresh without having to run a seperate application.
My new issues are:
1. Trying to get some of the python libraries I want to use to run on the ARM based diskstation.
2. The plex proxies are working right now so I can't test the validation on download code easily since no bad downloads are occuring.
After spending a few hours trying to repeat this functinality in a fork of theMovieDB plugin, during which I ran into numerous problems which made the changes very difficult to test, I went back and just finished up the origional.
Attached (hopefully the first time this time) is version 3 which:
1. Finds all truncated images in the metadata directory and removes them.
2. Triggers a metadata refresh so the agent hopefully redownloads a complete image.
3. Removes just the corrupted images from the cache directory so they will be recreated based upon the complete origional when a client next asks for them.
Be sure to empty the trash and clean bundles before running.
For me at least this made the process of fixing broken posters/covers/art/thumbnails/banners, whatever you call them, images, completely automatic while minimizing the amount of time and bandwidth the server used (ie not just starting over).
All the usual disclaimers apply, the script does delete files by default! Do not use if you dont understand it.
I suggest the built in metadata agents be updated to verify images after download and try again or delete!
Have Fun
