Absolute Series Scanner (for anime mainly)

@deusxanime filelist simplified and library name added although it was in the filename. will be in next submit. Posters downloaded but not added, agent issue.

mapping rule the same. it ends in same ep since they both are detected as s01e01 before mapping…
“Sword Art Online - Extra Edition - Special 01 - Sword Art Offline - Extra Edition (BD 1920x1080) [CBM] [FD28BF39].mkv” is not seen as a special so no season 0… due to fix i put for season 2008 due to year in title BUT since i fixed misc variable i can remove that regex line and all good

https://raw.githubusercontent.com/ZeroQI/Absolute-Series-Scanner/2dee820e706caa9ab37c81f3111765d6f062efac/Scanners/Series/Absolute%20Series%20Scanner.py

Agent doesn’t add posters for seasons so reuse serie one but download them…
the “*” at the end indicate the file is already there
Poster tvdb mxl: http://thetvdb.com/api/A27AD9BE0DA63333/series/259640/banners.xml

Weird, getImagesFromTVDB() is supposed to add season posters at the same time…
Episode screenshots and season posters are physically added but not used, i believe it is the metadata issue that make eps show out of order…
If the meta changed, we would get an error, and that’s not the case…

Please let me know any improvement to the scanner in the meantime.

@ZeroQI - Recreated library using latest ASS link that you posted and the fixes look good! SAO EE movie and special are both mapping correctly and s0e11 no longer ordering incorrectly either in Plex.

Strike Witches continues to work well too, everything mapping correctly.

Attaching logs for ASS/HAMA and filelist as always in case you want to look through them.

Also for the posters, etc, not sure why it is adding * in the log and thinking they are there. Nothing in those folders as I showed you, they are all empty still, unless I’m looking in the wrong location (under /Plug-in Support/Data/com.plexapp.agents.hama/DataItems/TVDB/posters). Does it cache somewhere that it thinks they were downloaded previously and so doesn’t redownload them? As I said I wiped out the Data and Caches folder for HAMA already trying to get it working, so was hoping that would have cleared out any previous references.

[mediapc@chrono TVDB] pwd /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/TVDB [mediapc@chrono TVDB] du -sh *
140K 259640.xml
48K 82484.xml
4.0K blank
12K _cache
4.0K episodes
12K fanart
4.0K graphical
4.0K posters
4.0K seasons
4.0K seasonswide
4.0K text
[mediapc@chrono TVDB]$

@deusxanime it adds a “*” if they were already present. Still working so it’s good…

For me seasons posters are downloaded, and in your case they also seem to be otherwise we would get errors:
url: ‘http://thetvdb.com/banners/seasons/82484-2-3.jpg’, num: ‘11’, filename: ‘TVDB/seasons/82484-2-3.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-0-2.jpg’, num: ‘12’, filename: ‘TVDB/seasons/82484-0-2.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-1-3.jpg’, num: ‘13’, filename: ‘TVDB/seasons/82484-1-3.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-1-2.jpg’, num: ‘14’, filename: ‘TVDB/seasons/82484-1-2.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-1.jpg’, num: ‘15’, filename: ‘TVDB/seasons/82484-1.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-2-2.jpg’, num: ‘16’, filename: ‘TVDB/seasons/82484-2-2.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-2.jpg’, num: ‘17’, filename: ‘TVDB/seasons/82484-2.jpg’*
url: ‘http://thetvdb.com/banners/seasons/82484-0.jpg’, num: ‘18’, filename: ‘TVDB/seasons/82484-0.jpg’*

However du -sh seem to say no jpg are downloaded, but the path are valid… Weird unless you have an ad-aware programs that prevent access to the URL due to the presence of “banners” in the URL…

@ZeroQI - Not sure I’m following. You say it puts a * next to things it has thinks it has already downloaded/exists?

Couple examples from HAMA log:

2016-12-20 09:14:39,324 - com.plexapp.agents.hama (7f76b2ffd700) : INFO (init/metadata_download:915) - url: ‘http://thetvdb.com/banners/posters/82484-3.jpg’, num: ‘2’, filename: ‘TVDB/posters/82484-3.jpg’*
2016-12-20 09:14:39,324 - com.plexapp.agents.hama (7f76b2ffd700) : INFO (init/metadata_download:915) - url: ‘http://thetvdb.com/banners/posters/82484-1.jpg’, num: ‘3’, filename: ‘TVDB/posters/82484-1.jpg’*

But there is nothing in those folders.

[mediapc@chrono TVDB]$ pwd
/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/TVDB
[mediapc@chrono TVDB]$ cd posters/
[mediapc@chrono posters]$ ls -l
total 0
[mediapc@chrono posters]$

Should there be?

def getImagesFromTVDB(… line 797 calls metadata_download line 898

  def metadata_download (self, metatype, url, num=99, filename="", url_thumbnail=None):  #if url in metatype:#  Log.Debug("url: '%s', num: '%s', filename: '%s'*" % (url, str(num), filename)) # Log.Debug(str(metatype))   #  return
    if url not in metatype:
      file = None
      if filename and Data.Exists(filename):  ### if stored locally load it
        try:                    file = Data.Load(filename)
        except Exception as e:  Log.Warn("could not load file '%s' present in cache, Exception: '%s'" % (filename, e))
      if file == None: ### if not loaded locally download it
        try:                    file = HTTP.Request(url_thumbnail if url_thumbnail else url, cacheTime=None).content
        except Exception as e:  Log.Error("error downloading, Exception: '%s'" % e); return
        else:  ### if downloaded, try saving in cache but folders need to exist
          if filename and not filename.endswith("/"):
            try:                    Data.Save(filename, file)
            except Exception as e:  Log.Error("could not write filename '%s' in Plugin Data Folder, Exception: '%s'" % (filename, e)); return
      if file:
        try:                    metatype[ url ] = Proxy.Preview(file, sort_order=num) if url_thumbnail else Proxy.Media(file, sort_order=num) # or metatype[ url ] != proxy_item # proxy_item = 
        except Exception as e:  Log.Error("issue adding picture to plex - url downloaded: '%s', filename: '%s', Exception: '%s'" % (url_thumbnail if url_thumbnail else url, filename, e)) #metatype.validate_keys( url_thumbnail if url_thumbnail else url ) # remove many posters, to avoid
        else:                   Log.Info( "url: '%s', num: '%d', filename: '%s'" % (url, num, filename))
    else:  Log.Info("url: '%s', num: '%d', filename: '%s'*" % (url, num, filename))

URL was already added to the database hence the asterisk and not trying
replace “if url not in metatype:” by “if not Data.Exists(filename):” and it should download until saved on disk…

Made this change to HAMA init py file:

    898   def metadata_download (self, metatype, url, num=99, filename="", url_thumbnail=None):  #if url in metatype:#  Log.Debug("url: '%s', num: '%s', filenam        e: '%s'*" % (url, str(num), filename)) # Log.Debug(str(metatype))   #  return
    899 ###    if url not in metatype:
    900      if not Data.Exists(filename):
    901       file = None

But I think that broke it, as it just hangs up now when trying to add library.

Reverted back and now working same as before.

@deusxanime number of spaces: 2 per indentation. you put one too many space. there should be 2 character difference with what is below…

@ZeroQI - didn’t realize it was that sensitive to whitespace! I fixed it as you said and then recreated test library (with only Strike Witches for now). It didn’t hang up like last time, but does seem to still break something as you can see how it looks after scanning in Plex.

Forgot to delete the hama logs before starting to make sure it was clean, but attaching the lastest one anyway if you want to take a look.

logs show hama start only.
Look like it’s split… does the custom scanner logs show the series name with tvdb-xxxx instead of anibd2 ?

I believe so. Here is the ASS log file.

@deusxanime look good seem to be plex issue causing series to be separated as all titles had the tvdbid. Didn’t like the leading brackets in serie name so scrubbed it out and improve logging file format, released new version on patch-3 branch. Updated hama to use custom mapping file in serie folder and update ep meta if there was no title to not update it each time

@ZeroQI - it hit me what my problem was. I had deleted the Plug-in Support Data and Caches directories for HAMA in an effort to fix the issue of season art not loading and forgotten to recreate the anime-list-custom.xml file for HAMA to use under /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems. Once I put that back in and recreated the library, Strike Witches combined correctly into a single entry as expected. Also added SAO and it worked as well. That is with the same version of ASS/HAMA I had before.

What is the new HAMA now? Sounds like you have it look for the custom xml’s under the series folder, same as ASS now? So the custom xml in the HAMA DataItems directory is no longer needed? I looked on github and just see the master branch (updated a few hours ago) and one other branch named update-1.

Should I just update to the newest master to get updates?

Tried updating to the latest master HAMA code from github, moved the anime-list-custom.xml in the HAMA DataItems directory out (to /tmp, just in case I need/want it again, but anyway it was no where HAMA could find it), and recreated test library. Still looks good! So HAMA must now be using the custom xml files in the series folder too along with ASS?

Both SAO and SW matched well and are in there now as single entries.

Attaching ASS/HAMA/filelist logs in case you want to see them.

@ZeroQI The Monogatari series specials If you still have the zip of files check
Tsukimonogatari [anidb2-10891] eps 1 and 2 are stacking while 3 shows as ep 2
Thats really only series I care about that was really messed up and you’ve made it look very nice now.

Hitsugi_no_Chaika/Specials/ shows as ep 1 in season 1
Fate Kaleid Prisma Illya 2wei [anidb2-10091] OVA is stacking with Fate Kaleid Liner Prisma Illya [anidb2-9042] OVA
Saki [anidb2-6246] Extras are stacking on ep 1 and 2 but I think I have to put those in a specials folder as they are not right now.

That’s all I’ve found nice work!

@deusxanime Well look good, apart the serie name contain leading brackets, so you did not had latest scanner code where i cleaned that up so logs are purer and less useless stuff passed the the agent. also separated the pre file anties from file entries if present, look cleaner now
folder_show = clean_string(folder_show)+" [anidb2-%s]" % anidb_id

That is the way it was meant to be. the scanner load custom file only and not online if present, simpler…

cannot find the line below output in logs so unsure if using latest code or it failed
Log.Info("Loading local custom mapping - url: '%s'" % scudlee_filename_custom)

@Munchkin_
https://forums.plex.tv/discussion/comment/1315078/#Comment_1315078
https://forums.plex.tv/discussion/comment/1315876/#Comment_1315876

I just push latest to master (was in pull request tested by deusxanime)
look like mapping file is off. both ep 1 and 2 are mapped to ep 15 so it’s working as designed. you need to write a custom mapping file to correct as latest update to xmlmapping files is from 2 months ago

https://raw.githubusercontent.com/ScudLee/anime-lists/master/anime-list-master.xml

Tsukimonogatari: Yotsugi Doll

;1-15;2-15;3-16;4-17;5-18;

Scanner call - root: '/volume1/downloads/_Plex Munchkin', path: 'Tsukimonogatari [anidb2-10891]', dirs: '0', files: '4'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Use existing: '/volume1/Plex/tmp_transcoding/ASS-tmp-anime-list-master.xml'
anidbTvdbMapping() - anidb: '10891', tvbdid: '102261', defaulttvdbseason: '0', name: 'Tsukimonogatari: Yotsugi Doll', mappingList: '{'s1e3': 's0e16', 's1e2': 's0e15', 's1e1': 's0e15', 's1e5': 's0e18', 's1e4': 's0e17', 'episodeoffset': ''}'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Tsukimonogatari [anidb2-10891]" s0000e015 (Orig: s01e01) "Tsukimonogatari 01 (720p Blu-ray 8bit AAC).mp4"
"Tsukimonogatari [anidb2-10891]" s0000e015 (Orig: s01e02) "Tsukimonogatari 02 (720p Blu-ray 8bit AAC).mp4"
"Tsukimonogatari [anidb2-10891]" s0000e016 (Orig: s01e03) "Tsukimonogatari 03 (720p Blu-ray 8bit AAC).mp4"
"Tsukimonogatari [anidb2-10891]" s0000e017 (Orig: s01e04) "Tsukimonogatari 04 (720p Blu-ray 8bit AAC).mp4"

specials folder should apply but it is detected seemingly as OAD therefore movie, should be able to fix

Scanner call - root: '/volume1/downloads/_Plex Munchkin', path: 'Hitsugi_no_Chaika/Specials', dirs: '0', files: '1'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Hitsugi no Chaika" s0001e001 "Hitsugi_no_Chaika_-_OAD_[1280x720_Blu-ray_FLAC][54920E82].mkv"

Seem as designed

Scanner call - root: '/volume1/downloads/_Plex Munchkin', path: 'Fate Kaleid Prisma Illya 2wei [anidb2-10091]', dirs: '0', files: '11'
        -------------------------------------------------------------------------------------------------------------------------------------------------------------
Use existing: '/volume1/Plex/tmp_transcoding/ASS-tmp-anime-list-master.xml'
        anidbTvdbMapping() - anidb: '10091', tvbdid: '269319', defaulttvdbseason: '2', name: 'Fate/Kaleid Liner Prisma Illya Zwei!', mappingList: '{'episodeoffset': ''}'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Fate Kaleid Prisma Illya 2wei [anidb2-10091]" s0000e501 "Fate-kaleid liner Prisma Illya 2wei - OVA [BD][720p][AAC].mp4" "OVA" 

And

Scanner call - root: '/volume1/downloads/_Plex Munchkin', path: 'Fate Kaleid Liner Prisma Illya [anidb2-9042]', dirs: '0', files: '12'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Use existing: '/volume1/Plex/tmp_transcoding/ASS-tmp-anime-list-master.xml'
anidbTvdbMapping() - anidb: '9042', tvbdid: '269319', defaulttvdbseason: '1', name: 'Fate/Kaleid Liner Prisma Illya', mappingList: '{'s0e4': 's0e5', 's0e1': 's0e2', 's0e2': 's0e3', 's0e3': 's0e4', 'episodeoffset': ''}'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Fate Kaleid Liner Prisma Illya [anidb2-9042]" s0000e502 "Fate-kaleid liner Prisma Illya - OVA [BD][720p][AAC].mp4" "OVA" 

Yes Speciqls golder wouls be nice, but try to update line 41 with the line below and let me know if scans better (might have to recreate library since the folder would not have changed and plex may not rerun the scanner):
'(^|(?P<show>.*?)[ _\.\-]+)(SP|SPECIAL|OAV|EXTRAS?) ?(?P<ep>\d{1,2}) ?(?P<title>.*)$', # 7 # 001-099 Specials

Anidb2 mode relies on proper mapping files. Don’t forget you can map directly to thetvdb: " [tvdb2-s02e13]" and it will offset all nicely, or use custom mapping files in the serie folder… If the mapping files are no longer maintained it could be a workaround. i forked his project in case…

@ZeroQI - Wasn’t sure where we were at so I grabbed the latest masters of ASS and HAMA off github, and I think I see what others were talking about as far as not grabbing metadata anymore, as least for my [anidb2-xxxx] tagged series. [anidb-xxxx] still seems to be working correctly. This is loaded up with a small variety of shows.

Attaching logs and filelist. Oddly no hama agent log got generated, which may be related.

@deusxanime sorry i messed up that one. Please replace:
folder_show = clean_string(folder_show)+" [anidb2-%s]" % anidb_id
With (respecing the same number of leading spaces)
folder_show = clean_string(folder_show)+" [tvdb-%s]" % a2_tvdbid
Hama should read custom mapping file in serie folder. Please confirm if works as expected or i need to work further

@ZeroQI - Figured out this was in ASS correct? Line 414? I replaced the code on that line as suggested. Lots of weirdness has ensued… Added the rest of my test files/series and created new library with this code. Lots of things that don’t seem to be grabbing metadata and doesn’t seem to be merging the anidb2-xxxx tags into single series like would be expected.

Oddly it didn’t generate any ASS log or filelist this time. Attaching the HAMA logs though.

edit: Switched the code back so I could at least generate a filelist for you to test with if you want. Attaching as well.

@deusxanime please update with upda

@deusxanime Please update with updated scanner source. this should fix scanner side. if you can’t get logs let me know

  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/__init__.py", line 740, in anidbTvdbMapping
    with open(os.path.join(dir, ANIDB_TVDB_MAPPING_CUSTOM), 'r') as scudlee_file:  scudlee_1 = scudlee_file.read()
NameError: global name 'open' is not defined

Well that explains…
creates new scanner and agent files. please update and test. since agent crashed, you may have to restart plex