Absolute Series Scanner (for anime mainly)

@ZeroQI - Not crashing is important! =)

Just put in new code, deleted and recreated testing library, and everything looks good! Specials all seem to be offsetting to their proper places. Here’s logs if you want to look them over.

edit: I see it looks like you committed to master branch about ~16 hrs ago, so I’ll pull that down and start playing with it. Thanks for all the hard work!

@ZeroQI - Running into a new anidb2 mapping oddity. This time it is Strike Witches, specifically the movie. In ScudLee’s file it is correctly set to map the movie “episode 01” to s0e02, but ASS keeps putting it to s0e01 and I’m not sure why. I wanted to run it by you though in case there is a bug.

  <anime anidbid="8272" tvdbid="82484" defaulttvdbseason="0" imdbid="tt2322603">
    <name>Strike Witches Gekijouban</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-2;</mapping>
    </mapping-list>
  </anime>

I see it picking up the correct mapping in the log, but for some reason it doesn’t use it and instead puts it in as s0e01 on the next line:

2016-11-28 23:42:13,531 - ASS : INFO - =============================================================================================================================================================
2016-11-28 23:42:13,531 - ASS : INFO - Scanner call - root: '/mnt/videos/anime/Strike Witches', path: '[2012-03-17] Strike Witches the Movie [anidb2-8272]', dirs: '0', files: '1'
2016-11-28 23:42:13,531 - ASS : INFO - =============================================================================================================================================================
2016-11-28 23:42:13,532 - ASS : INFO - Use existing: '/tmp/ASS-tmp-anime-list-master.xml'
2016-11-28 23:42:13,720 - ASS : INFO - anidbTvdbMapping() - anidb: '8272', tvbdid: '82484', defaulttvdbseason: '0', name: 'Strike Witches Gekijouban', mappingList: '{'s0e1': 's0e2', 'episodeoffset': ''}'
2016-11-28 23:42:13,724 - ASS : INFO - "[2012-03-17] Strike Witches the Movie [tvdb-82484]" s0000e001 (Orig: s1e1) "Strike Witches the Movie 01 (BD 1280x720 Hi10P, AAC 6ch) [Underwater] [BB288877].mkv"

Attaching filelist and scanner log as well. (Ignore the 2007 OVA and Victory Arrow not using anidb2 tagging, that is due to it being wrong in ScudLee xml… a whole different issue.)

Something else I’ve noticed (and this may be more HAMA than ASS, not sure) is that it seems to have stopped loading the season folder art. It is pulling down the season folder art to the directory “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/TVDB/seasons” from what I can see, but when looking in Plex it just duplicates the main series folder/cover art onto the series instead of loading the unique art for each season. Here’s is what art it has downloaded for Sword Art Online seasons:

259640-0-2.jpg
259640-0-3.jpg
259640-0.jpg
259640-1-2.jpg
259640-1-3.jpg
259640-1-4.jpg
259640-1-5.jpg
259640-1.jpg
259640-2-2.jpg
259640-2-3.jpg
259640-2-4.jpg
259640-2-5.jpg
259640-2-6.jpg
259640-2-7.jpg
259640-2-8.jpg

But in Plex it just has the same art for each season:

Even if I click the edit button for the season cover art and go to Poster to change it, it is empty.

I know at some point it was loading different art because I remember them all being different before, but for some reason it doesn’t anymore. It isn’t just SAO either I’ve noticed it for all the series I’ve been loading.

@deusxanime : yeah agent issue definitely. Will have a look latter, but first scanner issue. workaround: [anidb2-8272-s0e2]
Please try https://github.com/ZeroQI/Absolute-Series-Scanner/blob/352221f9ca6aae42d4d7f61110eea94b28d5ae1e/Scanners/Series/Absolute%20Series%20Scanner.py and let me know if fixes. i changed the order to do season first, so the mapping s0e1 can be mapped to s0e2 then

if it downloaded the posters, should be working, maybe a corruption of the plex database, need to check. can you give me the filelist? have a batch to create fake files with the filename to reproduce user libraries…

Hi,

Finally getting around to trying to sort my anime and I haven’t had much luck thus far. I installed the absolute series scanner and I am using TheTVDB for my agent but it just dumps all the files into “1 season” matches the episodes of that season and then just doesn’t match the rest

The episodes are labeled
Naruto Shippuuden - 01 [480p]
Naruto Shippuuden - 02 [480p]
Naruto Shippuuden - 03 [480p]

Maybe I am assuming something incorrectly. I assumed that because I was doing absolute order I could put all the files in 1 folder. Do I still have to figure out what seasons they belong to and put them in different folders?

@ZeroQI - Got a chance to try it out tonight. I created a new test library and used the ASS code linked. Looks like that worked for Strike Witches (it got the movie/specials placed correctly), but I also decided to try it with Sword Art Online since that was giving us trouble before, and it seems to be breaking on that again and doubling up the Extra Edition movie and special into a single special in the Plex listing.

Something else very weird going on that it is listing the episodes out of order in Plex, as you can see in the first screenshot with s0e11 listed after s0e14. It initially did that with the Strike Witches specials as well (s0e02 was listed first, then s0e01, s0e03, 04, 05), but eventually seemed to fix itself. So far, SAO specials don’t seem to be fixing themselves though.

I’m attaching the log file. I have been attaching the filelist, but will do so again for Strike Witches and SAO. Do you want others as well? So far those are what I’ve been testing with. I have a bunch of filelists due to having to add grouping folders as root.

@Blindsay - Not sure how well the TVDB agent would work with this scanner (ASS). @ZeroQI also made the HAMA (sorry don’t have the forum link handy, but there is a thread for it as well) agent to compliment his ASS scanner for bringing anime into Plex, so I think it is definitely a good idea to pair them together.

@deusxanime said:
@Blindsay - Not sure how well the TVDB agent would work with this scanner (ASS). @ZeroQI also made the HAMA (sorry don’t have the forum link handy, but there is a thread for it as well) agent to compliment his ASS scanner for bringing anime into Plex, so I think it is definitely a good idea to pair them together.

Thanks, that did the trick!

@Blindsay : Please chech my scanner readme, especially the section labelled “Which Metadata/Title source to select?”

@deusxanime : scudLee’s mapping for SOA

 <anime anidbid="10022" tvdbid="259640" defaulttvdbseason="0">
    <name>Sword Art Online: Extra Edition</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-11;</mapping>
      <mapping anidbseason="1" tvdbseason="0">;1-10;</mapping>
    </mapping-list>
  </anime>

Scanner log:
“[2013-12-31] Sword Art Online - Extra Edition [tvdb-259640]” s0000e011 (Orig: s1e1) “Sword Art Online - Extra Edition - Ep 01 (BD 1920x1080, JPN+ENG) [CBM] [F675782D].mkv” “(, Jpn+Eng)”
“[2013-12-31] Sword Art Online - Extra Edition [tvdb-259640]” s0000e011 (Orig: s0e1) “Sword Art Online - Extra Edition - Special 01 - Sword Art Offline - Extra Edition (BD 1920x1080) [CBM] [FD28BF39].mkv” “7” “Sword Art Offline - Extra Edition”

According to mapping above:

  • s0e1 remapped to s0e11 is done ok
  • s1e1 remapped to s0e10 (nop was put in s0e1 too). note for me: season issue to sort but minor fix

Alright I got my anime all sorted out finally but one thing I noticed which isn’t a huge deal but it was a feature that I liked is when opening the folder for a tv show it would play a short audio clip from that show. Is that an option that I have to re-enable or is it unavailable with this scanner/agent

Tick theme songs in agent settings and plex settings

@ZeroQI - sry been busy past couple days, but you aren’t waiting for anything from me right? Didn’t see any new code to test out yet.

@deusxanime : look like it was int displayed as strings type of bug… Easier than thought…
Pushed to master but look like it works, but let me know any issue. We are getting there :wink:

"[2013-12-31] Sword Art Online - Extra Edition [tvdb-259640]" s0000e010 (Orig: s1e1) "Sword Art Online - Extra Edition - Ep 01 (BD 1920x1080, JPN+ENG) [CBM] [F675782D].mkv" "(, Jpn+Eng)" 
"[2013-12-31] Sword Art Online - Extra Edition [tvdb-259640]" s0000e011 (Orig: s0e1) "Sword Art Online - Extra Edition - Special 01 - Sword Art Offline - Extra Edition (BD 1920x1080) [CBM] [FD28BF39].mkv" "7" "Sword Art Offline - Extra Edition"

@ZeroQI - ran the new code in master you linked to above through a test library with SAO and Strike Witches. SAO specials seem to be mapping correctly now again, but still getting the oddity of s0e11 being listed out of order in Plex.

Strike Witches reverted back it seems though. The movie is being put in s0e01, doubling up with the Prelude, rather than mapping to s0e02 as it should.

  <anime anidbid="8272" tvdbid="82484" defaulttvdbseason="0" imdbid="tt2322603">
    <name>Strike Witches Gekijouban</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-2;</mapping>
    </mapping-list>
  </anime>

Attaching logs and filelists.

Actually I just looked closer at the ScudLee code for the Strike Witches Movie I put above and I think it is wrong. It should have:

<mapping anidbseason="1" tvdbseason="0">;1-2;</mapping>

So the defined mapping is wrong in ScudLee. Look like ASS is working properly after all right? I’ll probably have to “fix” that one by using a [tvdb-xxxx-s0e2] tag for now.

That brings up a point though I was considering making. What do you think about allowing local file override to ScudLee’s list. As I’ve started to go through my anime I’ve found many missing and/or wrong things in the ScudLee file just like above. I’ve tried to submit corrections and additions to both the ScudLee forum thread and the github issues, but both have sat for nearly 2 weeks now without being addressed. I wonder if you could check for a local file in the folder before going out and using the ScudLee file, like you do for the [tvdb4-xxxx] tagging?

@deusxanime : let’s look at the theroy first

Source: https://github.com/ScudLee/anime-lists

  • defaulttvdbseason - The corresponding theTVDB.com season. For one-off titles it will be 1 unless associated to a multi-episode series, in which case it will be 0. Series that span multiple seasons on theTVDB.com may be marked as a if the absolute episode numbering is defined and matches AniDb.net.
  • anidbseason - The AniDb.net season (either 1 for regular episodes or 0 for specials).
  • tvdbseason - The corresponding theTVDB.com season.

Order is:

  • mapping tag - individual mapping
  • mapping tag - season start end offset mapping
  • anime tag - default tvdb season

if "anime anidbid=“8272” tvdbid=“82484” defaulttvdbseason=“0"” then it means it is season 0 then by default ? That give 2 possibilities:

  • mapping anidbseason=“0” tvdbseason=“0”>;1-2; yeah seem wrong
  • mapping anidbseason=“1” tvdbseason=“0”>;1-2; should be this indeed

there is a custom one in HAMA already BUT not in the scanner, and it should be consistent so will add it back but will take some work
ANIDB_TVDB_MAPPING_CUSTOM = ‘anime-list-custom.xml’
if url==ANIDB_TVDB_MAPPING and Data.Exists(ANIDB_TVDB_MAPPING_CUSTOM): # Special case: if importing anidb tvdb mapping, load custom mapping entries first
Log.Info(“Loading local custom mapping - url: ‘%s’” % ANIDB_TVDB_MAPPING_CUSTOM)
result_custom = Data.Load(ANIDB_TVDB_MAPPING_CUSTOM)
result = result_custom[:result_custom.rfind("/anime-list")-1] + result[result.find(“anime-list”)+len("")+1:] #cut both fiels together removing ending and starting tags to do so

The file will be in root of Agent Data folder., scanners don’t have access to that so here is locations possible:

  • one copy in logs folder for hte scanner
  • one copy in agent data folders
  • or single copy in library root (better choice maybe)

I see you’ve made some changes however it seems to have broke some things. The frist 2 are from an older version of ASS before you started making changes to fix issues deusxanime found. The last one (Ah! My Goddess) is from about 16 hours (w/e code was on git 16 hours ago) ago for both ASS and HAMA.
Ah! My Goddess has no issues on the old ASS/HAMA

Logs files are only good for Ah! My Goddess as the other screenshots are older and from another Plex instance

Queens Blade

Sousei no Onmyouji


Ah! My Goddess

Another note is that Fate/Kaleid Liner Prisma Illya this dose not merge all season all the time. I made the Library 2 times and for one is didn’t merge anything the second time it merged all but season 2. This is the same for Natsume Yuujinchou which has no issues on old ASS HAMA is having same issues as Fate/Kaleid Liner Prisma Illya.

I might be missing something but i might as well post what I got.

edit:
Is it possible to change how the scanner finds the anime file list?
So /home/kyubey/acd-stored/anime will open once and it can cache all folders so it only opens the path
/home/kyubey/acd-stored/anime
once
I ask because when you are scanning a mounted cloud drive it reloads the local cache each time its opened and that can get really slow. You’ll see this in the logs. Or am I looking at this all wrong as I have no idea how it really works or whats going on.

scanner issues can cause that as files in folders that didn’t change do not necessarily get re-scanned, that drove me mad during development as i was fixing already resolved things, but needed to recreate the library after cleaning the database…

Too many scanner issues in a row, need a plan:

  • build list of series/eps that are problematic and use every aspect of ScudLee’s mapping file

Scanners are built to only scan if the folder content changed.
It took me a while but ASS is built that way, which is why i skip grouping folders and they have to be added as additional root folders. If your “local” cache is remote you are doing it wrong

Issues:

  • all ending es ep2 - “Ah! My Goddess: Flights of Fancy [tvdb-78920]” s0002e002 (Orig: s1e2) “[Exiled-Destiny]_Ah!My_Goddess_TV_2_Ep01(2A549842).mkv” “Ep01”
  • no issue in scanner log - “Sousei no Onmyouji [tvdb-307061]” s0001e001 “[DeadFish] Sousei no Onmyouji - 01 [720p][AAC].mp4”
  • Queen’s blade specials seen with same ep number, all ending as ep2
  • “[MystVortex] Queen’s Blade Rebellion [tvdb-87491]” s0000e001 “Queens Blade - Special 1 - Instructor Alleynes Grueling Live Boot Camp.mp4” “7” “Instructor Alleyne’s Grueling Live Boot Camp”
  • “[MystVortex] Queen’s Blade Rebellion [tvdb-87491]” s0000e002 “Queens Blade - Special 2 - Siggys Passionate Sacred Pose Lessons.mp4” “7” “Siggy’s Passionate Sacred Pose Lessons”
    “[MystVortex] Queen’s Blade Gyokuza wo Tsugu Mono [tvdb-87491]” s0002e002 (Orig: s1e2) “Queens Blade 2: The Evil Eye - Special 1 - Lets Still Appear - Gainos Academy Rampage! Wow! A Beach Volleyball Tournament Between the Beautiful Warriors.mp4” “The Evil Eye - Special 1 - Let’s Still Appear - Gainos Academy Rampage! Wow! A Beach Volleyball Tournament Between the Beautiful Warriors”
    “[MystVortex] Queen’s Blade Gyokuza wo Tsugu Mono [tvdb-87491]” s0002e002 (Orig: s1e2) “Queens Blade 2: The Evil Eye - Special 2 - Lets Still Appear - Gainos Academy Rampage! The Suffering of Little Genius Teacher Ymir.mp4” “The Evil Eye - Special 2 - Let’s Still Appear - Gainos Academy Rampage! The Suffering of Little Genius Teacher Ymir”

seem to be a string/int conflict… Looking into it
Will reproduce library then bug, then see why as i hardly changed anything, and nothing that should have impacted that

Plex dose work fine with Amazon drive just scans are slower when you add 8k eps.
If the issue is with ScudLee’s mapping file, I’ll find issue series/eps and report there.

edit I’ve made a folder with 0 sized files its almost a copy of what I got in my library but I did see some errors when I made it so some stuff might be missing. I’ve removed the following from the file names
All “!” have been removed and replaced all Acute/grave symbols with “'

@Munchkin_ Thanks that helped. It’s been broken longer but the scanner didn’t rescan since nothing changed… solved the issue with all episodes sharing the same ep number.
For queen’s blade specials no mapping exist in scudlee file so as designed… need to add local mapping file but it’s a bit tricky…