Absolute Series Scanner (for anime mainly)

Angel Tales Chu [anidb2-680] had split eps in mapping file from scudlee that crashed the scanner, solved
Will work on the rest tomorrow. note for me:

  • remove regex old american eps: year no ep number
  • add ep2 in orig ep string if present
  • rebuild library + test year

@ZeroQI - Forgot to say that I am using custom xml for ā€œAngel Tales Chu [anidb2-680]ā€ because there are different ways that those episodes were released. Broadcast vs DVD etc. ScudLee list uses weird combinations to map them, like you said with split/combined episodes, so I made a custom xml instead so they would just map directly like normal and send the special to ep12. So mine shouldn’t even be using ScudLee list for mapping. Here’s the custom xml for that series:

<?xml version="1.0" encoding="utf-8"?>
<anime-list>
  <anime anidbid="680" tvdbid="180111" defaulttvdbseason="2">
    <name>Tenshi no Shippo Chu!</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="2">;1-12;</mapping>
    </mapping-list>
  </anime>
</anime-list>

New one I found just using straight anidb tagging - ā€œBlack Jack Special - The 4 Miracles of Life [anidb-2672]ā€

All episodes in this map to episode 4.

Scanner log of the files.

Scanner call - root: '/mnt/videos/anime', path: 'Black Jack Special - The 4 Miracles of Life [anidb-2672]', dirs: '0', files: '5'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
file: '/mnt/videos/anime/Black Jack Special - The 4 Miracles of Life [anidb-2672]/Thumbs.db', ext: 'db' not in video_ext
misc: 'black jack special - the 4 miracles of life - ep 01 black jack special - the 4 miracles of life - ep 02 black jack special - the 4 miracles of life - ep 03 black jack special - the 4 miracles of life - ep 04 '
"Black Jack Special - The 4 Miracles of Life [anidb-2672]" s0001e004 "Black Jack Special - The 4 Miracles of Life - Ep 01 (640x480) [Froth-Bite] [96D6F531].avi" "12" "Miracles of Life - Ep 01"
"Black Jack Special - The 4 Miracles of Life [anidb-2672]" s0001e004 "Black Jack Special - The 4 Miracles of Life - Ep 02 (640x480) [Froth-Bite] [D4F9FBDE].avi" "12" "Miracles of Life - Ep 02"
"Black Jack Special - The 4 Miracles of Life [anidb-2672]" s0001e004 "Black Jack Special - The 4 Miracles of Life - Ep 03 (640x480) [Froth-Bite] [1FC34C04].avi" "12" "Miracles of Life - Ep 03"
"Black Jack Special - The 4 Miracles of Life [anidb-2672]" s0001e004 "Black Jack Special - The 4 Miracles of Life - Ep 04 (640x480) [Froth-Bite] [77A5467E].avi" "12" "Miracles of Life - Ep 04"

I ran into my first big series, Bleach, and was debating about how to add it to use theTVDB so I could have it as a single entry with the movies. So the two options were to use ā€œBleach [anidb2-2369]ā€ (probably with custom xml since specials are wrong in scudlee, but leave that aside for now) vs. ā€œBleach [tvdb4-74796]ā€.

The current scudlee entry is:

  <anime anidbid="2369" tvdbid="74796" defaulttvdbseason="a">
    <name>Bleach</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0">;1-0;2-0;3-2;4-99</mapping>
    </mapping-list>
    <supplemental-info>
      <studio>Studio Pierrot</studio>
    </supplemental-info>
  </anime>

And I looked up on the scudlee readme and it says ā€œaā€ for the defaulttvdbseason means ā€œSeries that span multiple seasons on theTVDB.com may be marked as a if the absolute episode numbering is defined and matches AniDb.netā€, which I’m not even sure what that means…

tvdb4-xxxx of course uses your tvdb4 mapping file, which works great, but then it doesn’t merge the entry for the main series with the one it generates with the movies in as specials. I was able to manually merge the separate entries in Plex, so not a big deal, but it got me wondering what what even happen if using the scudlee mapping? [Side note/reminder for others: With the tvdb4 method of course you have to name any specials the same as they are on theTVDB since it is not using the scudlee mapping.]

How does ASS handle the ā€œaā€ for defaulttvdbseason? Would it just dump it all into a single season? Guessing maybe that is why you made the tvdb4 idea in the first place?!

edit: OK so I got curious and tried it out myself, switch folder name to ā€œBleach [anidb2-2369]ā€ and then put in my test library. It did just dump all 366 episodes into a single season, which is cumbersome. So I think I’ll stick to tvdb4 mapping and can manage merging it. =) I’ll leave this here in case anyone wonders in the future…

edit2: After some more thought, with a complicated custom (anidb2-xxxx/scudlee) xml mapping file I could probably accomplish this too, mapping anidb s1 episodes to various tvdb seasons using the start, end, and offset parameters. But I kind of like how the tvdb4 mapping maintains absolute numbering while still breaking it up into smaller seasons, so I think I’m sticking with that!

in the tvdb xml for the serie, you can have absolute ep numbering also (optional), that simplifies the mapping, in this case since we know which absolute number goes in which season since both anidb and thetvdb absolute number match

tvdb4 is for separations in sagas and require keeping the absolute ep number, was for long running series (detective conan, dbz, one piece,…)

to consolidate, it needs the same guid, so you might have to go [tvdb4-74796-s0e1] but am unsure if it works

Just pushed a new scanner which handle ep numbers better without using the regex and not give the year issue

Edit: Why not tvdb3 mode? it had tvdb seasons but keep the absolute numbering ? i even explained all in the readme

So many modes, I haven’t even thought about tvdb3. Mostly I just stick to anidb and anidb2 tagging, but I knew of the special tvdb4 mapping file so I thought I’d try that out on the longer series like Bleach. Given what you say about tvdb3 tagging, I guess I’m not understanding the difference between it and tvdb4 then.

I’ll give the tvdb4-xxxx-sXeY style tagging a shot and see if it works as it is cleaner to have them ID’d as a single series during scanning, rather than manually merge them after, if possible (also makes life easier if I have to recreate the library as then I wouldn’t have to remember to go in and do that everytime).

edit: So I tried it out with ā€œBlood - The Last Vampire [tvdb4-78960-s0e1]ā€ in conjunction with main series ā€œBlood+ [tvdb4-78960]ā€ (which I also added to tvdb4 mapping file on github btw), and seems to have worked well. Going to go back and do Bleach the same way.

edit2: Yep, worked great with Bleach movies as well.

try tvdb3. it just display with tvdb seasons. tvdb4 is for sagas using any arrangement of seasons
please try the scanner i published when you can it fixes all bad ep numbers reported

I’ll try to give tvdb3 tagging a go and see what happens when I get a chance. I can try it with Bleach and Blood+.

I did try the new ASS code in master and looks like it fixed the issue with Assassination Classroom (2015) special. Anidb special 01 now mapping to s0e02 in Plex as expected, no more Season 2015.

Ran into new problem with Blue Exorcist though. The scudlee mapping has a negative offset for the specials:

  <anime anidbid="8148" tvdbid="248035" defaulttvdbseason="1">
    <name>Ao no Exorcist</name>
    <mapping-list>
      <mapping anidbseason="0" tvdbseason="0" start="2" end="11" offset="-1">;1-11;12-0;</mapping>
    </mapping-list>
  </anime>

Not sure if it doesn’t know how to handle negatives or what, but doesn’t seem to apply the offset. Here’s the scanner log showing how it is mapping the specials:

Scanner call - root: '/mnt/videos/anime', path: 'Blue Exorcist [anidb2-8148]/specials', dirs: '0', files: '11'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Use existing: '/tmp/ASS-tmp-anime-list-master.xml'
anidbTvdbMapping() - anidb: '8148', tvbdid: '248035', defaulttvdbseason: '1', name: 'Ao no Exorcist', mappingList: '{'s0': ['2', '11', '-1'], 's0e1': 's0e11', 's0e12': 's0e0', 'episodeoffset': ''}'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
misc: 'special 01 - extra special edition special 02 - ura eku 1 special 03 - ura eku 2 special 04 - ura eku 3 special 05 - ura eku 4 special 06 - ura eku 5 special 07 - ura eku 6 special 08 - ura eku 7 special 09 - ura eku 8 special 10 - ura eku 9 special 11 - ura eku 10 '
"Blue Exorcist [tvdb-248035]" s0000e011 (Orig: s00e01) "Blue Exorcist - Special 01 - Extra Special Edition (BD 1280x720) [Coalgirls] [845CB200].mkv" "7" "Extra Special Edition"
"Blue Exorcist [tvdb-248035]" s0000e002 "Blue Exorcist - Special 02 v2 - Ura Eku 1 (BD 1280x720) [tlacatlc6] [9BA0331A].mkv" "7" "Ura Eku 1"
"Blue Exorcist [tvdb-248035]" s0000e003 "Blue Exorcist - Special 03 v2 - Ura Eku 2 (BD 1280x720) [tlacatlc6] [4384B3AE].mkv" "7" "Ura Eku 2"
"Blue Exorcist [tvdb-248035]" s0000e004 "Blue Exorcist - Special 04 v2 - Ura Eku 3 (BD 1280x720) [tlacatlc6] [57B6C494].mkv" "7" "Ura Eku 3"
"Blue Exorcist [tvdb-248035]" s0000e005 "Blue Exorcist - Special 05 v2 - Ura Eku 4 (BD 1280x720) [tlacatlc6] [65F2888B].mkv" "7" "Ura Eku 4"
"Blue Exorcist [tvdb-248035]" s0000e006 "Blue Exorcist - Special 06 v2 - Ura Eku 5 (BD 1280x720) [tlacatlc6] [F31552D0].mkv" "7" "Ura Eku 5"
"Blue Exorcist [tvdb-248035]" s0000e007 "Blue Exorcist - Special 07 v2 - Ura Eku 6 (BD 1280x720) [tlacatlc6] [8BC515F9].mkv" "7" "Ura Eku 6"
"Blue Exorcist [tvdb-248035]" s0000e008 "Blue Exorcist - Special 08 - Ura Eku 7 (BD 1280x720) [tlacatlc6] [5A76293B].mkv" "7" "Ura Eku 7"
"Blue Exorcist [tvdb-248035]" s0000e009 "Blue Exorcist - Special 09 - Ura Eku 8 (BD 1280x720) [tlacatlc6] [72D07013].mkv" "7" "Ura Eku 8"
"Blue Exorcist [tvdb-248035]" s0000e010 "Blue Exorcist - Special 10 - Ura Eku 9 (BD 1280x720) [tlacatlc6] [D05DE84C].mkv" "7" "Ura Eku 9"
"Blue Exorcist [tvdb-248035]" s0000e011 "Blue Exorcist - Special 11 - Ura Eku 10 (BD 1280x720) [tlacatlc6] [CA9A7EC4].mkv" "7" "Ura Eku 10"

As you can see doesn’t seem to apply a -1 offset for specials 2-11 like is defined in the mapping.

season offset didn’t work, fixed it already, pushed to master.
You are finding all the things that don’t work, the scanner keep improving :smiley:

Tried new scanner and now Blue Exorcist is working with the negative offset!

I also tried out tvdb3-xxxx tagging on Cardcaptor Sakura, but it seems to be having issues. Every time I scan (do Plex Dance in between on the Cardcaptor folders) it seems to have some random number of episodes that are just missing. Also it doesn’t seem to be pulling down the episode title when using tvdb3 mode. Bleach and Blood+ with tvdb4 didn’t have either of those issues. Switching Cardcaptor back to anidb2-xxxx tagging and it works fine, though everything is in s1 is all (other than specials of course), but at least all episodes are there and grabs episode titles etc.

Edit: actually I just scrolled down on s1 after using anidb2 tagging and not grabbing ep titles after s1e35 either, maybe I’m just not being patient enough for metadata to pull down.

Edit2: realized that would make sense, since I’m using anidb2 it is pulling meta from tvdb and s1 only goes to e35 on there.

Also I noticed that for the multi episode special file ā€œCardcaptor Sakura - Special 01-03 (BD 960x720 Hi10P) [Coalgirls] [A427519B].mkvā€ it only seems to pick it up as a single episode. Does that no matter which mode I’m using - tvdb3 or anidb2. Not a huge deal but don’t think that is expected? Should pick it up and assign it to all 3 episodes correct?

Gotta run so can’t generate logs right now, but let me know if you want me to for the tvdb3 or other issue.

@deusxanime added specials multi episode support to ASS, already on github. Please give me ass custom log for Cardcaptor Sakura tvdb3 as you added exact season mappings to tvdb4 for card captor sakura and Blood + and that is redundant. I need to fix tvdb3 and remove them.

@ZeroQI - Isn’t that what tvdb4 is for? If those are redundant wouldn’t Bleach be as well? Looking at it, it just maps to the theTVDB seasons the same way as CCS and Blood+ do.

I’ll create a new library and do some testing with tvdb3 for you, and get logs up.

@deusxanime: it’s simple enough

  • tvdb leave series as they are
  • tvdb2 insert season folders and reset ep number at 1 each season
  • tvdb3 insert season folders and keep absolute numbers (for long series, what you want i believe)
  • tvdb4 let user choose whatever folder and keep absolute numbers (for long series with sagas) and i used the tvdb4 mapping file to use arks instead of seasons, or to solve half season mess with thetvdb where it doesn’t sometime make sense. users can make their own season folders too as long as they leave the ep absolute number

mapping file is for mapping anidb to tvdb and exist as local,scudlee mod and scudlee (scudlee mod not supported in scanner yet)
tvdb4mapping file is for choosing seasons for episodes, can be local, or online but i only keep

I created it to have Sagas like dragon ball (namek, freezer) instead of numerous seasons and still have meta working.
There is no point in remapping seasons in tvdb4 since this is what tvdb does best.
Bleach called it Arks so i thought they differed from seasons, but no you are right, will needs removing.
This stems for the fact you can’t have 100+ eps in an android device for scrolling, but year seasons in db/dbz were stupid and sagas like namek or ennes lobby for one piece were fully meaniingfull…

I don’t really use this mode, but wanted to make ass/hama the best out there and i can say the functionalities are unrivaled and allow to make anime libraries bearable for all.

I promise i will fix tvdb3 so you have seasons and metadata and will amend the scanner now to have the mapping file mod handled at scanner level. We did solve loads of bugs pretty quickly so it gets better for everybody.
However this flexibility made it more complex for hama and issues popped up…

I look forward to your logs and making the agent behave as it should

@ZeroQI - Thanks for the explanation. Bleach was the first series I ran into where I wanted to use mappings in that way/style, and after looking at the way it was done in the tvdb4 mapping file I thought that was the way to go. But it turns out Bleach really shouldn’t be that way either, so I was led astray. =)

Downloaded new ASS code and created a new test library with Cardcaptor Sakura and Blood+ and switched them to use tvdb3-xxxx tagging now for testing. So far have not had the problem where episodes are completely missing like was happening before (that was very strange), but neither are downloading metadata/titles it seems. Both just have all episodes labeled generically like ā€œEpisode 1ā€. I did try do a force refresh on them like I have in the past for other series that are slow to grab that, but even after that the episode titles are usually populated within a few seconds. For tvdb3 tagged shows they don’t seem to be pulling down.

It does seem to be recognizing the multi-episode files now and working correctly with them, so that is good news!

Once we get the metadata problem figured out I’ll put them back in my main library and we’ll see if the missing episode issue manifests again.

Attaching ASS/HAMA logs.

@deusxanime tvdb3 fixed for you. show ep title and summary now. Your logs helped, i knew the tables were sound and the key used to store/retrieve and there was a mismatch. added the online mapping file mod to the scanner, wasn’t before (to be tested bu doesn’t crash with no mapping file present)

@ZeroQI - Did you push the update to master? Just updated to lastest master ASS from github, recreated test library with Cardcaptor Sakura and Blood+ using tvdb3-xxxx tagging, and still same problem with no metadata/titles on episodes (expect s0 which does seems to work).

Yes i did. the push should include the code changed:

Here are the scanner log and screenshot. i thing you didn’t update hama. Otherwise need logs

Scanner call - root: '/volume1/downloads/_Plex_Current issues', path: 'Cardcaptor Sakura [tvdb3-70668]', dirs: '0', files: '11'
-------------------------------------------------------------------------------------------------------------------------------------------------------------
folder_show: 'Cardcaptor Sakura [tvdb3-70668]', folder_season: 'None', tvdb mode: '3', tvdb id: '70668'
TVDB season mode (3) enabled, serie url: 'http://thetvdb.com/api/A27AD9BE0DA63333/series/70668/all/en.xml'
abs_manual_placement_worked, abs_manual_placement_info: '[]'
unknown_series_length: False, tvdb_mapping: {1: (1, 1), 2: (1, 2), 3: (1, 3), 4: (1, 4), 5: (1, 5), 6: (1, 6), 7: (1, 7), 8: (1, 8), 9: (1, 9), 10: (1, 10), 11: (1, 11), 12: (1, 12), 13: (1, 13), 14: (1, 14), 15: (1, 15), 16: (1, 16), 17: (1, 17), 18: (1, 18), 19: (1, 19), 20: (1, 20), 21: (1, 21), 22: (1, 22), 23: (1, 23), 24: (1, 24), 25: (1, 25), 26: (1, 26), 27: (1, 27), 28: (1, 28), 29: (1, 29), 30: (1, 30), 31: (1, 31), 32: (1, 32), 33: (1, 33), 34: (1, 34), 35: (1, 35), 36: (2, 36), 37: (2, 37), 38: (2, 38), 39: (2, 39), 40: (2, 40), 41: (2, 41), 42: (2, 42), 43: (2, 43), 44: (2, 44), 45: (2, 45), 46: (2, 46), 47: (3, 47), 48: (3, 48), 49: (3, 49), 50: (3, 50), 51: (3, 51), 52: (3, 52), 53: (3, 53), 54: (3, 54), 55: (3, 55), 56: (3, 56), 57: (3, 57), 58: (3, 58), 59: (3, 59), 60: (3, 60), 61: (3, 61), 62: (3, 62), 63: (3, 63), 64: (3, 64), 65: (3, 65), 66: (3, 66), 67: (3, 67), 68: (3, 68), 69: (3, 69), 70: (3, 70)}
-------------------------------------------------------------------------------------------------------------------------------------------------------------
"Cardcaptor Sakura [tvdb3-70668]" s0001e001 "Cardcaptor Sakura - Ep 01 (BD 960x720) [Coalgirls] [62E1631D].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0001e002 "Cardcaptor Sakura - Ep 02 (BD 960x720) [Coalgirls] [6ACE08AB].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0001e003 "Cardcaptor Sakura - Ep 03 (BD 960x720) [Coalgirls] [A81B915A].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0001e004 "Cardcaptor Sakura - Ep 04 (BD 960x720) [Coalgirls] [F859F5AD].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0001e005 "Cardcaptor Sakura - Ep 05 (BD 960x720) [Coalgirls] [BC4DED7C].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0001e006 "Cardcaptor Sakura - Ep 06 (BD 960x720) [Coalgirls] [F3BAA702].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0003e066 (Orig: s01e66) "Cardcaptor Sakura - Ep 66 (BD 960x720) [Coalgirls] [5F23EE87].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0003e067 (Orig: s01e67) "Cardcaptor Sakura - Ep 67 (BD 960x720) [Coalgirls] [58620343].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0003e068 (Orig: s01e68) "Cardcaptor Sakura - Ep 68 (BD 960x720) [Coalgirls] [5F5B21F1].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0003e069 (Orig: s01e69) "Cardcaptor Sakura - Ep 69 (BD 960x720) [Coalgirls] [9DC4A4FC].mkv"
"Cardcaptor Sakura [tvdb3-70668]" s0003e070 (Orig: s01e70) "Cardcaptor Sakura - Ep 70 (BD 960x720) [Coalgirls] [7DAD7227].mkv"

No you are right, I just updated ASS before. I was wondering if that was it. I’ll update both, try again, and report back.

@ZeroQI - After updating both ASS & HAMA, tvdb3 is working as it should! Also changed Bleach over to tvdb3 and is working as well.

I’ve gotten through my A-F folder. =) 1/4 of the way there…

A few minor things I’ve come across as I’ve been building. These may be bugs, so thought worth bringing up in the effort to get them squashed.

  1. For some reason ā€œBlack Jack (2004)ā€ is not pulling down the metadata/titles for s1. The specials/s0 is getting them though. This is a combination of 2 folders - ā€œBlack Jack - The Two Doctors of Darkness [anidb2-3506]ā€ and ā€œBlack Jack TV [anidb2-2389]ā€ using anidb2 tagging/mapping obviously. So it should be grabbing the info from TVDB in a pretty straightforward manner, but for some reason I can’t get it to work.

2) The series ā€œBlack Jack Special - The 4 Miracles of Life [anidb-2672]ā€ maps all episodes to s1e04, I’m sure due to the number ā€œ4ā€ in the title. I can change it to ā€œFourā€ in the folder and file names and then it works as expected.

…Actually this one seems to be working now, will try to move it back to my primary library and see if it works in there again…

  1. For ā€œClamp in Wonderland [anidb-1550]ā€ I have the ED in the specials folder, but for some reason it is mapping that Ending to s1e01 along with the actual primary episode. I have a very similar episodes structure/files with ā€œClamp in Wonderland 2 1995-2006 [anidb-4514]ā€ (one primary episode and the ED in the specials folder), and it works just fine.

  1. Last one is the Code Geass specials. It looks like it thinks many of them are double episodes from looking at the scanner log and so I’m getting a bunch of duplicates. I’m guessing this is due to the numbers in the episode names which are in the file name.

I created a new folder and moved this problem series into it, then created a new library with it as root. Attaching ASS/HAMA logs for its creation.

I think many of these errors are based on naming. Let me know if trying to fix anything would cause more things to break than it is worth. I can be flexible and change some of the file names if needed.

@deusxanime
1)agent issue due to ep 0 presence in tvdb xml. updated github master
2) yeah fixed it the other day
3) revamped the engine, works and we should get fewer scanner mishaps. updated github master
4) yeah all regex showed multiple due to ep2 being set. updated github master