The TVDB Metadata failing - Python2 openssl problem with Cloudflare?

General Discussions
1

I run my plex server on a Ubuntu 16.04 box in a data center. A few weeks ago I noticed that a lot of my TV shows stopped getting artwork or metadata.

I tried clearing the plugin cache directories without success. I upgraded the server.

I noticed in the logs that all the failing requests were to https://thetvdb.com and remembered that Plex was supposed to be using a TVDB proxy/mirror.

I nuked the TV show collection and recreated it to let it rebuild. Now it’s attempting to use the proxy/mirrors that Plex operates

I am now getting SOME metadata, but only from tvdb2.plex.tv and meta.plex.tv. Connections to thetvdb.com (mostly for artwork or banners or stuff the mirror hasn’t cached yet) are still failing.

2017-07-31 21:50:05,939 (7f6a677fe700) :  DEBUG (networking:166) - Requesting 'https://thetvdb.com/banners/_cache/fanart/original/212171-11.jpg'
2017-07-31 21:50:07,811 (7f6a47fff700) :  INFO (__init__:1059) - <urlopen error ('_ssl.c:574: The handshake operation timed out',)>

My OpenSSL info

OpenSSL 1.1.0f  25 May 2017
built on: reproducible build, date unspecified
platform: debian-amd64
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/lib/ssl\"" -DENGINESDIR="\"/usr/lib/x86_64-linux-gnu/engines-1.1\"" 
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"

I even ran lynx from the server to connect to https://thetvdb.com and it was successful. I wanted to rule out being blocked by Cloudflare for some weird reason.

Plex uses Python for it’s scanners, so maybe it’s something in there?

Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2k  26 Jan 2017'
>>>

I would open a Python2 console and type in something to basically try to curl the homepage of thetvdb to see if it can make the SSL connection, but I don’t know what ways the agents use Python to grab the data.

I’m not sure what to try next. Any suggestions?

2

Oh, forgot to mention that I also checked the connection with the version of SSL on my server, so I’m pretty sure it’s not some weird firewall issue. The lynks command line test was valid, but this is a bit of a clearer indication that it’s not my server or cloudflare, but something with Plex’s way of talking to Cloudflare.

It seems to be working for all the other metadata repositories, but maybe TheTVDB is the only one behind Cloudflare… and Cloudflare’s SSL configuration doesn’t work with ancient SSL key configurations (in the diagnostic below, openssl negotiated the modern TLSv1.2 ECDHE-ECDSA-CHACHA20-POLY1305 Cipher.

tvdb2.plex.tv uses ECDHE-RSA-AES128-GCM-SHA256 which is a bit more standard.

Does Plex use its own bundled Python implementation? Is there a way to test my phython2 installations ability to connect to SSL sites? Maybe something python related is failing with the newer SSL cyphers?

I appear to be using TheTVDB.bundle version 42bd8c6 (Wed Jul 26 01:01:43 UTC 2017)

openssl s_client -connect thetvdb.com:443
CONNECTED(00000003)
[ snip to be verbose ]
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3400 bytes and written 261 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-CHACHA20-POLY1305
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-CHACHA20-POLY1305
    Session-ID: 544015475DFC6A688665A7AEA461863217A7E73E939E24E258980702807A5B1E
    Session-ID-ctx: 
    Master-Key: 88ACB00FFF76D504FDF5B8206EA33484FB7697C534BCF8BB8ED49984476418DACF2E2F769619C936CAD0BB70880FE4AD
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - c5 4f 39 5d 58 38 67 84-6c 75 58 33 72 71 d9 8d   .O9]X8g.luX3rq..
    0010 - fe fa 47 11 f3 b6 18 86-2c 04 1f e2 0c 34 4c 73   ..G.....,....4Ls
    0020 - fe 18 7c 3f fe 2f 46 76-ff 1f 97 85 2c 30 1c 44   ..|?./Fv....,0.D
    0030 - 4f c1 64 10 ce 2d c2 ea-7d b7 81 e8 2e d7 60 53   O.d..-..}.....`S
    0040 - c3 58 79 2e d2 80 d8 3d-71 e2 d6 28 b4 33 60 b7   .Xy....=q..(.3`.
    0050 - f6 4a fe 9d c7 70 9f 0e-ba 8b 5f ba 7f 91 c6 f3   .J...p...._.....
    0060 - 8e 56 09 82 06 0c 19 e5-83 e7 23 74 65 63 0b 49   .V........#tec.I
    0070 - 9e 94 3c 8f cd 43 d7 7f-1d ac c7 5a 47 03 9e 7c   ..<..C.....ZG..|
    0080 - a7 2f 44 9f 6b 9c 5a 3a-9f 01 12 31 ff ea 62 4c   ./D.k.Z:...1..bL
    0090 - d9 73 d2 f2 c4 4a db 63-cb 7a 63 ce 1e e2 e3 8c   .s...J.c.zc.....
    00a0 - 96 3e e7 5d 8c b6 8a 11-12 a5 f8 57 84 98 d2 69   .>.].......W...i
    00b0 - c3 00 65 97 57 d9 f5 b2-3a 4d 65 1d bf 1a 3f 1b   ..e.W...:Me...?.

    Start Time: 1501538782
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
3

A few things of note…

I just tried modifying /usr/lib/plexmediaserver/Resources/Plug-ins-42bd8c63f/TheTVDB.bundle/Contents/Code/__init__.py so TVDB_IMG_ROOT uses TVDB_V2_PROXY_SITE instead of TVDB_BASE_URL

I figured that right now the scanner is pulling everything from the proxy site except the images so if I could get it to get the images from there too, that would be awesome.

Didn’t work… the proxy site doesn’t cache the images… but this time when I refreshed the metadata, the show actually has metadata (just no images)

Before I could match the show and it would try to get the metadata but the SSL handshake failure while getting the images caused the entire metadata refresh to fail.

Looks like the Parallel task ‘DownloadImages’ doesn’t fail gracefully with these SSL handshake errors that I’m getting.

I also tried updating the plexmediaserver/Resources/cacert.pem file with a new one from https://curl.haxx.se/docs/caextract.html without any success. I didn’t think it would matter much, but your bundle of root CA certificates is over 6 years old.

I’ll keep trying to diagnose until I get some suggestions… :smile:

4

I think the whole https error thing might be a red herring…

I changed the TVDB_BASE_URL from https to http and tried to update some metadata… Still getting errors

2017-08-01 02:59:04,743 (7f79837fe700) :  DEBUG (networking:166) - Requesting 'http://thetvdb.com/banners/_cache/seasons/303438-2.jpg'
2017-08-01 02:59:04,754 (7f799a7fc700) :  DEBUG (networking:166) - Requesting 'http://thetvdb.com/banners/_cache/posters/303438-4.jpg'
2017-08-01 02:59:04,759 (7f7982ffd700) :  DEBUG (networking:166) - Requesting 'http://thetvdb.com/banners/_cache/fanart/original/303438-6.jpg'
2017-08-01 02:59:04,763 (7f7983fff700) :  DEBUG (networking:166) - Requesting 'http://thetvdb.com/banners/_cache/fanart/original/303438-8.jpg'
2017-08-01 03:00:04,784 (7f79837fe700) :  INFO (__init__:1071) - <urlopen error timed out>
2017-08-01 03:00:04,788 (7f79837fe700) :  DEBUG (networking:166) - Requesting 'http://thetvdb.com/banners/_cache/fanart/original/303438-3.jpg'
2017-08-01 03:00:04,831 (7f799a7fc700) :  INFO (__init__:1071) - <urlopen error timed out>
2017-08-01 03:00:04,832 (7f7982ffd700) :  INFO (__init__:1070) - <urlopen error timed out>
2017-08-01 03:00:04,836 (7f7983fff700) :  INFO (__init__:1070) - <urlopen error timed out>

And once again, to rule network connectivity problems, I verified one of the files manually

root@plexhost ~# wget --user-agent="Plex/Nine" -S http://thetvdb.com/banners/_cache/fanart/original/303438-8.jpg
--2017-08-01 03:04:40--  http://thetvdb.com/banners/_cache/fanart/original/303438-8.jpg
Resolving thetvdb.com (thetvdb.com)... 2400:cb00:2048:1::6810:e40e, 2400:cb00:2048:1::6810:e30e, 2400:cb00:2048:1::6810:e50e, ...
Connecting to thetvdb.com (thetvdb.com)|2400:cb00:2048:1::6810:e40e|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 01 Aug 2017 03:04:40 GMT
  Content-Type: image/jpeg
  Content-Length: 11022
  Connection: keep-alive
  Set-Cookie: __cfduid=d242e9804503afef83d67aa16faee3e571501556680; expires=Wed, 01-Aug-18 03:04:40 GMT; path=/; domain=.thetvdb.com; HttpOnly
  Last-Modified: Sun, 30 Apr 2017 02:58:31 GMT
  Expires: Tue, 15 Aug 2017 03:04:40 GMT
  Cache-Control: public, max-age=1209600
  CF-Cache-Status: HIT
  Vary: Accept-Encoding
  Accept-Ranges: bytes
  Server: cloudflare-nginx
  CF-RAY: 38759247c69f57c5-IAD
Length: 11022 (11K) [image/jpeg]
Saving to: ‘303438-8.jpg’

303438-8.jpg                                       100%[=============================================================================================================>]  10.76K  --.-KB/s    in 0s      

2017-08-01 03:04:40 (161 MB/s) - ‘303438-8.jpg’ saved [11022/11022]
5

I might be onto something. The previous request was successful, but I noticed that it went over IPv6. I checked again but with IPv4

root@plexhost /u/l/p/R/P/T/Contents# wget --user-agent="Plex/Nine" --inet4-only --timeout=10 --tries=1 -S http://thetvdb.com
--2017-08-01 03:23:28--  http://thetvdb.com/
Resolving thetvdb.com (thetvdb.com)... 104.16.231.14, 104.16.230.14, 104.16.227.14, ...
Connecting to thetvdb.com (thetvdb.com)|104.16.231.14|:80... connected.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.

root@plexhost /u/l/p/R/P/T/Contents# wget --user-agent="Plex/Nine" --inet4-only --timeout=10 --tries=1 -S https://thetvdb.com
--2017-08-01 03:23:44--  https://thetvdb.com/
Resolving thetvdb.com (thetvdb.com)... 104.16.231.14, 104.16.230.14, 104.16.227.14, ...
Connecting to thetvdb.com (thetvdb.com)|104.16.231.14|:443... connected.
Unable to establish SSL connection.

root@atl6 /u/l/p/R/P/T/Contents# wget --user-agent="Plex/Nine" --inet6-only --timeout=10 --tries=1 -S https://thetvdb.com
--2017-08-01 03:30:57--  https://thetvdb.com/
Resolving thetvdb.com (thetvdb.com)... 2400:cb00:2048:1::6810:e30e, 2400:cb00:2048:1::6810:e70e, 2400:cb00:2048:1::6810:e60e, ...
Connecting to thetvdb.com (thetvdb.com)|2400:cb00:2048:1::6810:e30e|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Tue, 01 Aug 2017 03:30:57 GMT
  Content-Type: text/html
  Transfer-Encoding: chunked
  Connection: keep-alive
  Set-Cookie: __cfduid=d95ada420c74f9d456e63ac5cfcaef3531501558257; expires=Wed, 01-Aug-18 03:30:57 GMT; path=/; domain=.thetvdb.com; HttpOnly
  X-Powered-By: PHP/5.3.10-1ubuntu3.15
  Set-Cookie: PHPSESSID=gt93o8mj59p186dv6l94e0ji57; path=/
  Expires: Thu, 19 Nov 1981 08:52:00 GMT
  Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
  Pragma: no-cache
  Vary: User-Agent,Accept-Encoding
  Server: cloudflare-nginx
  CF-RAY: 3875b8c55f75575f-IAD
Length: unspecified [text/html]
Saving to: ‘index.html’

index.html                                             [ <=>                                                                                                          ]  56.84K  --.-KB/s    in 0.08s   

2017-08-01 03:30:57 (687 KB/s) - ‘index.html’ saved [58202]

It appears that TheTVDB has my IPv4 address blocked or something else is wrong.

It’s odd because I have Plex configured to support IPv6 networking. Maybe that only works for incoming connections?

6

I can ping and traceroute just fine

root@plexhost /u/l/p/R/Plug-ins-42bd8c63f# ping thetvdb.com
PING thetvdb.com (104.16.231.14) 56(84) bytes of data.
64 bytes from 104.16.231.14: icmp_seq=1 ttl=58 time=15.9 ms
64 bytes from 104.16.231.14: icmp_seq=2 ttl=58 time=15.9 ms
64 bytes from 104.16.231.14: icmp_seq=3 ttl=58 time=15.9 ms
64 bytes from 104.16.231.14: icmp_seq=4 ttl=58 time=15.9 ms

root@plexhost /u/l/p/R/Plug-ins-42bd8c63f# traceroute thetvdb.com
traceroute to thetvdb.com (104.16.227.14), 64 hops max
  1   10.40.170.254  0.353ms  0.246ms  0.238ms 
  2   40.135.62.200  2.148ms  2.137ms  2.129ms 
  3   169.130.170.174  2.381ms  2.309ms  2.318ms 
  4   169.130.169.14  2.465ms  2.343ms  2.342ms 
  5   40.132.59.34  19.845ms  16.156ms  16.138ms 
  6   40.128.10.172  16.995ms  16.972ms  16.982ms 
  7   206.126.237.30  31.402ms  15.795ms  15.798ms 
  8   104.16.227.14  15.788ms  15.787ms  15.773ms

I just can’t retrieve any data. Not even headers are returned as a response.

7

I am no expert in networkig matters, but I was told that Cloudflare doesn’t support IPv6 (currently).
Disable all IPv6 support in Plex and try again.
Settings - Server - Network - ‘Show Advanced’

Restart Plex Server for the change to become active.

8

Cloudflare does support IPv6… but I went ahead and disabled IPv6 in plex anyway. Also disabled it on the server for good measure.

Still doesn’t work.

I’m kind of baffled.