I’m willing to help troubleshoot. This is still a problem for me. I’ve been following since 6/6 when I first ran into this problem. Apologies in advance for the complicated setup and verbosity!
Versions of plex attempted:
- Works OK:
1.22.3.4392-d7c624def
- Works OK:
1.22.3.4523-d0ce30438
- DNS broken:
1.23.3.4707-ebb5fe9f3
- DNS broken:
1.23.4.4805-186bae04e
Environment:
- plex is running in k8s, single pod (replica=1), image is
plexinc/pms-docker
- k8s 1.20 on intel nuc (recently upgraded from 1.19, had the same plex dns issues on 1.19)
- k8s is dual-stack ipv4 and ipv6 via flannel
- non-k8s network is not dual-stack (ipv4 only)
- ISP is not dual-stack (ipv4 only)
- AdGuard DNS (no DHCP; enabled: 53/udp, 53/tcp, DoH, DoT, DoQ), with DoH quad 9 upstream, k8s internal is 53/udp only
- all hosts are Ubuntu Focal 20.04.2 (
focal-server-cloudimg)
- i do not run EERO devices, i run ubiquiti gear (USG-3P, USW-24-PoE, etc.)
- DNS path for k8s nodes: plex (source,
10.42.6.171) > k8s coredns (cache, 53/udp, 10.43.0.10) > USG-3P (cache, 53/udp, 192.168.68.1) > AdGuard (cache, DoH, 192.168.68.104) > quad-9 (public destination, https://dns10.quad9.net/dns-query)
- attached the deployment/network/storage spec i run (parts are redacted):
k8s-plex.yaml.txt (6.4 KB)
Now for some diagnostic info, :fingers-crossed: private stuff is redacted… -----BEGIN WALL OF TEXT-----
# plex version 1.23.4.4805
ᐅ kubectl get pods -n plex -o go-template='{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}'
plexinc/pms-docker:1.23.4.4805-186bae04e
# k8s version
ᐅ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T13:00:45Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.8", GitCommit:"5575935422cc1cf5169dfc8847cb587aa47bac5a", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:07Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"linux/arm64"}
# adguard version
ᐅ kubectl get pods -n home-dns -o go-template='{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}'
adguard/adguardhome:v0.106.3 adguard/adguardhome:v0.106.3
# date
Sun Jul 18 22:01:37 UTC 2021
# grep -iC2 'warn\|error' Plex\ Media\ Server.log
# ... truncated
--
Jul 18, 2021 21:53:02.596 [0x7f039d871b38] DEBUG - MyPlex: Updating device connections (from timer: 1)
Jul 18, 2021 21:53:02.596 [0x7f039d871b38] DEBUG - HTTP requesting PUT https://plex.tv/devices/xxxxSOME_SHA1xxxx?Connection[][uri]=https://xxxx.dev:32400&httpsEnabled=1&httpsRequired=0&dnsRebindingProtection=1&natLoopbackSupported=0&X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Jul 18, 2021 21:53:02.604 [0x7f039d871b38] ERROR - Error issuing curl_easy_perform(handle): 6
Jul 18, 2021 21:53:02.604 [0x7f039d871b38] WARN - HTTP error requesting PUT https://plex.tv/devices/xxxxSOME_SHA1xxxx?Connection[][uri]=https://xxxxx.dev:32400&httpsEnabled=1&httpsRequired=0&dnsRebindingProtection=1&natLoopbackSupported=0&X-Plex-Token=xxxxxxxxxxxxxxxxxxxx (6, Couldn't resolve host name) (Could not resolve host: plex.tv)
Jul 18, 2021 21:53:02.604 [0x7f039d871b38] WARN - MyPlex: Updating device connections failed, retrying in 2560 seconds.
Jul 18, 2021 21:53:04.833 [0x7f039da0eb38] DEBUG - Request: [10.42.1.1:34674 (Subnet)] GET /identity (7 live) Signed-in
Jul 18, 2021 21:53:04.833 [0x7f039ddcdb38] DEBUG - Completed: [10.42.1.1:34674] 200 GET /identity (7 live) 0ms 369 bytes
--
Jul 18, 2021 21:55:54.833 [0x7f039ddf0b38] DEBUG - Completed: [10.42.1.1:34828] 200 GET /identity (7 live) 0ms 369 bytes
Jul 18, 2021 21:56:00.561 [0x7f039c56ab38] DEBUG - [MediaProviderManager] HTTP requesting GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Jul 18, 2021 21:56:00.569 [0x7f039c56ab38] ERROR - [MediaProviderManager] Error issuing curl_easy_perform(handle): 6
Jul 18, 2021 21:56:00.569 [0x7f039c56ab38] WARN - [MediaProviderManager] HTTP error requesting GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx (6, Couldn't resolve host name) (Could not resolve host: plex.tv)
Jul 18, 2021 21:56:00.569 [0x7f039c56ab38] ERROR - [MediaProviderManager] Error parsing content.
Jul 18, 2021 21:56:00.569 [0x7f039c56ab38] ERROR - [MediaProviderManager] Error parsing XML: Error parsing file.
Jul 18, 2021 21:56:02.270 [0x7f039da0eb38] DEBUG - Request: [10.42.1.1:34838 (Subnet)] GET /identity (7 live) Signed-in
Jul 18, 2021 21:56:02.271 [0x7f039ddf0b38] DEBUG - Completed: [10.42.1.1:34838] 200 GET /identity (7 live) 0ms 369 bytes
--
Jul 18, 2021 22:00:54.835 [0x7f039ddcdb38] DEBUG - Completed: [10.42.1.1:35074] 200 GET /identity (7 live) 0ms 369 bytes
Jul 18, 2021 22:01:00.571 [0x7f039d871b38] DEBUG - [MediaProviderManager] HTTP requesting GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Jul 18, 2021 22:01:00.580 [0x7f039d871b38] ERROR - [MediaProviderManager] Error issuing curl_easy_perform(handle): 6
Jul 18, 2021 22:01:00.580 [0x7f039d871b38] WARN - [MediaProviderManager] HTTP error requesting GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx (6, Couldn't resolve host name) (Could not resolve host: plex.tv)
Jul 18, 2021 22:01:00.580 [0x7f039d871b38] ERROR - [MediaProviderManager] Error parsing content.
Jul 18, 2021 22:01:00.580 [0x7f039d871b38] ERROR - [MediaProviderManager] Error parsing XML: Error parsing file.
Jul 18, 2021 22:01:02.269 [0x7f039da0eb38] DEBUG - Request: [10.42.1.1:35084 (Subnet)] GET /identity (7 live) Signed-in
Jul 18, 2021 22:01:02.269 [0x7f039ddcdb38] DEBUG - Completed: [10.42.1.1:35084] 200 GET /identity (7 live) 0ms 369 bytes
# cat /etc/resolv.conf
search plex.svc.cluster.local svc.cluster.local cluster.local xxxxx.dev
nameserver 10.43.0.10
options ndots:5
# cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: files
group: files
shadow: files
gshadow: files
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
ᐅ kgs -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.43.0.10 <none> 53/UDP,53/TCP,9153/TCP x
...
ᐅ kds -n kube-system kube-dns
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: prometheus.io/port: 9153
prometheus.io/scrape: true
Selector: k8s-app=kube-dns
Type: ClusterIP
IP Families: <none>
IP: 10.43.0.10
IPs: 10.43.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 10.42.2.33:53,10.42.5.19:53
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 10.42.2.33:53,10.42.5.19:53
...
ᐅ kgp -n kube-system -lk8s-app=kube-dns -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-74ff55c5b-f5nc7 1/1 Running 0 12d 10.42.2.33 x <none> <none>
coredns-74ff55c5b-tpjmv 1/1 Running 0 12d 10.42.5.19 x <none> <none>
# dig plex.tv A @10.43.0.10
; <<>> DiG 9.16.1-Ubuntu <<>> plex.tv A @10.43.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56660
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;plex.tv. IN A
;; ANSWER SECTION:
plex.tv. 15 IN A 99.81.164.127
plex.tv. 15 IN A 99.81.153.144
plex.tv. 15 IN A 108.128.10.254
;; Query time: 32 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Sun Jul 18 22:08:34 UTC 2021
;; MSG SIZE rcvd: 105
# dig plex.tv AAAA @10.43.0.10
; <<>> DiG 9.16.1-Ubuntu <<>> plex.tv AAAA @10.43.0.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3736
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;plex.tv. IN AAAA
;; Query time: 4 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Sun Jul 18 22:08:51 UTC 2021
;; MSG SIZE rcvd: 36
# NOTE: same result in both container and host
# dig plex.tv AAAA
; <<>> DiG 9.16.1-Ubuntu <<>> plex.tv AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15251
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;plex.tv. IN AAAA
;; Query time: 4 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Sun Jul 18 22:09:47 UTC 2021
;; MSG SIZE rcvd: 36
# NOTE: same result in both container and host
# dig plex.tv A
; <<>> DiG 9.16.1-Ubuntu <<>> plex.tv A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42323
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;plex.tv. IN A
;; ANSWER SECTION:
plex.tv. 30 IN A 108.128.10.254
plex.tv. 30 IN A 99.81.153.144
plex.tv. 30 IN A 99.81.164.127
;; Query time: 36 msec
;; SERVER: 10.43.0.10#53(10.43.0.10)
;; WHEN: Sun Jul 18 22:10:11 UTC 2021
;; MSG SIZE rcvd: 105
# NOTE: same result in both container and host
# curl --ipv4 https://plex.tv
<html><body>You are being <a href="https://www.plex.tv/">redirected</a>.</body></html>
# NOTE: same result in both container and host
# curl --ipv6 https://plex.tv
curl: (6) Could not resolve host: plex.tv
# NOTE: same result on both container and (roughly, in veth form) host
# cat /sys/module/ipv6/parameters/disable
0
# sysctl -a 2>/dev/null | grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.lo.disable_ipv6 = 0
# NOTE: inside the container
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if71: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
link/ether ee:71:0b:a4:8b:36 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.42.6.171/24 brd 10.42.6.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::ec71:bff:fea4:8b36/64 scope link
valid_lft forever preferred_lft forever
Proof Adguard is getting these queries (formatted to avoid screenshots):
14:50:19 7/18/2021 plex.tv Type: A, Plain DNS Processed 26 ms 192.168.0.0
Response details
Status: Processed
DNS server: https://dns10.quad9.net:443/dns-query
Elapsed: 26 ms
Response code: NOERROR
Response:
A: 99.81.153.144 (ttl=20)
A: 99.81.164.127 (ttl=20)
A: 108.128.10.254 (ttl=20)
14:50:17 7/18/2021 plex.tv Type: AAAA, Plain DNS Processed 26 ms 192.168.0.0
Response details
Status: Processed
DNS server: https://dns10.quad9.net:443/dns-query
Elapsed: 26 ms
Response code: NOERROR
More DNS info from the host perspective (snippets):
# resolvectl status
# ...truncated
Link 8 (vethe7350ab5)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 5 (cni0)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 4 (flannel.1)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 2 (eno1)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.68.1
DNS Servers: 192.168.68.1
DNS Domain: xxxxx.dev
I downgraded to 1.22.3, and now things are working just fine, e.g.:
# grep -iC2 'plex.tv' /config/Library/Application\ Support/Plex\ Media\ Server/Logs/Plex\ Media\ Server.log
Jul 18, 2021 22:33:55.143 [0x7f1ed77fe700] DEBUG - MyPlex: Updating device connections (from timer: 0)
Jul 18, 2021 22:33:55.143 [0x7f1ed77fe700] DEBUG - HTTP requesting PUT https://plex.tv/devices/xxxxSOME_SHA1xxxx?Connection[][uri]=https://xxxxx.dev:32400&Connection[][uri]=http://x.x.x.x:32400&httpsEnabled=1&httpsRequired=0&dnsRebindingProtection=1&natLoopbackSupported=0&X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Jul 18, 2021 22:33:55.342 [0x7f1edd0ec700] DEBUG - EventSource: Got event [data] '<Message address="x.x.x.x" port="32400" asyncIdentifier="xxxxx-SOME-UUID-xxxxx" connectivity="0" command="notifyConnectivity"/>'
Jul 18, 2021 22:33:55.342 [0x7f1edd0ec700] DEBUG - PubSub: Got notified of reachability for async identifier xxxxx-SOME-UUID-xxxxx: 0 for x.x.x.x:32400 (responded in 199 ms)
Jul 18, 2021 22:33:55.342 [0x7f1edd0ec700] DEBUG - MyPlex: reachability check - current mapping state: 'Mapped - Publishing'.
