Name resolution issues causing inability to claim server - Arch Linux

Plex Media Server Desktops & Laptops
1

Server Version#: 1.23.2.4656-85f0adf5b
Player Version#: N/A
OS: Arch Linux 2021.06
Firewall: iptables, no rules at all
Headless server local IP: 192.168.1.69
Testing PC local IP: 192.168.1.173

Behavior: Either I am told from logging in with correct credentials on http:192.168.1.69:32400/web that “An error occurred during login” and rejected, or if I am allowed in “This server is unclaimed and not secure” on the general settings page and when I click “Claim Server”, nothing changes, with the below log entries being directly caused by me clicking “Claim Server”.

Jun 12, 2021 16:48:32.849 [0x7f8f71965b38] DEBUG - Request: [192.168.1.173:50928 (Subnet)] POST /myplex/claim?token=xxxxxxxxxxxxxxxxxxxxDugg4f (4 live) GZIP
Jun 12, 2021 16:48:32.850 [0x7f8f71965b38] DEBUG - HTTP requesting POST https://plex.tv/api/claim/exchange?token=xxxxxxxxxxxxxxxxxxxxDugg4f
Jun 12, 2021 16:48:37.855 [0x7f8f71965b38] ERROR - Error issuing curl_easy_perform(handle): 6
Jun 12, 2021 16:48:37.855 [0x7f8f71965b38] WARN - HTTP error requesting POST https://plex.tv/api/claim/exchange?token=xxxxxxxxxxxxxxxxxxxxDugg4f (6, Couldn't resolve host name) (Could not resolve host: plex.tv)
1 Like
2

Can you verify the DNS configured on your system?
Seems your current setup fails to find plex.tv

3

Thank you for answering quickly. I agree it seems there is a DNS issue. I am using systemd-resolved for DNS resolution, and not generally having issues. I am able to ping www.plex.tv and get responses, but not plex.tv.

$ ping www.plex.tv
PING www.plex.tv (104.18.12.47) 56(84) bytes of data.
64 bytes from 104.18.12.47 (104.18.12.47): icmp_seq=1 ttl=59 time=5.74 ms
64 bytes from 104.18.12.47 (104.18.12.47): icmp_seq=2 ttl=59 time=5.00 ms
64 bytes from 104.18.12.47 (104.18.12.47): icmp_seq=3 ttl=59 time=6.32 ms
64 bytes from 104.18.12.47 (104.18.12.47): icmp_seq=4 ttl=59 time=5.29 ms
64 bytes from 104.18.12.47 (104.18.12.47): icmp_seq=5 ttl=59 time=5.77 ms
^C
--- www.plex.tv ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 5.001/5.623/6.317/0.451 ms

$ ping plex.tv
PING plex.tv (99.81.164.127) 56(84) bytes of data.
^C
--- plex.tv ping statistics ---
42 packets transmitted, 0 received, 100% packet loss, time 41554ms
4

plex.tv has ping disabled.

5

My solution (fixed dig and plex name resolution issues):

Comment out all lines of /etc/systemd/resolved.conf.
Make /etc/resolv.conf a symlink to /run/systemd/resolved/resolv.conf

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

where /run/systemd/resolve/resolv.conf contains the contents below.

# Resolver configuration file.
# See resolv.conf(5) for details.
nameserver 1.1.1.1 8.8.8.8

This solution was not necessary before v1.23.

$ /usr/lib/plexmediaserver/Plex\ Media\ Server --version
v1.23.2.4656-85f0adf5b

Failed attempted solutions:
This forum post reports the same symptoms, and gave me many ideas for solutions.

  • Adding Linux user plex to systemd-resolve and systemd-network Linux user groups
  • Setting Domains=plex.tv in /etc/systemd/resolved.conf
  • Booting with kernel parameter ipv6.disable=1 to see if this was IPv6 related (it wasn’t).
  • Setting ISP-provided DNS servers as my DNS= parameter in /etc/systemd/resolved.conf instead of Cloudflare (1.1.1.1) or Google (8.8.8.8)
  • Trying enabling and disabling DNS settings like LLMNR, DNSSEC, and ResolveUnicastSingleLabel in /etc/systemd/resolved.conf

Notes:
While I would not describe my Arch Linux system as having global DNS issues, (curl and ping work), the post linked at the top of this reply showed me the value of dig as another DNS resolution testing tool that can reveal extra DNS issues (comes in the bind-tools Arch Linux package). Before implementing the fix at the bottom, dig never returned anything for any domain unless I provided a DNS for it to use on the command line, which told me it did not know where it could find a list of DNS servers to use.

dig plex.tv AAAA
; <<>> DiG 9.16.16 <<>> plex.tv AAAA
;; global options: +cmd
;; connection timed out; no servers could be reached

and

dig @1.1.1.1 plex.tv AAAA
; <<>> DiG 9.16.16 <<>> @1.1.1.1 plex.tv AAAA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11060
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;plex.tv.                       IN      AAAA

;; AUTHORITY SECTION:
plex.tv.                2486    IN      SOA     jeremy.ns.cloudflare.com. dns.cloudflare.com. 2037520463 10000 2400 604800 3600

;; Query time: 6 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Jun 13 15:23:37 EDT 2021
;; MSG SIZE  rcvd: 100

so it became apparent that some apps on Arch Linux (and maybe linux in general, I don’t know) can check in different places for which DNS servers to use (some check /etc/systemd/resolved.conf, others (plex and dig) seem to only check /etc/resolv.conf).

6

On your system, is /etc/resolv.conf a symlink or a normal file?

What is the output of resolvectl status?

7

It is a regular file that apparently, even when the “nameserver” line is nonexistent, takes some amount of precedence over /etc/systemd/resolved.conf, which is also a normal file.

$ ls -la /etc/resolv.conf
 -rw-r--r--  1 root root     101 Jun 13 15:32 resolv.conf

$ ls -la /etc/systemd/resolved.conf
 -rw-r--r-- 1 root root 1401 Jun 13 15:35 /etc/systemd/resolved.conf

Example of a symlink on my system:

$ ls -la /etc/localtime
lrwxrwxrwx  1 root root      30 Jun 12 15:52 localtime -> /usr/share/zoneinfo/US/Eastern

I have been using resolvectl status for this work. You should know before reading this that I am using a bridge device (br0) for giving internet to a VM I sometimes run, and Arch Linux itself uses device (eno1).

By unimplementing the fix to /etc/resolv.conf and confirming it broke my Plex Server’s DNS capabilites, resolvectl status printed out:

$ sudo resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 9.9.9.10 8.8.8.8 1.0.0.1 8.8.4.4 2001:4860:4860::8888 2606:4700:4700::1111 2606:4700:4700::1001

Link 2 (eno1)
Current Scopes: LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (br0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

and now after populating /etc/resolv.conf's nameserver argument and showing it fixed all Plex Media Server issues with resolving “plex.tv”, it printed:

$ sudo resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 9.9.9.10 8.8.8.8 1.0.0.1 8.8.4.4 2001:4860:4860::8888 2606:4700:4700::1111 2606:4700:4700::1001

Link 2 (eno1)
Current Scopes: LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (br0)
Current Scopes: LLMNR/IPv4 LLMNR/IPv6
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

These outputs seem identical, and like I mentioned before, I already had DNS capabilities using ping and curl before I fixed my /etc/resolv.conf file, so that isn’t surprising. However, you have just drawn my attention to the fact that I re-enabled IPv6 on this system, so I rebooted with kernel argument ipv6.disable=1, which did not help me previously but does not break my fix either. resolvectl status on IPv4-only gave:

$ sudo resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 9.9.9.10 8.8.8.8 1.0.0.1 8.8.4.4 2001:4860:4860::8888 2606:4700:4700::1111 2606:4700:4700::1001

Link 2 (eno1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (br0)
Current Scopes: LLMNR/IPv4
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
8

That absolutely agrees with your comment that different portions of your system are looking at different resolver configurations.

Usually on a systemd system, /etc/resolv.conf is a symlink, and both systemd-aware and “normal” /etc/resolv.conf-based utilities (such as Plex) get the same info.

9

Interesting, I found some people discussing PopOS who mentioned /etc/resolv.conf is meant to be a link to /run/systemd/resolve/resolv.conf, so I moved the contents of /etc/resolv.conf to /run/systemd/resolve/resolv.conf and turned it into a symlink

$ sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

Now I get

$ sudo resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: uplink
  Current DNS Server: 1.1.1.1
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 9.9.9.10 8.8.8.8 1.0.0.1 8.8.4.4 2001:4860:4860::8888 2606:4700:4700::1111 2606:4700:4700::1001

Link 2 (eno1)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (br0)
Current Scopes: LLMNR/IPv4
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Thanks for the tip!

1 Like
10

Let’s ask the big question.

What is the real issue?

  1. Is it preferring IPv6 over IPv4 when it shouldn’t be ?
    a. IPv4 LAN - WAN configuration with no IPv6 present?
    b. Incorrect handling of dual stack configurations?

  2. How it’s communicating with the resolver (whatever it might be) and not handling it correctly in certain cases ?
    a. If so, Which cases does it work
    b. and which does it fail?

If we can bound the problem and define a solution then we can get there.

11

Good questions.

a. As I tried above, booting with ipv6.disable=1 had no effect on the logged behavior both before and after @Volts and I found a fix based on some of the finer details of configuring systemd-resolved.

b. As in 1a, IPv4-only mode had identical behavior as my normal dual stack configuration before and after the fix.

Definitely the right way to go. As I have written above, plex ignores configuration changes when they are instituted in /etc/systemd/resolve/resolv.conf, so in my solution, that file is entirely commented out.

a. Plex succeeds at resolving plex.tv when I add the line nameserver 1.1.1.1 8.8.8.8 9.9.9.10 to /etc/resolv.conf regardless of whether /etc/resolv.conf is a symlink to a file with that line or a normal file itself.

b. Plex fails to resolve the name plex.tv when the line is missing from /etc/resolv.conf or the file its symlink points to.

How can I help to further diagnose communications between plexmediaserver.service and systemd-resolved?

12

  1. Plex doesn’t look at any network configuration files beyond finding the default adapter and gateway address. (the routing table)

  2. Plex does rely on your system’s TCP/IP stack, resolver, and routing tables to get traffic out. It literally does the
    a. Get the IP of “blah.plex.tv”
    b. Open a connection to X.X.X.X
    c. send()/recv() to/from that socket.

1 Like
13

How can I help to further diagnose communications between plexmediaserver.service and systemd-resolved?

It seems like the change of toolchain I keep hearing about [1] resulted in a change in the way Plex, curl, glibc, I don’t know what, resolves domain names, moving Plex from one group of programs (programs which have had and still have working DNS resolution, like ping, wget, and curl) into another group (dig and plex, which required the correction @Volts and I reached in order to have working DNS resolution), and I am not aware of a way to log more detail than I already have.

14

When the Linux standard tool dig needs “correcting” then the host itself needs more refinement.

nslookup, dig, ping, curl, wget, etc are all standard Linux tools.

If they can’t run without “adjustment”, is it really fair to expect Plex to run?

15

Yes, I think it is fair to expect Plex to continue to run the same way after an update. Instead, it developed an entirely new DNS issue that brought dozens of people here today from Ubuntu, Synology, Docker, and Arch Linux to get help, only to find someone who essentially takes offense at the suggestion that Plex has a new issue.

I came here today to help people with the same issue diagnose why their server has entirely stopped functioning due to an update, and I have posted my solution. All I wanted to do was get info to users and developers that might remove this issue from the next update. I can’t help you beyond that.

16

Time out?

I am NOT taking offense to the issues.

I KNOW there are issues. I’m all over the Linux forum supporting users, finding the root causes, and getting them resolved.

I respectfully request you work with me and not think of me as an adversary because I’m not.

I DO have issues with ANY OS distribution where dig or nslookup don’t work OUT OF THE BOX.

Therefore, to avoid any further confusion because apparently I missed something —

  • Did dig , nslookup, or any other system utilities require adjustment?

Is this above-quoted statement applicable here?

FWIW: I am in the Engineering meeting tomorrow morning (my time) to discuss all this . It is already on my agenda

1 Like
17

Plex may behave slightly differently from other apps on a Linux system in several cases.

Depending on the system configuration, systemd may be in control of - or independent of - /etc/resolv.conf. If you’re drinking the systemd Kool-Aid, I think you want them linked.

There are also some differences in how musl-linked apps (like Plex) perform DNS resolution vs. glibc.

musl libc - Functional differences from glibc

# Name Resolver/DNS

Traditional resolvers, including glibc’s, make use of multiple nameserver lines in resolv.conf by trying each one in sequence and falling to the next after one times out. musl’s resolver queries them all in parallel and accepts whichever response arrives first.

If DNS configuration problems exist, this could expose Plex to them, such as nameservers that respond incorrectly, or if any configured servers respond incorrectly to AAAA requests.

On a mostly-glibc system, other apps (dig, etc.) could be “working”, but insulated from configuration problems by the glibc resolver’s sequential behavior.

18

It keeps being suggested that this happened because my server needed “adjustment”, but the server and Plex <v1.23 has run for years on this installation without any domain name resolution issues, then v1.23 introduced one, an issue that is shared by dig and nslookup (which I downloaded just to diagnose this, as I have never needed DNS diagnosis tools). The two groups of behavior that I already pointed out regarding how programs interact with the system’s domain name resolution are below.

Group 1 - DNS works with or without nameserver entry in /etc/resolv.conf (or the file it links to, symlink or not does not affect functionality)

  • ping
  • curl
  • wget
  • Plex v1.22 and older

Group 2 - DNS only works with nameserver entry in /etc/resolv.conf (or the file it links to, symlink or not does not affect functionality)

  • nslookup
  • dig
  • Plex v1.23.0 and newer

To summarize: this update caused Plex to lose a method of resolving domain names that it previously had, affecting users on Ubuntu, Synology, Docker, and Arch Linux (Fedora package).

To answer your question: No, as I already wrote above, the symlink was not what fixed Plex’s new DNS troubles, but rather manually filling out the nameserver argument in /etc/resolv.conf (or the file it links to) was. That’s why I wrote that, and marked it as the solution.

dig and nslookup don’t come with any of the Linux distributions Plex officially supports, not Ubuntu/Debian (dnsutils), not CentOS/Fedora/SUSE (bind-utils), not Synology (DNS Server) and they won’t work until a DNS server is provided in the file linked to by /etc/resolv.conf, so they are not “standard Linux tools” and they are not going to work "OUT OF THE BOX", so I guess you “have issues with” all Linux distributions. They might come with the Plex Docker image, I’ve never used it, but as we know, people had the new DNS issues there too.

19

Wait - you had no ‘nameserver’ configured in /etc/resolv.conf?

Edit: Ok, I believe the glibc default in that case is to use a DNS server on the local machine. I’m wondering if musl behaves the same.

Edit2: Yup. Musl also uses a dns server on the local machine by default. So that should behave the same … assuming the dns server on the local machine is behaving. :slight_smile:

20

Let me detail:

  1. For all versions of Plex PRIOR to 1.23.0, we did use glibc
  2. Starting with 1.23.0, Engineering switched to MUSL to free itself from the multitude of glibc versions out there ; each with its own set of quirks.

They chose MUSL.

They did it because, for once, there is no dependency on glibc.

You can take PMS, and it’s runtime libs, craft up an environment, set the environment variables, and a launch script, and just Start Up anywhere on any Linux machine with a decent kernel.

Now, we’re finding out MUSL’s quirk – and it’s apparently a BIG one they didn’t find or research well enough…

( Guess that makes it our job. You spotted, we dig into it deeper, I focus it down to text, and write up for them… :roll_eyes: )

2 Likes