Plex server not working anymore after container upgrade from 1.29.0.6244 to 1.29.1.6316 and upwards

Plex Media Server NAS & Devices
,
1

Server Version#: 1.29.1.6316

Dear community,

I use Plex on my QNAP TS-673A for 4 months now, I use docker and upgrading very often to the latest plexinc/pms-docker.

BUT, unfortunately, since version 1.29.1.6316 (same issue with newest 1.29.2.6364), my server is not working anymore, I can’t figure out why.

Here are the logs from "Plex Media Server.log ":
Nov 08, 2022 19:08:21.487 [0x7ff5052ecb00] DEBUG - Request: [127.0.0.1:54784 (Loopback)] GET /identity (6 live) #3dd Signed-in
Nov 08, 2022 19:08:21.487 [0x7ff5082a7b00] DEBUG - Completed: [127.0.0.1:54784] 200 GET /identity (6 live) 0ms 398 bytes (pipelined: 1)
Nov 08, 2022 19:08:24.852 [0x7ff5082a7b00] DEBUG - Request: [10.0.3.1:54842 (Subnet)] GET / (6 live) #3de GZIP Signed-in
Nov 08, 2022 19:08:24.853 [0x7ff5082a7b00] DEBUG - Completed: [10.0.3.1:54842] 401 GET / (6 live) GZIP 0ms 435 bytes
Nov 08, 2022 19:08:26.739 [0x7ff5052ecb00] DEBUG - Request: [127.0.0.1:54918 (Loopback)] GET /identity (6 live) #3e1 Signed-in
Nov 08, 2022 19:08:26.739 [0x7ff5080a4b00] DEBUG - Completed: [127.0.0.1:54918] 200 GET /identity (6 live) 0ms 398 bytes (pipelined: 1)
Nov 08, 2022 19:08:31.989 [0x7ff5052ecb00] DEBUG - Request: [127.0.0.1:55070 (Loopback)] GET /identity (6 live) #3e3 Signed-in
Nov 08, 2022 19:08:31.989 [0x7ff5080a4b00] DEBUG - Completed: [127.0.0.1:55070] 200 GET /identity (6 live) 0ms 398 bytes (pipelined: 1)
Nov 08, 2022 19:08:37.285 [0x7ff5052ecb00] DEBUG - Request: [127.0.0.1:55174 (Loopback)] GET /identity (6 live) #3e6 Signed-in
Nov 08, 2022 19:08:37.285 [0x7ff5082a7b00] DEBUG - Completed: [127.0.0.1:55174] 200 GET /identity (6 live) 0ms 398 bytes (pipelined: 1)
Nov 08, 2022 19:08:40.416 [0x7ff5084e7b00] DEBUG - Shutting down with signal 15 (Terminated)
Nov 08, 2022 19:08:40.416 [0x7ff5084e7b00] DEBUG - Ordered to stop server.
Nov 08, 2022 19:08:40.416 [0x7ff50c8951c0] DEBUG - Stopping server…
Nov 08, 2022 19:08:40.416 [0x7ff50c8951c0] DEBUG - HttpServer: Stopping server.
Nov 08, 2022 19:08:40.416 [0x7ff507d9bb00] DEBUG - MetadataItemClusterRequestHandler: a shutdown event has been received
Nov 08, 2022 19:08:40.416 [0x7ff507d9bb00] DEBUG - TranscodeJobManager: [Running] onServerShutdown
Nov 08, 2022 19:08:40.416 [0x7ff507d9bb00] DEBUG - TranscodeJobManager: [Running] → [Shutdown]
Nov 08, 2022 19:08:40.416 [0x7ff50c8951c0] DEBUG - Server stopped.
Nov 08, 2022 19:08:40.416 [0x7ff50c8951c0] DEBUG - NetworkService: Stopping advertisement.
Nov 08, 2022 19:08:40.416 [0x7ff502505b00] ERROR - Network Service: Error in advertiser handle read: 125 (Operation canceled) socket=70
Nov 08, 2022 19:08:40.416 [0x7ff50c8951c0] DEBUG - Killing plug-ins.
Nov 08, 2022 19:08:40.455 [0x7ff50c8951c0] DEBUG - [com.plexapp.agents.imdb] Killing.
Nov 08, 2022 19:08:40.459 [0x7ff5084e7b00] WARN - JobManager: Could not find job for handle 356
Nov 08, 2022 19:08:40.474 [0x7ff50c8951c0] DEBUG - [com.plexapp.system] Killing.
Nov 08, 2022 19:08:40.478 [0x7ff5084e7b00] WARN - JobManager: Could not find job for handle 282
Nov 08, 2022 19:08:40.481 [0x7ff50c8951c0] DEBUG - ChildProcessMonitor: Waiting for Plex Tuner Service to exit.
Nov 08, 2022 19:08:40.481 [0x7ff50c8951c0] DEBUG - [JobRunner] Signalling job ID 358 with 2
Nov 08, 2022 19:08:40.508 [0x7ff5084e7b00] DEBUG - Jobs: ‘/usr/lib/plexmediaserver/Plex Tuner Service’ exit code for process 358 is 0 (success)
Nov 08, 2022 19:08:40.508 [0x7ff501e9eb00] DEBUG - [ChildProcessMonitor] Plex Tuner Service exited.
Nov 08, 2022 19:08:40.508 [0x7ff50c8951c0] DEBUG - Database: Shutting down.
Nov 08, 2022 19:08:40.516 [0x7ff502505b00] INFO - Network Service: Abandoning advertise socket, it was closed.
Nov 08, 2022 19:08:40.764 [0x7ff50c8951c0] DEBUG - Database: Shutting down.

and from "Plex Tuner Service.log :
Nov 08, 2022 19:08:40.481 [0x7f5b019c1808] INFO - Received SIGINT: Interrupt from keyboard.
Nov 08, 2022 19:08:40.505 [0x7f5b019c1808] INFO - [I] device_manager_t::term
Nov 08, 2022 19:08:40.505 [0x7f5b019c1808] INFO - [I] device_manager_t::term. Stop all devices
Nov 08, 2022 19:08:40.505 [0x7f5b019c1808] INFO - [I] device_manager_t::term. Shut down all device auxes
Nov 08, 2022 19:08:40.505 [0x7f5b019c1808] INFO - [I] tvbutler_aux_t::term
Nov 08, 2022 19:08:40.506 [0x7f5b019c1808] INFO - [I] web_server_t::stop. Web server is stopped
Nov 08, 2022 19:08:40.506 [0x7f5b019c1808] INFO - [I] device_manager_t::term
Nov 08, 2022 19:08:40.506 [0x7f5b019c1808] INFO - [I] device_manager_t::term. Stop all devices
Nov 08, 2022 19:08:40.506 [0x7f5b019c1808] INFO - [I] device_manager_t::term. Shut down all device auxes
Nov 08, 2022 19:08:40.506 [0x7f5b019c1808] INFO - [I] broker_t::shutdown. Shutdown is completed.

I can’t figure out why the server receives a SIGINT.
When I revert to previous version, server is back online.

For info, my YAML for the container definition :
version: ‘3’

services:
plex:
restart: “no”
image: plexinc/pms-docker:1.29.2.6364
container_name: “plex”
environment:
LD_LIBRARY_PATH: “/usr/local/cuda/extras/CUPTI/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64”
TZ: “Europe/Zurich”
ADVERTISE_IP: “http://192.168.1.200:32400/”
PLEX_CLAIM: “claim-XXXXXXXXXXX”
UID: “1001”
NVIDIA_VISIBLE_DEVICES: all
NVIDIA_DRIVER_CAPABILITIES: all
PATH: “/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin”
ports:
- 32400:32400
devices:
- /dev/nvidia0:/dev/nvidia0
- /dev/nvidiactl:/dev/nvidiactl
- /dev/nvidia-uvm:/dev/nvidia-uvm
volumes:
- type: tmpfs
target: /transcode
tmpfs:
size: 10000000000
- /usr/local/cuda/extras/CUPTI:/usr/local/cuda/extras/CUPTI
- /share/CACHEDEV1_DATA/.qpkg/NVIDIA_GPU_DRV/usr:/usr/local/nvidia:rw
- /share/docker/plex:/config

Could you provide any guidance to troubleshoot ?

Regards,

Valentin

2

Effective PMS 1.29.1 and above, Docker containers are no longer required to have access to HW HDR Tonemapping.

  1. Full DEBUG server logs will be needed which capture the failure to engage.
    ( Make a tar.gz of the “Logs” directory if necessary )

  2. Which QNAP model, QTS firmware version, NVidia GPU card, and NVIDIA_GPU_DRV package version are installed

3

Here are the informations regarding my NAS, and the logs from Plex server.

NAS : QNAP TS-673A, with a Nvidia 1050TI 4GB RAM
QTS 5.0.1.2194
NVIDIA drivers 5.1.0822

For the logs analysis, bear in mind that I switched numerous times between container versions, from 1.29.0.X (working) and 1.29.1.X or 1.29.2.X (not working). Last try this morning @08h08 with 1.29.2.6364
Logs_Plex.zip (3.5 MB)

4

@ChuckPa , any chance you analyze the logs ? Let me know if you need more details/logs.
I don’t feel comfortable with my outdated version in production.

Regards,

5

@v_de6

Apologies for the delay. Somehow I didn’t receive notification.

In your logs,

  1. it appears your docker container certificate has issues (you re-created it?)
Nov 08, 2022 21:45:23.734 [0x7f72b9a16b00] DEBUG - Content-Length is -1 (of total: -1).
Nov 08, 2022 21:45:23.743 [0x7f72c9557b00] WARN - [CERT] TLS connection from [::ffff:192.168.1.18]:56903 came in with unrecognized plex.direct SNI name '192-168-1-200.30ce5c9573ec4710a0cd25d72a33a6e8.plex.direct'; using installed plex.direct cert
Nov 08, 2022 21:45:23.766 [0x7f72c9557b00] DEBUG - CERT: incomplete TLS handshake from [::ffff:192.168.1.18]:56903: sslv3 alert certificate unknown

There are several instances of this throughout your logs.

I can reset your certificate and then you can restart the server AND connected clients so they receive the new certificate ID.

I notice in your account you have a container “V2” and a native “V3” server ??

Which server would you like to work with?

I ask because:

  1. PMS 1.29.2, Native app,
  2. QTS 5.0.0.2194 (current)
  3. NVIDIA_GPU_DRV 5.1.0

provide the same HDR tone mapping you were previously getting only in the Docker container.

6

Thanks for your proposals @ChuckPa.

The main reason why I use docker, is that I open the plex to the outside world, so for cyber security reasons, I prefer to expose a container more than directly the QNAP OS itself. If my plex container is compromised thru an attack, I suspect that it is an additional fence before the attack will find a way to access my QNAP itself, providing me more time so that I can stop it.

Regarding my container issue, I admit that those logs are too complicated, as it is a mixture of logs coming from my Plex app pluged in my QTS and from the container itself. This morning, I cleared all the logs and tried again to start the Plex container 1.29.2.6364. Then, I went to retrieve the logs, but no logs at all were generated ! It seems that there is a problem at the very beginning of the continer boot, prior to the PMS start.

I found a script called /healthcheck.sh, here is the result :
root@69654feaad81:/# ./healthcheck.sh
curl: (7) Failed to connect to localhost port 32400: Connection refused

We have to find another place where some system logs could be found I guess, to better understand what’s going on.

Regardig the certificate issue, it is related to the Plex app (NASV3) which is working, so let’s concentrate on making 1.29.2.6364 working, then we will fix this secondary issue if necessary. I use Plex app only after a reboot of the QNAP, because container needs Nvidia devices that are created by Plex (like /dev/nvidia0), these devices will disappear after a cold start and my container will fail to start if those devices are not present when the container is started.

I switched back to 1.29.0.6244, Plex container is working again, of course.
I also uploaded the logs from proper start of my 1.29.0.6244 (which is working well), just in case it could help.

Regards,

Valentin
NASV2_logs.tar (270 KB)

7

Dear @ChuckPa , any idea I could test during the week-end ? :slight_smile:

8

Please reconstruct your logs.

You attached a tar of the tuner log inside another tar (the Plex Media Server.log)

Plex Tuner Service.log^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@0000644^@0001750^@0001750^@00000003433^@14335113414^@013634^@ 0^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ustar  ^@plex^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@plex^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Nov 16, 2022 08:53:41.014 [0x7fcaa6c31b38] INFO - Plex Tuner Service v1.0.0.0 - Docker Docker Container x86_64 - build: 
Nov 16, 2022 08:53:41.015 [0x7fcaa6c31b38] INFO - Linux version: 5.10.60-qnap (#1 SMP Sat Oct 22 07:29:25 CST 2022), language: en-US
Nov 16, 2022 08:53:41.015 [0x7fcaa6c31b38] INFO - Processor AMD Ryzen Embedded V1500B                      
Nov 16, 2022 08:53:41.015 [0x7fcaa6c31b38] INFO - /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.29.0.6244-819d3678c 32600
Nov 16, 2022 08:53:41.014 [0x7fcaa8532808] INFO - [I] broker_t::start...
Nov 16, 2022 08:53:41.055 [0x7fcaa8532808] INFO - [I] device_manager_t::init.
Nov 16, 2022 08:53:41.055 [0x7fcaa8532808] INFO - [I] device_manager_t::init. Initialize provider manager
Nov 16, 2022 08:53:41.056 [0x7fcaa8532808] INFO - [I] device_manager_t::init. Initialize all device auxes
Nov 16, 2022 08:53:41.056 [0x7fcaa8532808] INFO - [I] dvb_device_factory_t::get_aux_list

On the other ‘cyber security’ point, my approach differs.

I put a hardware firewall device between my machines and the internet (pfSense).
I only allow those who I share with through that firewall on the Plex port (only).
I could give you my IP:port / FQDN right now and you’d not be able to verify it exists.

Given this level of security, why burden the system config?

Security in the Security box and Streaming in the Streaming box.

9

Hi @ChuckPa,

here are the logs zipped. Again, these logs are provided “for information”, coming from my container working in 1.29.0.6264, as I can’t retrieve any log from upper Plex versions.

As I’m more a system guy than a network guy, this is why I concentrate on system cyber sec. I will envisage a physical FW, but it will be on top of system cyber security, following my principle of multiplying fences in front of the attacker.
Logs_Plex_Container_working_1.29.0.6264.zip (4.3 MB)

Have a nice Week-end

10

If you will backup your /config (where it’s mapped to in QTS using tar from the SSH command line)

If you go into the container and stop PMS from inside the container the container remains running.

Now is where it will get interesting and why I suggest the all-important backup tarball

  1. Download the 1.29.2 Ubuntu/Debian x86_64 .deb file

  2. upload the file to the qnap where you can see if from within the container
    – ( Usually at the level under /config but above Library )

  3. At the container command line, dpkg -i <filename.deb> will install the deb file and upgrade the package.

  4. Start PMS while you’re still there in the container

  5. Now we can find out why it’s failing. Be careful to allow the database maintenance (schema upgrade) to complete before declaring failure.

  6. Lastly, you always have the ability to use FileStation to ZIP the “Logs” directory where they map into your QTS filesystem.

If you have any questions, please don’t hesitate to ask

11

Hi @ChuckPa , I’m a bit anxious to destroy my Plex.
A few details :

  • can you provide the link to Debian package 1.29.2 ? I only find the latest which 1.30 now
  • What is the command inside the container to start and stop properly PMS ?

Regards,

Valentin

12

Here’s how I would do it.

  1. Stop Plex
        # Stop Plex when using official Plex,inc image
        /plex_service.sh -d
--or--
        # Stop Plex when using Linuxserver.io Plex image
        s6-svc -d /var/run/service/svc-plex
  1. While in the container’s shell, make a tar file of all databases
cd /config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
tar cf Databases.backup.tar *
  1. Now upgrade PMS (DEB file right where /config maps to on the host.)
dpkg -i /config/plexmediaserver-xxxxx.deb
  1. Start PMS (change the -d option to -u in the Stop PMS command. d=down,u=up)

What we’ve done is save the most important part – Your databases.

This guarantees that if something goes wrong with 1.29.0 → 1.29.2 database migrations, you still have an image backup to restore to.

Normally, upgrade

13

Hi @ChuckPa , sorry for very late reply, I had personal issues that I had to tackle before coming back to “standard” life !

Well, I upgraded directly inside the container with the package you provided, I fall back to the exact same issue : PMS is not starting, no logs created.
Here is the list of processes after PMS restart /plex_service -u :
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 07:09 ? 00:00:00 s6-svscan -t0 /var/run/s6/services
root 38 1 0 07:09 ? 00:00:00 s6-supervise s6-fdholderd
root 234 1 0 07:09 ? 00:00:00 s6-supervise plex
root 431 0 0 07:09 pts/0 00:00:00 /bin/bash
root 1940 234 0 07:16 ? 00:00:00 bash ./finish 127 0
root 1947 1940 0 07:16 ? 00:00:00 sleep 5
root 1957 431 0 07:16 pts/0 00:00:00 ps -ef

Nothig else. We are in the exact same situation after the standard upgrade of the container.
Any idea ?

14

Do you want to try using my PlexDBRepair tool ? (it runs native to QNAP and can fix the basic stuff or at least tell us how badly things are mangled.

15

Yes, I will run your tool in my container, with pleasure.

16

stand by please… getting links now

17

To get it into the container on QNAP, it’s probably easiest to copy the tar.gz to where you store the media and move it from there once you get inside the container.

(this is container sensitive … you only need stop PMS before you do it… Docs in the README.md )

Recommend: Options 1 - 4 - 3

if those fail, we’ll advance to plan “rebuild”

18

The tool has a issue on “symbol not found” :
root@dd5e953760c9:/data/Movies/plex_tools# ./DBRepair.sh

  Plex Media Server Database Repair Utility (Docker)

Select

  1. Check database
  2. Vacuum database
  3. Reindex database
  4. Attempt database repair
  5. Replace current database with newest usable backup copy
  6. Undo last successful action (Vacuum, Reindex, Repair, or Replace)
  7. Import Viewstate / Watch history from another PMS database
  8. Show logfile
  9. Exit

Enter choice: 1

Checking the PMS databases
Error relocating /usr/lib/plexmediaserver/Plex Media Server: drmGetDevices2: symbol not found
Check complete. PMS main database is damaged.
Error relocating /usr/lib/plexmediaserver/Plex Media Server: drmGetDevices2: symbol not found
Check complete. PMS blobs database is damaged.

  Plex Media Server Database Repair Utility (Docker)

Select

  1. Check database
  2. Vacuum database
  3. Reindex database
  4. Attempt database repair
  5. Replace current database with newest usable backup copy
  6. Undo last successful action (Vacuum, Reindex, Repair, or Replace)
  7. Import Viewstate / Watch history from another PMS database
  8. Show logfile
  9. Exit

Enter choice: 4

Exporting current databases using timestamp: 2022-12-10_08.02.11
Export: (main)…Error relocating /usr/lib/plexmediaserver/Plex Media Server: drmGetDevices2: symbol not found
Error 127 from Plex SQLite while exporting com.plexapp.plugins.library.db
Could not successfully export the main database to repair it. Please try restoring a backup.

19

You forgot to stop Plex

I have no such dependency

20

No, PMS is stopped :
root@dd5e953760c9:/data/Movies/plex_tools# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 07:53 ? 00:00:00 s6-svscan -t0 /var/run/s6/services
root 38 1 0 07:53 ? 00:00:00 s6-supervise s6-fdholderd
root 341 1 0 07:54 ? 00:00:00 s6-supervise plex
root 794 0 0 07:55 pts/0 00:00:00 /bin/bash
root 4267 794 0 08:06 pts/0 00:00:00 ps -ef