Server Version#: 4.100.1
Yesterday I started having issues accessing Plex with using my custom TLS certificate, it defaulted to the plex.direct cert and for some reason the Plex Roku app stopped working (but the Android app kept working).
After turning on debug logging I found this in my Plex logs:
Apr 05, 2023 10:28:18.519 [0x7f6fb0519a90] ERROR - [CERT] PKCS12_parse failed: error:0308010C:digital envelope routines::unsupported
Apr 05, 2023 10:28:18.519 [0x7f6fb0519a90] ERROR - [CERT] Found a user-provided certificate, but couldn't install it.
From doing some research it looks like Plex is using openssl 3, and won’t work with .pfx certificates created with openssl 1.x, because the default options for 1.x are deprecated/removed from 3.x.
openssl 1.x:
-keypbe val Private key PBE algorithm (default 3DES)
-macalg val Digest algorithm used in MAC (default SHA1)
-certpbe val Certificate PBE algorithm (default RC2-40)
openssl 3.x:
-keypbe val Private key PBE algorithm (default AES-256 CBC)
-certpbe val Certificate PBE algorithm (default PBES2 with PBKDF2 and AES-256 CBC)
-macalg val Digest algorithm to use in MAC (default SHA256)
You can verify if it’s using the unsupported type by running openssl pkcs12 -in mycert.pfx -info and looking for this:
MAC: sha1, Iteration 2048
MAC length: 20, salt length: 8
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
So the fix for me was recreating the PKCS12 file using openssl 3.x instead of 1.x. Very annoying to track down, hope this helps someone else.
