Yep I will have a look at it and click refresh on the timeline for when i get back from work.
thats the total time line
Time from start to first pink dip 8:29:48pm to 5:11:25pm
Time from Pink dip to yellow peak 5:11:25pm to 8:42:19 am
Time for yellow slight dip 298514.7 on timeline 8:42:19am to 7:38:13am
Time from Yellow dip to crash 7:3813am to 5:29:14pm
Hope that breaks down the time for you
Thank you very much for getting all the diagnostics. The process is close to the 2Gb maximum and evidence suggests it did hit 2Gb at the peak times shown
This is now being investigated using the excellent data that you provided in terms of the 81 log files and the VMMap saved data
No thank you for taking the time to look into this issue. Hope this can get resolved and other users can benefit from it too.
@puddness said:
No thank you for taking the time to look into this issue. Hope this can get resolved and other users can benefit from it too.
Have been looking at the VMMap data and logs and discussing with the development team.
The memory is being used up threads that are being created but do not appear to be doing anything - nothing in the logs for 1110 threads of the 1165 threads that show up in VMMAP at the peak at 08:42:19. With each thread taking up minimum of 1.25 MB memory that used up about 1.5GB. There should not be that many threads created and it is not clear as to why that is happening.
The server I believe was running for 44 hours when the steep climb started at 17:11 2nd March and peaked after 59 hours of running at 08:42:19 3rd March.
Other users that had this issue found that Bitdefender was a factor. There was another user with these errors that did not have BitDefender but i have not see VMMAP evidence from the user.
I would like to see a list of other software is running, so could you send me by private message the output of this command. Please start a command line window with Run As Administrator option {start / run / cmd and then right click and select Run As Administrator)
In the command line window, type this command
wmic /output:C:\users\xxxxx\appdata\local\InstallList.txt product get name,version
Substitute your windows profile name for the xxxxx
and zip the output file and send me the zip
Do you recall when the issue started ?
Just sent you a DM for the link it was very small no need to zip it.
I would say about 2 months or so now is when the issue started to happen
@puddness said:
Just sent you a DM for the link it was very small no need to zip it.I would say about 2 months or so now is when the issue started to happen
Thank you. One program name is blank.
so will add some extra parameters
wmic /output:C:\users\xxxxxx\appdata\local\InstallList.txt product get Description, InstallDate, Name, Vendor, Version
The issue is the amount of memory used up for the stack (for Plex Media Server threads) - it should not be climbing to the level it has got to) . The following shows 3 snapshots which shows the stack totals at 3 different times. It is not clear yet as to why this is happening and why it is happening on your system but not other users. Previous users with the errors this gives rise to found that BitDefender was a factor. Why that was I do not know. I am not sure if Trend Micro Maximum Security` would cause similar issues.
It could be some exception path that is resulting in new threads not being terminated.
I would like to check out these errors
Mar 03, 2018 06:34:08.955 [6256] ERROR - SQLITE3:0x791e549, 13, statement aborts at 20: [INSERT OR REPLACE INTO gnsdk_queries ("key","value","timestamp") VALUES(?,?,?);] database or disk is full
Mar 03, 2018 06:34:08.968 [6256] ERROR - SQLITE3:0x791e549, 262, statement aborts at 25: [INSERT OR REPLACE INTO gnsdk_queries ("key","value","timestamp") VALUES(?,?,?);] database table is locked: gnsdk_queries
Mar 03, 2018 06:34:09.130 [6256] ERROR - SQLITE3:0x791e549, 13, statement aborts at 20: [INSERT OR REPLACE INTO gnsdk_queries ("key","value","timestamp") VALUES(?,?,?);] database or disk is full
Mar 03, 2018 06:34:09.135 [6256] ERROR - SQLITE3:0x791e549, 1, statement aborts at 1: [COMMIT;] cannot commit - no transaction is active
Could you check out the file sizes of the .gdb files in this directory
C:\Users\Owner\AppData\Local\Plex Media Server\Cache
And is the C:\ drive close to being full?
C:\ Isn’t full i have about 800gb free on that drive.
First of all, would like thank you for all the diagnostics
The list of programs only shows the following as being non standard
Description InstallDate Name Vendor Version
Paragon HFS+ for Windows™ 10.5 20170210 Paragon HFS+ for Windows™ 10.5 Paragon Software 1.00
Trend Micro Maximum Security 20180219 Trend Micro Maximum Security Trend Micro Inc. 12.0
20170901 Dell 16.8.0.0
iStat Server 20170210 iStat Server
This gracenote error is probably unrelated - there is a cap on the queries cache file of 20Mb
Going back to the memory issue which appears to be due to extra threads being created within the Plex Media Server process adding 25Mb memory use for the stacks every hour, I have been discussing it with the development team and would like to go through this plan of action
- would like to try with the
Trend Micro Maximum Securityproduct disabled for a while. You can consider switching to the free Microsoft Windows Defender which is standard now on Windows 10 but for Windows 7 there isMicrosoft Security Essentials- see https://support.microsoft.com/en-gb/help/14210/security-essentials-download
You can continue to have Plex Media Server.exe monitored with auto-refresh by VMMap
If the problem does not go away and VMMap still shows the stack total memory figure for the process getting very high - it should never reach 1Gb for example, then do the following
- Save VMMap data into mmp file
- Capture the Plex Media Server logs
- Take a process dump of Plex Media Server.exe process. We need to use the 32-bit windows task manager to do that. Launch
C:\Windows\System32\Taskmgr.exeand select the Plex Media Server.exe process and right click and selectCreate Dump - Upload vmmap, logs zip, and zipped dump to dropbox and send me link by PM
May at some point need to use Process Monitor to establish what is creating these threads
Plex Just crashed at 3:40am March 12’
here are screen shots of the timeline for you and i will pm you a link of all the data above.
Thanks for the new diagnostics - so this was without Trend Micro Maximum Security
I will be looking at the evidence but I really need to try to get closer to see what is creating the large number of threads.
The problem is not being spotted quickly enough and we need to change so as not to wait for the crash and capture diagnostics before that and also introduce SysInternals Process Monitor
-
The server was launched Mar 09, 2018 07:35:34 am. VMMap capture started at the beginning at about 07:36 am
-
After about 26 hours from launch at 09:39:05 am on March 10, the stack memory use showing on VMMap was 742,400 KB (the first climb e peak in the timeline)
-
After about 57 hours from launch the first thread allocation boost error occurred at Mar 11, 2018 16:51:46 with memory errors started about an hour after that at 18:06:02
-
After about 61 hours from launch at 21:15:24 am on March 11, the maximum memory usage was reached (the 2nd peak in the timeline)
-
The logs cover Mar 11, 2018 16:50:55 to March 12 03:35:36 (crash time)
Could you for next time -
- keeping Trend Micro Maximum Security off,
- starting with VMMap as you have done before.
- after 26 hours of running from launch - check VMMap screen and select View / Refresh, and see what the total is for type Stack (first column) with value in 2nd column. If it is high - over 500,000 K then Run SysInternals ProcMon.exe to capture all events and run it for 5 minutes and then save all captured events into a PML file. Noting down the times and capture the Plex Media Server Logs.
- If the stack total memory was below 500,000 wait few hours and do the above then
- 24 hours after that refresh VMMap and note down the stacks memory total as before and again run ProcMon.exe - this time for 10 minutes and save all events to PML file and collect the server logs
- The rest would be as for now, save the VMMap mmp file and get logs - assuming you have had peaks of memory usage
The PML files would be large but they zip very well and please avoid leaving procmon.exe running for longer than the suggested 5 and 10 minutes as procmon.exe gobbles lots of system resources
Thank you for all your help
I have done the above instructions and sent you a DM with the dropbox link with screenshots and files and logs for you to look at.
any more data just let me know.
Thanks @puddness - waiting for you to let me have the VMMap saved environment (vmm file zipped)
Thanks i now have the vmmap data covering 46 hours up to 18:20 on 14th March
as mentioned in my PM - please avoid zipping logs in-situ as that would skip all currently open log files
Always copy Logs out and then zip the copy
I have been analysing the massive data captured - The procmon captures only picked the creation of a few of the hundreds of threads created leading to the memory issue. I have referred it to the development team to see if these few threads that could be seen are actually a small sample of the bigger issue. These threads were being created when the premium music library was being scanned and gracenote data searched - appeared to coincide with transcode jobs started for each music file - as part of the gracenote fingerprinting process
I’ve noticed that the media scanner for the Music does take a really long time more than any other library.
What is puzzling at this point is why I cannot reproduce the problem and why it is not widespread
Just noticed that Plex Media Server process was an RDP session. Microsoft made a lot of security lockdowns on processes started through RDP - it is NOT common that PMS runs in an rdp session - just thinking out aloud here - no evidence to suggest that RDP has any bearing - but worth an experiment to see if RDP is avoided for launching PMS - if the problem magically goes away !
One of the changes Microsoft did was for Per Session Temp Directory but there is no evidence of that being used in your environment
I do use Windows Remote Desktop to login to the Server to make changes as I have my Windows 7 computer as a headless computer on 24/7
@puddness said:
I do use Windows Remote Desktop to login to the Server to make changes as I have my Windows 7 computer as a headless computer on 24/7
just wondering if it could be to do with launching PMS in RDP session. Wild thought that if true, would explain why it is not a common issue
@puddness could you tell me how many music tracks are in the premium music library that was being scanned?
