Primary Site Server High CPU Usage - IIS Worker Process for WSUSPool

[gus-bus]

Hello SCCM Folks,

 

We are seeing an abnormally high CPU usage on our primary site server and have tracked it down to the WSUSPool of the IIS Worker Process. Digging even further we are finding that it is being caused by our Server 2016 boxes hitting the WSUS site consistently almost every three minutes.

 

We run our primary site server on Server 2012 R2 using System Center Configuration Manager 1610 (site version: 5.0.8458.1000). The resources on this machine have been sufficient for some time and serving our needs until recently with the CPU being maxed out. We have thrown additional resources at it, but to no surprise those are consumed quickly and not the answer.

 

Our Server 2016 VM's are seeing the following error in the WUAHandler.log:

 

--------------------

 

OnSearchComplete - Failed to end search job. Error = 0x8024401c. WUAHandler 2/1/2017 12:52:55 PM 2844 (0x0B1C)

Scan failed with error = 0x8024401c. WUAHandler 2/1/2017 12:52:55 PM 2844 (0x0B1C)

The new content version (191) of update source is less than before (499), continuing. WUAHandler 2/1/2017 12:52:55 PM 5040 (0x13B0)

---------------------

I have tried a few different suggestions from other forums about changing the proxy settings in Internet Options, which did not resolve the issue (we do not use a proxy). I am able to get the 2016 boxes to point to Microsoft Update servers externally for patches at this time, but these settings are changed back to our internal WSUS server per Group Policy as they should just to see the same errors.

The group policy we have in place is pointing all OSes to the same WSUS path (our primary site server) with no issues on 2008 R2, 2012 R2, Windows 7, 8, or 10.

Has anyone else seen this issue or have any thoughts? We started seeing this on the January 24th, 2017 according to the performance chart in VMware for that server.

I would appreciate any feedback.

Thanks

/Gus

Edited February 1, 2017 by gus-bus

[gus-bus]

Sorry folks, I think i got this one on my own. Below is what I have found:

 

**The IIS Worker service was using high amounts of CPU. After further investigation we found the issue to be with the WsusPool Worker Process on the server. It would climb up and service a few clients and then recycle on a regular basis. We noticed the same IP addresses were in the "current requests" for the worker service each time before it crashed. It was the Windows Server 2016 servers, which were coincidentally the Advanced Threat Analytics servers (not related). **

 

 

After many searches and digging on this topic it would appear it is an IIS configuration change issue, which is reflected in the following articles:

 

• http://serverfault.com/questions/522832/problems-with-sup-on-sccm-2012-sp1
• http://blog.coretech.dk/kea/house-of-cardsthe-configmgr-software-update-point-and-wsus/
• https://social.technet.microsoft.com/Forums/en-US/a629c131-1c12-4803-a479-e5c6ca784b10/sccm-wsus-failed?forum=configmanagergeneral
• http://stackoverflow.com/questions/16162524/how-to-increase-memory-and-cache-size-for-application-pool-in-iis-7-efficiently
• https://blogs.msdn.microsoft.com/the_secure_infrastructure_guy/2015/09/02/windows-server-2012-r2-wsus-issue-clients-cause-the-wsus-app-pool-to-become-unresponsive-with-http-503/

 

 

So here are the settings that I changed within IIS on

 

• Private Memory Usage Increase (for Recycling Conditions)

 

- Private Memory Usage increase to 8388608 KB

 

• Output Cache Increase (for response file size):

 

- Output Cache Settings increase to 1048576 (in bytes)

• Maximum Worker Processes:

 

-Maximum Worker Processes from 1 to 0 (System must be NUMA aware for this and will essentially trigger as many worker processes as there are NUMA nodes).

 

Thanks

/Gus

[creaton]

Hey,

Thank you gus-bus!

I did everything except "Maximum Worker Processes from 1 to 0 (not the NUMA info)." That is works absolutely fantastic! I've never seen wsus (+sccm) works that fast. 

Today I notice that one client again hang wsus app pool. I try to play with Max work process settings: put 1 and recycle. And again a lot of client hang wsus. I can see that almost immediately. I  also try to put different values to that setting - no luck.

Best for now is leave 0.

I'm absolutely sure that Microsoft should write detailed manual about that cause out of the box it doesnt work. At least in windows 2016 + sccm CB configuration.

 

Maybe you found the way how to completely resolve that issue?

[gus-bus]

Hello Creaton,

 

I've actually moved to a different organization since the original posting and to my knowledge they have not had an issue since. However, at the new organization I am at we are currently seeing a similar issue again with Server 2016 VM's. I've made the required tweaks from the posting above but didn't notice a full fix on it. I do want to point out that in this new environment they have the server patching and workstation patching in two completely separate SCCM environments. The server patching is done quarterly so this server isn't touched all that often. I also want to note that Server 2008, 2008 R2, 2012, and 2012 R2 are receiving their SCCM updates flawlessly, it's just the 2016 boxes. I've bumped up the memory configurations quite a bit and still see the issue so there are a few maintenance items that needed to be checked, primarily WSUS (see links in the post above). In this server SCCM environment the expire superseded updates and WSUS clean up were unchecked so that is likely the issue at this time and I am working through that process now. 

 

Are you only seeing the issue on the 2016 servers in your environment or everywhere? 

 

/Gus

[TrialandError]

There are a couple known issues with high cpu utilization on SUPs right now which could be effecting you both. Scott Williams has written a pretty good article on it. I believe MS has released some private patches for this issue and plan to issue a hotfix in September.

https://kickthatcomputer.wordpress.com/2017/08/13/wsus-sup-causes-high-cpu-and-clients-fail-updates-scan/amp/

[creaton]

Alright..... I'm not alone

thanks for reply. I'm glad I wrote here.... That's exactly what I'm talk about  no enough information from MS....

 

4 hours ago, gus-bus said:

Are you only seeing the issue on the 2016 servers in your environment or everywhere? 

Yep, have issue with modern windows (ws16, w10).

After read article TrialandError (thank you for that!) I get the point.....

So next step is: did anyone try to use latest CB feature: Express files for modern system? I did try in 1702 but it was so slowly.... they fix that in new 1706: 

• Improvement for Express Update support – Express file download time has been significantly improved.

About remediation/workaround I will try it and post here. Looks very promising.

 

[gus-bus]

Exactly, we're not the only people with this issue. TrialandError, great article/find! I appreciate you guys posting here as this has been frustrating to deal with...again. I've got our environment partially tweaked to referenced the article from TrialandError and it seems to be functioning in bandaid capacity until we hear something further from MS. 

 

As for the Express files, we had tried that in the last couple of weeks with not the world's greatest success and our results matched what many others had been posting. If that feature-set begins to work properly I could seem some value; however, with the size of some of the download files, it just isn't feasible to send to our low bandwidth sites. I will warn you that once you start synchronizing the updates, it hasn't been just a simple checkbox to uncheck and the files don't show up anymore. In our environment the files continued to download time and time again.

 

About 1/2 hour ago I too submitted a ticket with MS to get another person hitting them up on this issue as it is rather important in my opinion. Hopefully we'll get access to this magical patch that may fix the issue. 

 

I'll let you know what I hear. 

 

/Gus

[gus-bus]

So a brief update here on the MS ticket submitted. Of course just before the MS call, we had things fairly leveled off with our WSUS/SCCM environment, but we still dug into it. Here's the closing of the ticket:

--------------------------------------------------------------------------------------------

 

SYMPTOM:

2016 server 'sqlsbx02' failing scan with error code '0x8024401c' on SCCM server 'srvsccmprd01'.
 
 
CAUSE:  

High CPU utilization.
 
 
RESOLUTION: 
We declined KB 4034658 and removed the obsolete updates from SUS DB which helped in resolving the issue.

Also the resolution for the issue mentioned in KB4034658 is being worked upon and will soon be released publically.

You can refer to the following articles which are useful in bringing down high CPU utilization:

https://blogs.msdn.microsoft.com/the_secure_infrastructure_guy/2015/09/02/windows-server-2012-r2-wsus-issue-clients-cause-the-wsus-app-pool-to-become-unresponsive-with-http-503/  

https://blogs.technet.microsoft.com/configurationmgr/2016/01/26/the-complete-guide-to-microsoft-wsus-and-configuration-manager-sup-maintenance/
 

--------------------------------------------------------------------------------------------

In addition to the above references we also performed the following against the WSUS database in SQL:

 

Obsolete Updates Check Query:  

 

exec spGetObsoleteUpdatesToCleanup

 

Cleanup Query:  

 

                 DECLARE @var1 INT 
                DECLARE @msg nvarchar(100) 
                
                CREATE TABLE #results (Col1 INT) 
                INSERT INTO #results(Col1) EXEC spGetObsoleteUpdatesToCleanup 
                
                DECLARE WC Cursor 
                FOR 
                SELECT Col1 FROM #results 
                
                OPEN WC 
                FETCH NEXT FROM WC 
                INTO @var1 
                WHILE (@@FETCH_STATUS > -1) 
                BEGIN SET @msg = 'Deleting ' + CONVERT(varchar(10), @var1) 
                RAISERROR(@msg,0,1) WITH NOWAIT EXEC spDeleteUpdate @localUpdateID=@var1 
                FETCH NEXT FROM WC INTO @var1 END 
                CLOSE WC 
                DEALLOCATE WC 
                DROP TABLE #results 

 

That's what I have so far and to be honest, things have significantly improve post WSUS database maintenance which took place before the support call. 

 

/Gus

[Bebop]

Hi all,

Here are the new updates that fix the problem :

1. Windows Server 2016 (KB4039396)
2. Windows Server 2012 R2 (KB4039871)
3. Windows Server 2012 (KB4039873)
4. WSUS 3.0 SP2 (KB4039929)

New update of August 28 (today). I installed it, and it works.

Enjoy

Bebop.

[gus-bus]

Hi Bebop,

This is great news! We'll install these updates in our environment and see what happens. 

Thanks again for everyone's contributions on this one. Hopefully others will find some value from our headaches. 

/Gus