updated wildcard cert on CM server, now IBCM systems no longer communicating

[ITAdminTL]

Relatively new here to managing internet-based systems in SCCM, but I'm stuck and need some advice.   Our org is managing a few hundred internet based devices that check into a dmz management point via https.   Our current certificate was expiring this month so we create a new one and deployed it to the CM server.    Now all of our clients are failing to connect in and are dropping off as inactive.    

Location services log shows this


Settings in client computer communications tab are set as this:

 

However, none of these settings were touched, only the certificate updated.     All clients were communicating properly before we updated the certificate.  When visiting the management site directly from a client, it shows the updated cert on the management point Site as being the updated one, and all intermediate certificates are shown and have the status of OK.   

Cert was generated using CheapSSL along with intermediates and all of them imported into the server without issue.   I'm kinda out of ideas here.  Anything else I can check?

[ITAdminTL]

So I think we have made some progress on this....we removed the AddTrustCA Root cert from the server (which appears to be what the issue is; client is seeing that cert and skipping the correct one (Thumbprint XXX  cert showing screenshot).  On a handful of working machines, they have gone back to validating the correct cert (one with Thumbprint) and they are communicating again.  However, Most all other clients are still using the AddTrustCA one, even though it's been removed from the server.  

How can we force clients to 'recheck' for the cert to use?   We've tried rebooting the systems to no avail.  Is there a way to force this?

[anyweb]

how were your clients getting certs in the first place ? was it via active directory gpo ? if so they'll need to be online long enough to get that updated cert and then should be fine, a reboot or gpupdate /force might speed it up

[ITAdminTL]

On 4/28/2020 at 12:04 PM, anyweb said:

how were your clients getting certs in the first place ? was it via active directory gpo ? if so they'll need to be online long enough to get that updated cert and then should be fine, a reboot or gpupdate /force might speed it up

This is a unique situation...these are all workgroup systems out in the field and were imaged with their personal store-specific certificate when they were first built.  We would update these certs using SCCM when they approached their expiration date using package deployments.  However now we're in a "chicken or the egg" situation since these clients are pulling the wrong cert (AddTrustCA) instead of the cert in the personal store to authenticate with.      Since yesterday, we've determined that if we remote into an individual system and clear out the HKLM:\SOFTWARE\Microsoft\CCM\Security\Certificate Issuers key (which on broke systems shows the AddTrustCA) to be blank, they systems immediate fall back to using their personal store cert and they all start communicating properly again.  

However we have a large number of these systems that are not communicating back properly to get this change from the MP.  I'm assuming because they can't authenticate...however what's weird is that a handful of them (20 or so) did succesfully connect back in and get the update to the trusted root authorities change we made (back to none specified).    

I think our issues started when we incorrectly changed that field in Client computer Communication tab to point to the root CA.   They all pulled that setting, and now they can't communicate. Switching it back isn't working for all the systems. 

The fact that some of these systems eventually trickle in and start communicating (it's been a week since we made the Trusted Rooth Authorities change back to "none specified") and intially we had a steady stream of systems start tricking back in and communicating.  Then we've hit a wall now, where nothing new is coming back, and we're stuck at about 100 machines out of a 1000.     I can't help but feel that there's some timeout on the clients that gets hit and eventually they "check back in" somehow with the MP and get the change, or they fall back to the personal store cert.    Is this true?  If so,  is this some kind of timeout based situation?  Can this be forced somehow?

[anyweb]

clients will try to check for machine policy as defined in the Client Settings of your site, by default it's once per hour, how have you defined your settings ? to skip the client settings defined settings you can manually open the Configuration Manager client and trigger a Machine Policy in the Actions tab.