Server hang with 100% CPU on Sitecore 10.1

Just a short PSA for anyone using Sitecore 10.1.0–10.1.2 with SXA (and possibly this affects 10.2 as well as Sitecore registered a new bug), if your CD server is locking up with 100% CPU:

I created a memory dump and profiler trace and then opened a Sitecore support ticket.

They came back:

From the provided memory dump (CPU usage is 100%), there are 5 requests stuck for quite some time. They have below call stack:

0000005a1a37ce30 00007ff7f95cfc1e System.Collections.Generic.HashSet`1[[System.__Canon, mscorlib]].AddIfNotPresent(System.__Canon)                                                                                                                                       0000005a1a37ceb0 00007ff7fdac8d05 Sitecore.XA.Foundation.Presentation.Services.LayoutXmlService.RegisterInjectedRenderings(Sitecore.Data.ID, System.Collections.Generic.IEnumerable`1<System.Xml.Linq.XElement>)                                                         0000005a1a37cf50 00007ff7fdac89fa Sitecore.XA.Foundation.Presentation.Services.LayoutXmlService.RegisterInjectedRenderings(System.Collections.Generic.IEnumerable`1<System.Xml.Linq.XElement>)                                                                           0000005a1a37cfb0 00007ff7fdac6005 Sitecore.XA.Foundation.Presentation.Pipelines.GetXmlBasedLayoutDefinition.AddPartialDesignsRenderings.Process(Sitecore.Mvc.Pipelines.Response.GetXmlBasedLayoutDefinition.GetXmlBasedLayoutDefinitionArgs)

The issue is caused by corrupted cache instance: under load several threads can write to the cache “SXA[InjectedRenderingsCacheName]” concurrently without locking it. As the result the underlying HashTable collection is corrupted and its enumerator never completes execution once called.

This issue was registered as a bug in our bug tracking system with the reference number 516880. More information about public reference numbers can be found here: https://support.sitecore.com/kb?id=kb_article_view&sysparm_article=KB0853187

To work around the issue I’d like to suggest disabling the “SXA[InjectedRenderingsCacheName]” cache from Sitecore.XA.Foundation.Presentation.config by setting its max size to 0 MB. For this purpose please add the following node under configuration/sitecore/settings (it can be added via Patching Config File):

<setting name="XA.Foundation.Presentation.InjectedRenderingsCacheMaxSize" value="0MB" />

I updated this config setting and it looks to have resolved the issue — it hasn’t happened again in the last two weeks.