We've noticed that sporadically our sites will return a 404 response for all pages except the home page. We've witnessed that the behavior always starts to be exhibited directly after the application pool is recycled. Once the 404 results start occurring the only resolution is to recycle the application pool again which then clears the tab dictionary from cache so that it can be reloaded again.
I've reviewed the code & believe that the root cause of the issue is due to the fact that the code that builds the tab index, portalDepths dictionary & tabPaths dictionary is not thread safe. I can see code in the method TabIndexController.FetchTabDictionary is using SharedDictionary classes to store the tab dictionaries, however the code is not threadsafe when adding the dictionaries to the cache. Therefore when multiple threads are executing the FetchTabDictionary method it's possible for an empty dictionary to be added to the cache.
To resolve this issue the code needs to be updated so that only one thread can add the dictionaries at a time. I have provided an updated version of the TabIndexController class with the FetchTabDictionary method modified as an attachment. I've also added another private helper method called PortalExistsIndex in the TabIndexController class, that is used by the FetchTabDictionary to determine if the portal has already been added to the index.
This is an issue that seems to have been in DNN for some time. Here are a couple of links to others that have experienced similar behavior which seems like it might be due to the same issue.