Schedule REQUEST_MODE can cause multiple instances of the scheduler provider to be loaded on application startup
Description
QA Test Plan
Activity
Will MorgenweckDecember 13, 2016 at 12:12 AM
Batch update of issues that won't be addressed due to change in functionality from version 9.0
Stephen LimJune 27, 2014 at 2:38 PM
AFAIK, it happens when you have multiple IIS sites sharing the same App Pool process (regardless of how the folders are shared or not).
Wes TattersJune 26, 2014 at 8:31 AM
FWIW when we started testing this - we observed any number of combinations of above that caused potential issues.
And one of the most scary was the fact that under certain circumstances - it seems possible that even using a double bind semaphore with a static object - like the solution detailed in our original post could possibly still fail.
The reason for this seemed to be as a result of some fundamental issues with how asp.net operates.
Under certain circumstances - you can wind up with more that one application context executing the same global.cs - as opposed to the normal situation where its multiple instances of global.cs firing in the same application context. While in the same application context the memory is shared - when you get more than one application context - no form of semaphore lock will be successful.
Yes ideally - there needs to be either an explicit rule of 1 DNN app per application pool - and NO other applications or sub applications in the same pool - but of course thats bordering on impossible to enforce.
One other option that we looked at was taking a hold on something other than a semaphore - and by something other - what we considered was something like an explicit lock on a SQL record - but of course this also adds complications like - deadlocks on a failed server restart or the like.
Bottom line - a double bind semaphore on a static object will deal with anything in the same application context
Note: while not fully test - i believe you can also get weird forms of this in TIMER mode - and let not even get started on webfarm issues
Westa
Ben ZhongJune 26, 2014 at 7:05 AM
Hi Stephen,
im not very clear about "same app pool", is it mean:
1. different websites(different site folder, different database) but same app pool;
2. different websites(but point to same site folder and database) but same app pool;
3. different websites(only 1 is DNN instance, other ones are other application) but same app pool.
which one is correct?
Stephen LimJune 26, 2014 at 3:17 AM
Hi Ben,
When you review the code, can you please take a look and see if the scheduler can also incorrectly run simultaneously (even in Timer mode) when there are multiple IIS sites binded to the same App Pool? I understand it's a best practice to isolate each DNN instance with its own dedicated App Pool, but if you search Google, you'll find many examples of cases where people observed the scheduler running multiple times simultaneously and causing grief. It may very well be because everyone has a slightly different IIS setup (sometimes by accident or the hosting company makes that decision). It may be worthwhile to prevent the issue in the code in the first place so it causes less confusion especially for users on shared hosting who don't necessarily see how the App pool is being configured behind the scene. Here are a few examples:
http://old.datasprings.com/news/blog/postid/11/dotnetnuke-scheduler-running-multiple-instances-
http://www.dnnsoftware.com/forums/forumid/118/threadid/304221/scope/posts
Thanks!
On systems where the scheduler mode is set to REQUEST_MODE - there is a possibility that the code that is triggered on the httprequest handler - to attempt to start the scheduler - can be executed more that once.
http://www.dnnsoftware.com/forums/forumid/216/threadid/497534/scope/posts discusses the issue at some length - including debug logs that demonstrate it happening.
But in essence - the call to RunSchedule() is made every time a request is made for a page on a dnn site. When the application pool is starting up - it is currently possible for the SchedulingProvider.ReadyForPoll test to return true more than once. If there are multiple new page requests hitting the server during the application pool startup.
This is a common occurrence on low demand dnn sites that dont utilize any sort of keep alive tool - when those sites are visited by a crawler. For example.
Site is offline
crawler targets the site - potentially firing off a number page requests at all but the same time.
Asp.net sees first request and starts the process of bringing the dnn application into the application pool.
while this is happening other requests start arriving.
During this phase the DotNetNuke Http Application handler starts catching Application_BeginRequest events.
One of the actions this triggers is the call to RunSchedule.
Inside RunSchedule - the systems tests to see if SchedulingProvider.ReadyForPoll is true - which basically it is taking to mean the scheduler is not currently running.
If this is the case is then starts the process of creating a new SchedulingProvider.Instance() thread - which ultimately brings the scheduler online.
BUT the problem is - there is potentially a considerable lag between when SchedulingProvider.ReadyForPoll is tested and found to be true - and when the system has got the scheduler thread running and SchedulingProvider.ReadyForPoll is eventually put into a start that allows it to return a value of false.
IF there are multiple httprequest arriving at the server during this phase - see the forum thread for example from debug log test - in the time between when the first request tests ReadyForPoll - other requests can also have tested ReadyForPoll - and all get to see the result of the test as TRUE.
At which stage - they each of those page requests all try to start triggering the process of starting up a schedulerprovider background worker thread.
On my test system - i was easily able to trigger 3 or 4 separate instances of the schedulerprovider on a regular basis - simply by manually starting 4 browser pages at the same time – on a system being hit by a search crawler - that often fire off 5-10 or more requests at once - it would not be unlikely for that to be the case .... AND ironically may be the cause of a number of different system event log errors and forced application pool restarts that clients have reported in the past that seem to always relate to sites being hit be web crawlers.
To fix the issue - I am suggesting that a lock() statement on a static local object should be added - this will prevent more than one instance of the schedulerprivider ever being created
If the file - DNN Platform\Library\Common\Initialize.cs
-------------------------------------------------------
private static readonly object _schedulerStarting = new object();
public static void RunSchedule(HttpRequest request)
{
if (!IsUpgradeOrInstallRequest(request))
{
try
{
lock (_schedulerStarting)
{
if (SchedulingProvider.SchedulerMode == SchedulerMode.REQUEST_METHOD && SchedulingProvider.ReadyForPoll)
{
Logger.Trace("Running Schedule " + (SchedulingProvider.SchedulerMode));
var scheduler = SchedulingProvider.Instance();
var requestScheduleThread = new Thread(scheduler.ExecuteTasks) { IsBackground = true };
requestScheduleThread.Start();
SchedulingProvider.ScheduleLastPolled = DateTime.Now;
}
}
}
catch (Exception exc)
{
Exceptions.LogException(exc);
}
}
}