Catastrophic Faillure and Delay Notification issues with WCF

Before adding some new functionality to a customer’s existing site, my development runtime environment would hang, throw a timeout exception and after about a minute later a Catastrophic Failure message would appear. I also noticed a Visual Studio icon appeared in my system tray giving me a Delay Notification message.




So I set break points at the entry points of the possible services having an issue but the code would never get executed.  After turning on Service Tracing and sifting through tons of information, I found the issue and saw this in the Tracing Viewer (bold text below is key):
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
            <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
                        <EventID>458797</EventID>
                        <Type>3</Type>
                        <SubType Name="Information">0</SubType>
                        <Level>8</Level>
                        <TimeCreated SystemTime="2011-04-15T19:37:16.9652192Z" />
                        <Source Name="System.ServiceModel" />
                        <Correlation ActivityID="{ac42c2e0-b83a-4380-9586-fb894c291ec9}" />
                        <Execution ProcessName="WebDev.WebServer" ProcessID="1788" ThreadID="8" />
                        <Channel/>
                        <Computer> </Computer>
            </System>
            <ApplicationData>
                        <TraceData>
                                    <DataItem>
                                                <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information">
                                                            <TraceIdentifier>
                                                                        http://msdn.microsoft.com/en-US/library/System.ServiceModel.Security.SecurityPendingServerSessionActivated.aspx
                                                            </TraceIdentifier>
                                                            <Description>
                                                                        A pending security session was activated by the server.
                                                            </Description>
                                                            <AppDomain>76f9f50a-1-129473695859863966</AppDomain>
                                                            <ExtendedData xmlns="http://schemas.microsoft.com/2006/08/ServiceModel/SecuritySessionTraceRecord">
                                                                        <SessionId>urn:uuid:91ce3aae-eac4-432b-ae7f-8ea401759e84</SessionId>
                                                                        <ListenAddress>http://localhost:18309/ProfileService.svc</ListenAddress>
                                                            </ExtendedData>
                                                </TraceRecord>
                                    </DataItem>
                        </TraceData>
            </ApplicationData>
</E2ETraceEvent>
<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
            <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
                        <EventID>524337</EventID>
                        <Type>3</Type>
                        <SubType Name="Information">0</SubType>
                        <Level>8</Level>
                        <TimeCreated SystemTime="2011-04-15T19:37:16.9652192Z" />
                        <Source Name="System.ServiceModel" />
                        <Correlation ActivityID="{80b77366-a1cb-4354-9f91-489701444372}" />
                        <Execution ProcessName="WebDev.WebServer" ProcessID="1788" ThreadID="8" />
                        <Channel/>
                        <Computer> </Computer>
            </System>
            <ApplicationData>
                        <TraceData>
                                    <DataItem>
                                                <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Information">
                                                            <TraceIdentifier>
                                                                        http://msdn.microsoft.com/en-US/library/System.ServiceModel.ServiceThrottleLimitReached.aspx
                                                            </TraceIdentifier>
                                                            <Description>
                                                                        The system hit the limit set for throttle 'MaxConcurrentSessions'. Limit for this throttle was set to 10. Throttle value can be changed by modifying attribute 'maxConcurrentSessions' in serviceThrottle element or by modifying 'MaxConcurrentSessions' property on behavior ServiceThrottlingBehavior.
                                                            </Description>
                                                            <AppDomain>76f9f50a-1-129473695859863966</AppDomain>
                                                </TraceRecord>
                                    </DataItem>
                        </TraceData>
            </ApplicationData>
</E2ETraceEvent>

The first Trace Event is simply the call of the service and the second Trace Event is the “notification” that you reached the MaxConcurrentSessions limit.  You tie these together by the ProcessID (1788).

After more searching through articles and forums, apparently what is happening is that once this limit is reached, subsequent requests for this service are marked as inactive and queued.  The system will then process all of the active requests including any new ones marked as active prior to processing the inactive requests.  Once the active service requests are processed, the inactive requests would be processed if they haven’t already hit the timeout limit. 

This application’s timeout limit was 1 minute.  These services are synchronous so the front end reference was waiting for a reply that was marked as inactive and hence the hanging effect and the timeout exception.  The Catastrophic Failure and Delay Notification message was due to the hosting process (ASP.NET Development Server) Visual Studio was using to host the requests in local debug mode.  Once I received the timeout error, I would close the browser but the hosting process was still trying to complete these inactive requests.  And because I had stopped debugging and the runtime environment, the application would then give me the Catastrophic Failure and Delay Notification message.

The easiest solution (which the client implemented) was increasing the service throttling setting of maxConcurrentCalls, maxConcurrentSessions and maxConcurrentInstances.