This is a follow up post about another article I previously wrote about FCN on ASP.NET and how it affects your application, performance, etc…
Since that post, I’ve discovered a few more tidbits about FCN and application restarts and have decided to release a Nuget package that you can install to generate a report of all file/directory change monitors that ASP.NET creates.
The code for the FCN report generator lives on GitHub and you can install it via nuget:
PM> Install-Package FCNViewer
Once that is installed you’ll get a readme on how to enable it, basically just this in your WebApi startup/route config:
Then you can navigate to /fcn and you’ll get a nice report showing all of the files/folders being watched by ASP.NET.
FCN modes & app domain restarts
I won’t go into full details about all the FCN modes again (you can find those details on my previous post) but I want to provide some info about “Single” vs “NotSet” (the default).
When the default (“NotSet”) is used, ASP.NET will create a directory monitor inside of every folder that the ASP.NET runtime accesses. This includes accessing a folder to return an asset request like CSS or JS, or rendering a razor file or reading from a data file. Basically it means ASP.NET has the potential to create a directory monitor for every directory you have in your site. The good part about the default mode is that each directory monitor will have it’s own buffer to manage the files it is watching so there’s less chance these buffers can overflow. The problem with the default is file system performance especially if you are hosting your site from a remote file server – the more directory monitors that are created the more strain will be put on your file server/network which could cause unwanted app domain restarts.
When set to “Single”, ASP.NET will create one directory monitor and this single directory monitor will be used to monitor all folders that ASP.NET accesses. This means there is a single buffer that is used for monitoring all of the files and folders. This buffer is larger than the buffer used for each individual monitor created with the default is used, however the problem with “Single” is that it means that if you have tons of folders there’s a higher chance this buffer can overflow which could cause unwanted app domain restarts. The good part about “Single” is that its much better for performance for the file system especially when you are hosting your site from a remote file server.
As you can see there’s no perfect scenario and both can cause unwanted app domain restarts. These restarts will end up in your logs with very peculiar reasons like:
Application shutdown. Details: ConfigurationChange _shutDownMessage=Overwhelming Change Notification in HostingEnvironment initiated shutdown CONFIG change Overwhelming Change Notification in D:\inetpub\test CONFIG change Change Notification for critical directories. Overwhelming Change Notification in bin Change Notification for critical directories. Overwhelming Change Notification in App_LocalResources CONFIG change CONFIG change CONFIG change CONFIG change
Or other strange “ConfigurationChange” reasons that might tell you that all sorts of files have been changed – but you definitely know that is not the cause. The cause of this is very likely FCN issues.
Files outside of the web root
An interesting bit of info is that ASP.NET wont create directory/file monitors for files that exist outside of the web root even if ASP.NET accesses them. I’d recommend where possible that you store any sort of cache files, data files, etc… that ASP.NET doesn’t need to serve up as static file requests outside of the web root. This is actually the default way of working in ASP.NET Core too. As an example, you may have heard or use Image Processor which creates quite a substantial amount of cache folders for processed images. These files by default exist in /App_Data/cache and since that is in the web root, ASP.NET will create a directory watcher for each folder in there … and there can be literally thousands if you use Image Processor extensively! If you are using FCN “Single” mode and have that many folders in the Image Processor cache, there’s a good chance that you’ll have app domain restart issues - though changing it to “NotSet” could result in serious file system performance issues. The good news for Image Processor is I created a PR to allow for storing it’s cache outside of the web root: https://github.com/JimBobSquarePants/ImageProcessor/pull/521 so that functionality should be available in an upcoming release soon.
As it turns out, Virtual Directories in IIS are treated differently as well! Any fcnMode that you configure has no affect on files within a Virtual Directory except for if you disable FCN. So if you are thinking of using “Single” and are using Virtual Directories, think again. The only FCN setting that affects Virtual Directories is “Disabled”. Here’s the actual code that creates directory watches for Virtual Directories: https://referencesource.microsoft.com/#System.Web/FileChangesMonitor.cs,0820c837402228ef and as you can see, if it’s not disabled, it will be creating a directory watcher for every sub directory in the Virtual Directory. This problem is compounded if the Virtual Directory is pointing to a network share.
FCN & ASP.NET CacheDependency
Another thing to be aware of is that if you use ASP.NET’s cache (i.e. HttpRuntime.Cache or HttpContext.Cache or MemoryCache) and you create cache dependencies on specific files, this effectively goes straight to the underlying FCN engine of ASP.NET and it will create these same file/directory monitors that ASP.NET creates by default … even if these files are stored outside of the web root! So if you are creating a ton of CacheDependency objects, you will be creating a ton of FCN directory/file monitors which could be directly causing FCN issues with app domain restarts.
The FCN Report
At least with this report viewer you can see how many files and folders are being watched. It’s important to know that these watchers are created lazily whenever ASP.NET accesses files so on first load you might not see too many but if you start browsing around your site, you’ll see the number grow.
You can also modify the default route by using an overload:
This can hopefully help debug these strange restart issues, give you an idea of how much ASP.NET is actually watching and show you how the FCN Modes affect these monitors.