@Shazwazza

Shannon Deminick's blog all about web development

FCN (File Change Notification) Viewer for ASP.NET

December 19, 2016 05:04

This is a follow up post about another article I previously wrote about FCN on ASP.NET and how it affects your application, performance, etc…

imageSince that post, I’ve discovered a few more tidbits about FCN and application restarts and have decided to release a Nuget package that you can install to generate a report of all file/directory change monitors that ASP.NET creates.

The code for the FCN report generator lives on GitHub and you can install it via nuget:

PM> Install-Package FCNViewer

Once that is installed you’ll get a readme on how to enable it, basically just this in your WebApi startup/route config:

config.Routes.MapFcnViewerRoute();

Then you can navigate to /fcn and you’ll get a nice report showing all of the files/folders being watched by ASP.NET.

FCN modes & app domain restarts

I won’t go into full details about all the FCN modes again (you can find those details on my previous post) but I want to provide some info about “Single” vs “NotSet” (the default).

When the default (“NotSet”) is used, ASP.NET will create a directory monitor inside of every folder that the ASP.NET runtime accesses. This includes accessing a folder to return an asset request like CSS or JS, or rendering a razor file or reading from a data file. Basically it means ASP.NET has the potential to create a directory monitor for every directory you have in your site. The good part about the default mode is that each directory monitor will have it’s own buffer to manage the files it is watching so there’s less chance these buffers can overflow. The problem with the default is file system performance especially if you are hosting your site from a remote file server – the more directory monitors that are created the more strain will be put on your file server/network which could cause unwanted app domain restarts.

When set to “Single”, ASP.NET will create one directory monitor and this single directory monitor will be used to monitor all folders that ASP.NET accesses. This means there is a single buffer that is used for monitoring all of the files and folders. This buffer is larger than the buffer used for each individual monitor created with the default is used, however the problem with “Single” is that it means that if you have tons of folders there’s a higher chance this buffer can overflow which could cause unwanted app domain restarts. The good part about “Single” is that its much better for performance for the file system especially when you are hosting your site from a remote file server.

As you can see there’s no perfect scenario and both can cause unwanted app domain restarts. These restarts will end up in your logs with very peculiar reasons like:

Application shutdown. Details: ConfigurationChange
_shutDownMessage=Overwhelming Change Notification in 
    HostingEnvironment initiated shutdown
    CONFIG change
    Overwhelming Change Notification in D:\inetpub\test
    CONFIG change
    Change Notification for critical directories.
    Overwhelming Change Notification in bin
    Change Notification for critical directories.
    Overwhelming Change Notification in App_LocalResources
    CONFIG change
    CONFIG change
    CONFIG change
    CONFIG change

Or other strange “ConfigurationChange” reasons that might tell you that all sorts of files have been changed – but you definitely know that is not the cause. The cause of this is very likely FCN issues.

Files outside of the web root

An interesting bit of info is that ASP.NET wont create directory/file monitors for files that exist outside of the web root even if ASP.NET accesses them. I’d recommend where possible that you store any sort of cache files, data files, etc… that ASP.NET doesn’t need to serve up as static file requests outside of the web root. This is actually the default way of working in ASP.NET Core too. As an example, you may have heard or use Image Processor which creates quite a substantial amount of cache folders for processed images. These files by default exist in /App_Data/cache and since that is in the web root, ASP.NET will create a directory watcher for each folder in there … and there can be literally thousands if you use Image Processor extensively! If you are using FCN “Single” mode and have that many folders in the Image Processor cache, there’s a good chance that you’ll have app domain restart issues - though changing it to “NotSet” could result in serious file system performance issues. The good news for Image Processor is I created a PR to allow for storing it’s cache outside of the web root: https://github.com/JimBobSquarePants/ImageProcessor/pull/521 so that functionality should be available in an upcoming release soon.

FCN & ASP.NET CacheDependency

Another thing to be aware of is that if you use ASP.NET’s cache (i.e. HttpRuntime.Cache or HttpContext.Cache or MemoryCache) and you create cache dependencies on specific files, this effectively goes straight to the underlying FCN engine of ASP.NET and it will create these same file/directory monitors that ASP.NET creates by default … even if these files are stored outside of the web root! So if you are creating a ton of CacheDependency objects, you will be creating a ton of FCN directory/file monitors which could be directly causing FCN issues with app domain restarts.

The FCN Report

At least with this report viewer you can see how many files and folders are being watched. It’s important to know that these watchers are created lazily whenever ASP.NET accesses files so on first load you might not see too many but if you start browsing around your site, you’ll see the number grow.

You can also modify the default route by using an overload:

config.Routes.MapFcnViewerRoute("myfcnreport");

This can hopefully help debug these strange restart issues, give you an idea of how much ASP.NET is actually watching and show you how the FCN Modes affect these monitors.

All about ASP.Net File Change Notification (FCN)

October 15, 2015 10:45

There’s a chance you might not have heard of FCN (File Change Notification) in ASP.Net and there’s an even bigger chance you didn’t realize how much it might affect you.

What is FCN in ASP.Net?

As you know ASP.Net monitors a few files/folders such as the ~/web.config and ~/App_Code and will restart your app domain when it detects changes. This is part of FCN in ASP.Net but it goes much deeper than that. There are a few other files & folders that ASP.Net monitors which will also cause an app domain restart: bin, App_WebReferences, App_GlobalResources, App_Code, Global.asax, and others. However what you might not realize is that ASP.Net actually monitors every single folder (+ files) in your web app!

Update 29/07/2016! I found a nice MS article about FCN here. I’ve been looking for more resources about this since a few new issues have been cropping up and I’m wondering if another MS update has caused another problem. Recently we’ve seen an increase in the error: “Overwhelming Change Notification in …” which has everything to do with FCN. I’ve also added a few links to the bottom of this post.

Why do I care?

If you have a web app that contains a lot of folders there is a good chance that the performance of your file system is affected in one way or another. To give you an example of how many file system watchers ASP.Net generates I created a Razor view (.cshtml)  (which you should definitely test on your own site!) to display what is actually happening behind the scenes. Here’s the output for the first run page from the Visual Studio 2015 MVC template site:

image

The table above lists all of the “DirectoryMonitor” instances and the folder they are attached to along with all of the “FileMonitor” instances attached to that DirectoryMonitor. It’s worth nothing that these are not simply just .Net’s FileSystemWatcher, but some sort of native windows IO using a delegate called NativeFileChangeNotification. The above table doesn’t seem too scary but these monitors are not static either, they grow with every directory and file that is accessed in your web app.  For example, if I navigate to these pages: /Home/About, /Home/Contact, /Account/Register, /Account/Login and go back to view this table it looks like:

image

Things start to get interesting when you have a web application that has a lot of folders. Some examples of this might be:

  • You are using a dynamic image processor such as Image Resizer or Image Processor since these will create a lot of folders based on hashes to store these dynamic images
  • Maybe you have a lot of members/users on your site and you have one or more folders for each one
  • Maybe you use nodejs or bower and you store these generated folders in your web root and references the assets directly in those folders … there can be tons of folders in bower_components and node_modules
  • You could be using a web framework or CMS like Umbraco or Orchard (I’m sure there are plenty of others) that contain quite a lot of folders by default

Here’s the truncated result of an Orchard site after visiting the admin area, creating a page and displaying it:

image

Whoa!

ASP.Net’s FCNMode

At some stage in ASP.Net’s lifetime somebody must have complained about this to Microsoft and they released a hotfix (seen here: https://support.microsoft.com/en-us/kb/911272). Then I can only assume that other people complained about this and with .Net 4.5 a new configuration setting appeared: FCNMode. This documentation explains a little about what each option does:

  • Default - For each subdirectory, the application creates an object that monitors the subdirectory. This is the default behavior
  • Disabled - File change notification is disabled.
  • NotSet - File change notification is not set, so the application creates an object that monitors each subdirectory. This is the default behavior.
  • Single - The application creates one object to monitor the main directory and uses this object to monitor each subdirectory.

Unfortunately these docs don’t tell us the whole story. It’s obvious that Default/NotSet are the same thing and are the default. Disabled is fairly clear but what it doesn’t mention is that if you set it to Disabled, this will disable all FCN for your web app, so if you change the contents of /bin or /App_Code, the site will not restart. However, Disabled still means that the web.config file is monitored so if you use this setting you can still bump your web.config to restart your site.

What exactly is “Single” though?

The folks over at DNN seem to have quite a bit of experience with FCN and this article seems to be the only place that actually explains “Single” mode correctly:

“FCNMode creates a monitor object with a buffer size of 4KB for each folder. When FCNMode is set to Single, a single monitor object is created with a buffer size of 64KB. When there are file changes, the buffer is filled with file change information. If the buffer gets overwhelmed with too many file change notifications an “Overwhelming File Change Notifications” error will occur and the app domain will recycle. The likelihood of the buffer getting overwhelmed is higher in an environment where you are using separate file server because the folder paths are much larger.”

If I change fcnMode=”Single” in my web.config for the same Orchard site above, the results are:

image

That’s quite a bit less!

I’m sure there are pros to using “Default” instead of “Single” but I’m not actually sure what they are.

Real world problem

The circumstance where “Default” FCN mode becomes a real problem is when you have a website that is running off of a remote file share.  As noted in Shawn Walker’s DNN article mentioned above:

“The likelihood of the buffer getting overwhelmed is higher in an environment where you are using separate file server because the folder paths are much larger.”

I can’t actually see a performance difference between Default or Single when running locally with an SSD HD, but I have seen some big issues running with Default when hosting on a remote file server:

  • Constant app domain restarts – I’ve seen constant app domain restarts even when a site is just serving requests without any IO activity apart from just reading. And by ‘Constant’ … I mean every few seconds all day long.
  • File server performance suffers severely – When multiple sites are active and are being hosted on a remote file server with Default FCNMode, the writing performance of the file server is drastically degraded even though when monitoring IOPS there doesn’t appear to be any issues

To solve this issue we changed fcnMode=”Single” in the machine.config so that all sites would effectively use “Single”… and the result was instant: No more constant app restarts, file server performance was instantly back to normal. And as far as I can tell, there has been no downside to running FCNMode in Single.

So I really wonder what the up-side of Default is when it seems that running in Single mode is perfectly stable running on any hosting environment… ?

As far as I can tell, this won’t be an issue with aspnet5 … let’s just hope!

More info?

Here’s a list of helpful links about FCN in ASP.Net:

Updated 29/07/2016! – More links I’ve discovered. Turns out this has been an issue for IIS + ASP.Net for quite some time with various older hotfixes, some of these new links might shed some light on the particular problem you might be having.