I had to troubleshoot a problem over this weekend that arouse after an IIS server farm (Windows 2003 SP2, IIS 6.0, Application Center, 4 servers) had its storage moved from internal disks to external Storage (NAS, Netapp FAS 2020 using CIFS protocol). When running with NAS, the application would lose session state frequently and that was a big problem. BTW, the application writes to its own directories under normal operations.

Microsoft Internet Information Server creates a “change notification” for each web application it has, so when the folder changes it can recompile/restart the application. This sounds like a good idea and it works like a charm with local disks, the standard deployment scenario for web applications. Indeed, I wrote a program that is able to watch for more than one thousand notifications at the local disk.

However, things break down when you use network storage because both the client and the server have a default limit of 50 “SMB slots” for communication between client and server. Those slots are used (and reused) for every file operation, but each notification “hogs” one of them. My test program confirms this by opening just 50 notifications with a network share. There’s a solution (or workaround): increase this limit both at the file server and the client (web server in this case). This is well documented at http://support.microsoft.com/kb/843584.

So problem solved? Maybe. I did some research and I concluded that although “change notifications” are fine with local storage, it’s a somewhat brittle mechanism over a network and you should avoid it whenever possible, either directly in your own code or indirectly, such as when you use IIS with network storage. This is why:

1) It’s a state full, resource intensive feature

Having a permanent connection between the client and the server is probably a bad idea. Elsewhere we stopped doing this: HTTP is connectionless; application servers use atomic methods (the A in ACID); databases are accessed without permanent connections. Permanent connections are now considered non-scalable, performance hogs. Change notifications require permanent connections.
BTW, Microsoft stopped using them for cache control by default in IIS starting with IIS6: http://www.microsoft.com/technet/prodtechnol/windowsserver2003/technologies/webapp/iis/remstorg.mspx. But it seems that I have no control over what Application Center and/or the ASP subsystem does regarding the use notifications for whatever use they find fit.

2) The CIFS specification say that this is a brittle mechanism
The CIFS specification clearly states that the NT_TRANSACT_NOTIFY_CHANGE command might “loose it cool” under load: “If too many files have changed since the last time the command was issued, then zero bytes are returned and an alternate status code is returned”.

3) ReadDirectoryChangesW has a 64k limit
The Windows API used for notifications, ReadDirectoryChangesW, has a documented 64k buffer limit (actually the size of the SMB package) when working over the network. This limits the amount of data a notification might send and might break the mechanism altogether.

In short, avoid change notification over the network in your code and try to prevent programs that you use from doing this. For instance, if you do have to use network storage with IIS, try to keep the application files in local storage and put only application data on the network.