Severe Issue with macro caching under heavy load

Greetings!

Please let me know if this should go into a different forum, or if this needs to be posted to the issues list in codeplex. Thank you in advance for your time in reading and helping me with this. This is a major issue that we are having, affecting the performance of our production web site (we've had to turn Umbraco caching completely off, and can sometimes get thousands of requests per hour).

I'm hoping I'm an idiot and there's one setting I'm forgetting to set that will fix all of this!

We are using Umbraco for a fairly busy website, and we are encountering problems when caching is enabled on our macros - specifically on macros that are .NET user controls. The short version of the problem is: When the website is under moderately heavy load (20+ users at the same time reloading the same page), the content that is cached gets messed up. Sometimes it doesn't load at all, so all of areas in the page where the macros are supposed to be are blank; sometimes the macro content seems to overwrite itself - we'll see data repeated in the same control.

We are using a fairly-heavily customized installation of Umbraco 4.0.3, using ASP.NET 3.5. We have about 50 user controls (almost all of them pull data from a DB) that we use at many places within the site. I was concerned that maybe we broke this when customizing it, or maybe our user controls were doing something out of the ordinary, but I was able to recreate the problem using a stock installation of Umbraco (installed via Web Installer), and a project with a single user control that just queries the Northwind database (Orders table) and writes the data to the page. Of course, the macro that is this user control has caching enabled. Changing any of the cache settings (time, cache by page/personalized) does not seem to change the outcome.

To simulate the load, I'm using a load test within Visual Studio Team Edition. If I set the number of concurrent users at 20 or more hitting a page that only contains a single macro/user control (the one that queries the Northwind database) with caching enabled, I see this issue. I don't see the issue if I just try to hit the page as hard as I can (by hitting refresh in the browser repeatedly, or by opening a bunch of browser windows and refreshing them). 

Our best guess is that the problem is concurrency issues storing content in the cache - normal multi-threaded types of issues. For example:

1. While under heavy load, request #1 is being processed, it checks the cache to see if the content for macro A is stored in it; it does not find macro A, so it starts rendering macro A.
2. Before step #1 is complete, request #2 is being processed. It also checks the cache to see if the content for macro A is stored; it doesn't find macro A (because step #1 hasn't completed yet), so it starts rendering macro A.
3. Request #1 completes rendering of macro A and writes it to the cache.
4. Before step #3 is complete, request #2 completes rendering of macro A and also writes it to the cache, resulting in the content in the cache being some overlapped combination of the two renderings.

Some more details:

• The "Maximum Worker Processes" setting in IIS 7 is set to 1 (that's the first thing I checked)
• Our production web site is using LINQ-to-SQL to access the data for the user controls/macros, but for simplicity, my test project is just using a SqlDataReader. 
• I've tested having a user control that is just a GridView hooked to a SqlDataSource that just dumps the Orders table from the database. When I test using this, the problem I most often have is that the macro fails to load occasionally. I then created a new user control that uses the same query, but uses a SqlDataReader (in the code) and manually outputs the data to a PlaceHolder in the user control. This user control more accurately reflects a lot of what we're using in the production web site. The problem I most often have with this method is that the data often overlaps or repeats itself. For example, the user control is supposed to write the first 40 rows from the DB table to the page, but when the problem occurs, I get the first 40 rows correctly, followed by 10-15 of the same rows, with the last row being cut off mid-row. (These same rows are from the beginning of the page, as if the user control got rendered again and started over from the top.)
• When the problem of overlapping occurs, the overlapped content stays in the cache until the next time the cache expires. (I usually set the cache very low, like 5 or 10 seconds, for testing, but this problem still occurs when the cache is set higher (120 is our preferred timeout).)
• Compiling/changing configuration to be in debug or release mode does not seem to affect the problem.
• Changing the Xml Cache settings (enabling or disabling) does not seem to affect the problem.
• I've been able to recreate this on many different machines, including my development server (a fairly well-powered machine with Windows 7), a desktop machine running Windows 2008 server, and on our production server, a decent machine also running Windows 2008 server. The biggest thing in common that all of the machines I've recreated this on have is that they're all running IIS7. I have not tried recreating this in IIS 6.