If you have already downloaded the April CU, STOP, do not install it. If you haven’t yet, well don’t worry because it was pulled
Overview:
I have been working with Microsoft over the last two weeks on issues we were experiencing after installing the April 2012 CU in all of our production environments.
We started receiving reports the Monday after installing the update that document libraries and lists were no longer present on some sites. There were also reports of icons changing for document libraries to link icons.
At first with this type of request you immediately think that this has to be some type of user problem. So I investigate the report and find that the item they are complaining about is in the recycle bin.
So I figure the user just deleted the list on accident. So I try to restore the list and receive an error that the list already exists. I go looking through the site via gui and cannot find this list anywhere. I dropped to power shell and could not find the list that way either.
The next twist is we pulled an audit report for this list in question and find that users were making changes in the list on Friday. The recycle bin shows this list was deleted on Tuesday. So how does a user update content in a list that has been deleted.
By this time we are now having multiple reports coming in from different farms and site collections of the same behavior. Only now we are seeing this behavior reported on lists, document libraries, and even sub sites.
So we are seeing the following behavior on our farms
Missing Items - Document libraries, Lists, Sub sites disappear from view on sites. Cannot view them in gui or powershell
Icons Change - On link lists the icons would change to a link icon, when you click on the link you receive a 404 or 503 error and content fails to load. When I go to the list/library settings page for that list the URL for the item is incorrect
Investigation:
I confirmed that reported sub sites just threw a 404 error like they have been deleted. We find on some items a 404 is thrown, some throw http 500, and then some throw an unexpected error with correlation id.
I dug a bit further into the issue and started finding that the items affected by these issues had been deleted from the site and then recreated with the same name. The deleted item was still in the 1st stage recycle bin.
If I try to restore the item in the recycle bin it throws an error that the item already exists. Problem is that you cannot see the item to delete it. Trying to delete from powershell failed as well as it could not find the item.
There were no errors in the upgrade logs from the upgrades or any further evidence that I could find related to these missing items.
I contacted Microsoft on this issue as this clearly looks like an issue with the CU.
We spent the first couple days with different engineers going through the typical info gathering process.
We were not making much progress and it seemed it was trying to be directed at customizations we had in our environment.
My colleague remembered a strange issue in Sharepoint 2007 where permissions would get hosed and the item would not appear due to having no permissions assigned.
I looked into the broken lists and found that they did have broken inheritance on them. So to test this theory I took a backup of one of the broken sites prior to the update being applied and restored into one of our test environments.
The list that was missing in production was present on the restored site.
I then set inheritance on this item in question back to inherited, and ran the upgrade on the content database.
This time after the upgrade was complete the list in question was still visible on the site
Steps to Repro
· Take a copy of the affected database prior to the April CU
· Stand up the site in a test environment that has the April CU installed
· Attach your DB copy via central admin so upgrade process is not run
· Browse your site and you will see all data you expect
· Now perform the upgrade-spcontentdatabase on the content db, once this is done you will see that your document library is no longer on the site.
After working through the issue for several more days with various escalation engineers we finally were able to get them to admit a problem in the CU.
I worked with an engineer, Sojesh Sreelayam, who had seen some similar behavior before in an unrelated case. I name him specifically as he was the tech that figured out the issue we were having and came up with the solution.
So he started going down the path of looking at the scope id's and perms table for the affected items and confirmed this was in fact the same behavior he saw before..
The problem ended up being that the scope ID had been removed from the perms table in the content database for the affected items. To resolve they had to restore the scope ID to the perms table for each content database on the affected items. In order to do this we had to have the databases restored prior to the updates. Then the missing data was pulled from the database prior to it being broken.
There were some additional items found where items in a library that were not in the recycle bin were affected and were no longer visible on the site. These could not be repaired in the same way and required additional steps to be taken to correct the files.
Resolution:
In the end, to resolve this issue it required SQL scripts to be run against the production content databases. All direct database edits were done by the MS tech and were approved form the product team to resolve the issues we had.
I have never been involved with this type of issue related to a CU or any other update for that matter. It required a large amount of time and a crit sit case with MS to resolve, but I will say that once we were able to get them to agree that it was caused by the CU things moved along much quicker and they resolved all identified issues.
Whether we have further items reported is yet to be seen, but as of today the case is closed and it’s on to the next item.
Interesting thing a business analyst sent me a few days ago, http://bjarnegram.wordpress.com/2012/05/29/sharepoint-2010-cu-april-2012-retracted/
The April CU has been pulled due to technical problems - http://support.microsoft.com/kb/2598151
