A corrupted entity
I ran into a problem today with my client which frankly had me stumped. For some reason that I have yet to determine (though possibly related to a broken install of Update Rollup 2), the account entity got corrupted. We discovered this when somebody attempted to publish their changes to the account entity and ran into a “red” error dialog: “An error has occurred…please contact your system administrator”. This error was replicated if we tried to publish, delete any attributes from the entity or export the entity customizations. Not much scope for fixing the situation!

Troubleshooting…
After studying the server trace logs (thanks to CrmDiagTool4 and Michael Hoehne’s Trace Log Viewer for easing this process), the situation wasn’t much clearer. The underlying error was:

Request URL: http://localhost/MyOrgName/AppWebServices/SystemCustomization.asmx
Stack Trace Info: [NullReferenceException: Object reference not set to an instance of an object.]
at Microsoft.Crm.ObjectModel.OrganizationUIService.LabelLoaderAllLanguages.LoadMetadataLabel(Int32 entityType, String attributeName, ExecutionContext context)

I also did a SQL Trace using SQL Profiler on the server, but this didn’t really provide anything new at all.

So, when faced with a problem I can’t resolve myself, I turned to Google for help. I managed to find KB article 947096 which seemed very near to my problem, but sadly none of the resolution steps would help me. I then tried importing a known good copy of the account entity customizations but that failed too. At this point I looked back at the Trace Log and tried to determine what was going on “under the hood” prior to the failure…

It was clear from the logs that CRM was trying to pull in LocalizedLabel information for my entity attributes and form controls, however I couldn’t quite fathom out which attribute was causing the NullReferenceException (it seems that the last statement in the logs – MetadataProcessObject.ExecuteQuery – was successfully completing its SQL command and the following statement which was failing was not logging any parameters and therefore didn’t provide me with a “smoking gun”). In fact there were many calls to both CrmDbConnection.InternalExecuteReader and MetadataProcessObject.ExecuteQuery so clearly there was an interative loop going on. I didn’t know which mechanism within the CRM server black box was selecting the list of attributes to process, so I looked further back and found that this loop was likely being “triggered” by the FormXML property on the OrganisationUI view.

At this point everything clicked into place: the problem, somehow, was with the published version of my form (some of you may have realised this long ago). This form xml was being re-evaluated (for some reason) prior to every publish, export or delete request and then failing some integrity or logical check. Re-importing a known good copy would not help, since it would need to be published to take effect. I only had one option left – to break all the rules and manually amend the database in order to correct the corrupted FormXml property…

So here’s what I did:

Resolution
Took a backup of the current formXml property by running the following SQL statement within SQL Server Management Studio in the “_MSCRM” database (then right click on results tab and click “save as”…):

select organizationui0.ObjectTypeCode organizationui0.FormXml as 'formxml',
from OrganizationUI as organizationui0
where (organizationui0.InProduction = 1 and (organizationui0.ObjectTypeCode = 1))

Next I found my “good” copy of the customizations file. I identified the start and end node that was required in order to match the same format as the existing formXML property (the starting tags were <form><entity…>). Next I copied this xml to a new file and removed all tabs and new lines, to give a single continuous string.

I saved this xml file onto my CRM server then went back to SQL Management Studio and ran the following SQL to import this long string directly into the SQL table:

declare @xmlText varchar(max)
select @xmlText =(select * from openrowset (bulk 'c:\amendedformxml.txt',SINGLE_CLOB)x)
update dbo.OrganizationUI
set FormXml = @xmlText
where (InProduction = 1 and (ObjectTypeCode = 1))

This effectively “forced” my old formXML into a published state and magically fixed my problems. Further, once I clicked “Publish” following these steps, the formXML was consequently re-evaluated again and the hitherto unpublished customizations become published (as expected) and therefore effectively no information had been lost by importing the older xml.

Conclusion
So I still don’t know what caused this issue… I’d love to know if anyone else has had a similar situation. I’m not entirely comfortable with the way I had to fix the problem either, but then sometimes your hand is forced into doing things you don’t want to do. I’ll monitor the situation and let you all know if there are any repercussions. In the meantime, the usual caveats apply – the steps above are definitely unsupported and could definitely render your CRM installation inoperable, so use the steps above at your own risk!