Bordecal tools for FxCop documentation

The Bordecal tools for FxCop include two main components:
  1. A framework for assisting with the development of custom rules for FxCop/Visual Studio Code Analysis, and
  2. A set of custom rules for FxCop/Visual Studio Code Analysis.
The custom rules are built using the Bordecal rules framework, and they are intended partly as an example of how to use the framework. However, if you’re not interested in developing your own custom rules, but would like to use the Bordecal rules, please feel free to do so. You will need to copy both the Bordecal.FxCop.Rules.dll and Bordecal.FxCop.Sdk.dll files to the directory from which you wish to load the rules. For documentation on the rules, please see the Rules page.

For those of you are interested in developing custom rules, please read on…

The rule development framework

If you’ve ever tried your hand at developing custom FxCop rules, you’ll know that there already is a Microsoft-provided rules framework, even if there is no official SDK. The Bordecal toolkit is not meant to replace this existing framework, but rather to help address what I’ve come to consider its greatest pain points after a few years of working with it: a lack of testing tools and a rather inconvenient rule “publishing” mechanism.

To address these issues, the Bordecal rule development framework provides two distinct APIs:
  1. A testing framework to allow testing of custom rules against your choice of sample assemblies, and
  2. Rule base types that allow for specification of rule description data (e.g.: name, check ID, description) in code rather than in an embedded XML resource (as is required when inheriting from the Microsoft.FxCop.Sdk.BaseIntrospectionRule class).
It should be noted that these two APIs are completely independent of one another. If you want to use the testing framework for testing rules that inherit from Microsoft.FxCop.Sdk.BaseIntrospectionRule, this can be done without making any changes to your existing rule code.

The rule testing framework

Perhaps the single most important thing you need to know about the rule testing framework is that it is not a unit testing framework. It is explicitly intended for integrated rule testing in a manner that will allow for detection of both false negatives and false positives. The testing framework uses fxcopcmd.exe to fully execute custom rules over designated target assemblies, then compares the resulting violations to a set of expected violations. The actual violation set must match the expected violation set exactly for a rule test to pass.

Preparing for rule testing

Prior to testing FxCop rules, you will need to create two sets of inputs: the target assemblies over which the rules will be run and the set of expected violations.

For the target assemblies, you will probably want to build custom assemblies with code that covers a range of scenarios that are either “interesting” to your rule or that might create problems for your rule. I wouldn’t normally recommend using your “real” project code as a test target assembly. Instead, I would run custom rules over real code manually during rule development and use the results of manual executions to provide input for the test target development process. Production targets can be particularly useful for identifying false positives, but using them as automated test targets tends to create too much noise due to repetition of similar problems within a large code base.

For the expected violations, you will need to create FxCop report files that create your expected violations. The XML schema for the FxCop report file format can be found in the following location: <FxCop install directory>\Xml\FxCopReport.xsd. A few tips to get you started:
  1. It’s not necessary to create your expected violation files manually. The FxCop UI executable can export the results of an analysis run to a report file. Personally, I like to use this approach for creating my baseline expected results file, although I do tend to edit it by hand after the initial dump from FxCop.
  2. The test framework allows you to import as many FxCop report files as you like to build up the full expected violation set. While you could use a single report file, I tend to prefer using a single report file per rule. You can see an example of this in the source code for this project.
  3. The test framework will ignore the following data when comparing expected and actual results, so you can ignore potential differences in paths and execution times when setting up your expected results files:
    1. The directory portion of the Name attribute value of a Target element,
    2. The Path attribute of an Issue element, and
    3. The Created attribute of a Message element.
If you find it annoying to have to specify the expected violations in this way, please be aware that I’ve actually tried several alternate approaches, and that they all pretty much suck. This one has a slight advantage over the others in that some of the drudge work can be alleviated by exporting the initial report file, which is the main reason that I chose the approach over a purely code-based solution. If the consumption of SuppressMessage attributes by FxCop allowed for finer-grained exclusions, it would make for a potentially interesting expected violation flagging mechanism, but it doesn’t, so it can’t.

I have considered adding a similar attribute that would be used just for testing, but I’m not sure if it’s really worthwhile. If you use the testing framework and think that you would prefer an attribute-based approach for declaring expected violations, please vote for work item # 5487. By the way, if I do add attribute-based violation declarations, they will not replace the existing XML-based approach. Instead, the testing framework would accumulate expected violations from both the source code and any supplied XML files, much as FxCop does for violation suppressions.

Test execution

The only class in the testing framework with which you will need to interact for rule testing is Bordecal.FxCop.Sdk.Testing.FxCopRunner. When you create a new FxCopRunner instance, it will immediately run an FxCop analysis of your rules over your target assemblies. The results of this analysis run will be cached in the internal state of the FxCopRunner instance. You can then invoke the FxCopRunner.AssertActualViolationsMatchExpectedViolations instance method to verify that the actual violations for any given rule match the expected violations for that rule.
All the information that the FxCopRunner will need for both FxCop execution and later assertions is supplied to the FxCopRunner constructor. There are three constructor overloads which allow providing this information is slightly different formats. However, regardless of which constructor you use, you will need to provide the following information:
  1. The path to fxcopcmd.exe. In the source code for this project, you will see that I have hard-coded "C:\Program Files\Microsoft FxCop 1.36\FxCopCmd.exe". I wouldn’t really recommend this for your own rule tests. Instead, I would suggest that you copy the entire FxCop folder for whatever FxCop version you are targeting into your solution tree, and use its contents for both assembly references and rule testing. The only reason that I haven’t done this for the Bordecal.FxCop.Rules project is that the FxCop license does not seem to allow for distribution of the FxCop binaries in this manner.
  2. The path(s) to the assembly or assemblies containing rules that you wish to test.
  3. The path(s) to the sample target assembly or assemblies over which the custom rules should be run.
  4. Optionally, configuration settings for any configurable rules.
  5. The path(s) to the FxCop report file(s) containing the expected violations for the rules over the target assemblies.
  6. A reference to the method that should be used to indicate that a comparison between an expected and actual violation set has failed. This will usually be the Assert.Fail method from your testing framework of choice (e.g.: Microsoft.VisualStudio.TestTools.UnitTesting.Assert.Fail in MSTest). The method must take a single string parameter.
One of the FxCopRunner constructor overloads allows you to use an FxCop project file to specify #2-4. If you choose this approach, keep in mind that the project file must be constructed so that the paths to the rule and target assemblies will be correct when the analysis is run, and that these paths may be affected by any file deployment performed by your testing framework of choice.

You can execute the runner pretty much any way you like. However, given the overhead involved in spinning up FxCop and running it over a set of test targets, I prefer to run it only once per test assembly. I create a single test per rule which invokes FxCopRunner.AssertActualViolationsMatchExpectedViolations indirectly via a static helper method, and I organize these tests into one test class per rule category. For an example of this approach, see the Bordecal.FxCop.Rules.Tests project in the source code.

Last edited Mar 18, 2010 at 12:53 AM by calinoiu, version 2


No comments yet.