I Break Tests Because I Care

First, a sincere apology to all of you who woke up this morning, updated your Plone bundle, ran your tests and suffered unexplainable breakage. Thank you to those of you(particularily the observant stalwart developers of Zest Software) who provided feedback and helped identify the following Zope 2 bug.

It's 4am, do you know who your tests are bad touching?

Lately I've been working on the testing frameworks that are derivative of Testing.ZopeTestCase. You may have read that ZopeTestCase must die. You may have received a tongue lashing from a Five developer about writing a test that uses the infamous "installProduct('Five')". You may simply have used the provided test framework unaware that your tests were treating other packages' tests like the Bush Administration treats personal civil liberties or poor people and minorities.

It's ok.

If you can forgive me for breaking your tests, I'm sure all those hard working Five developers will find it in their hearts to forgive you for the fact you can't run Plone's test and Five's tests at the same time.

Until now (or about six months ago shrouded in the vault of zope.*), the right way was hard and painful and full of stinky boilerplate.

Layers: The Neighbourly way to Set Up Your Zope 3 Components

In zope 2.9, the zope 3 testrunner became available to us z2 trogdolytes and with it the zope3 strategy for more efficient functional and integration testing:layers.

Layers are a little like social clubs for tests. Just because your unit tests in your package live next to your integration tests doesn't mean that they should run together or even like each other. Layers let everyone run where they are comfortable and happy and passing; the frat layers spills no beer on the goth layer, the goth layer doesn't bum out the church goers and so on. This applies whether you run your tests alone, or with everyone elses.

The testrunner groups test cases (think classes like ZopeTestCase, PloneTestCase etc) by their layer and runs them together (any tests lacking a testcase with a layer run in a group called unit tests). Between the setup and the teardown of a layer, all tests run inbetween are assure of having the environment they expect.

Cut to our dilemma. What we are currently most concerned about in this picture is the component architecture. The entire component architecture: all the views, adapters, events, etc that make up a running Zope 2 site. If you have ever had to debug a rogue
ComponentLookUpError,

spent hours wondering where your views and adapters disappeared to, you know this pain.

The current method of doing this, Testing.ZopeTestCase.installProduct('Five'), can only be called once and only once to set up the component architecture; all subsequent calls have no effect once Five has been loaded by ZopeTestCase. Therefore, If a unit test happens to call zope.testing.placelesssetup.tearDown or zope.testing.cleanup.cleanUp, all following tests will have no component architecture. Welcome to test debug hell.

Fear not. There is hope for your fragile testing setup. layers can setUp and tearDown the CA as often as needed.

PloneTestCase and CMFTestCase now have some basic layer support to make this easy.

• http://svn.plone.org/svn/collective/CMFTestCase/trunk/
  
  
• http://svn.plone.org/svn/collective/PloneTestCase/trunk/
  
  

Any test that inherit from these classes gets the layer treatment that handle setting up the CA. Usage is usually as easy as simply putting these branches in your products directory.

Since layers are a useful tool to the test writer, let's take a looK
at what is going on here. It's best to simply think of the layer as a named data block with 2 methods: setUp, and tearDown.


Here we see the layer that loads the CA. It is an old style class with a classmethod setUp loads the entire CA(and does a bit to make sure ZCML error messages are not supressed due to ZTC optimization). The classmethod tearDown cleans up afterwards.


Pretty simple.

The classmethod calls may look a bit strange. Again, think of this class as data block rather than an object factory; the layer will never be instanciated in the classic OOP sense.

Note: The conventional pattern for layer is to use an old style class and classmethods; concievably, you could use a module with setUp and tearDown functions or an instanciated object with setUp and tearDown methods, but an old style class with classmethod seems to be the most reliable and flexible.

Applying the layer is easy too. The layer is assigned as an attribute to the test class. Currently for BBB a set of checks makes sure that layers are available(hence setup.USELAYER). If you know you are developing against 2.9 or greater and don't need any BBB, this check is unnecessary.

Now when we run the the Plone tests ($ bin/zopectl test -s Products.CMFPlone), we see the following ::

Running unit tests:
Running:

......................................................................................................................

Ran 158 tests with 0 failures and 0 errors in 1.465 seconds.

Running Products.PloneTestCase.layer.ZCMLLayer tests:

Set up Products.PloneTestCase.layer.ZCMLLayer in 0.883 seconds.

Running:
..................................................................................................... etc....

Ran 1750 tests with 11 failures and 0 errors in 182.200 seconds.
Tearing down left over layers:
Tear down Products.PloneTestCase.layer.ZCMLLayer in 0.002 seconds.
Total: 1908 tests, 11 failures, 0 errors

All tests not requiring the CA or tests which handle all setup and teardown inside the test run first. Then ZMCLayer is set up, all tests that use it run, and the layer is torn down. If you just wanted to run the CA dependent tests you could do so with the following command:

~/zopesite/ $bin/zopectl test -s Products.CMFPlone --layer=Products.CMFPlone.layer.ZCMLLayer

What does this mean to me and my work?

if you are lucky, it may just be a few more statistics spit out of your test runner (and the ability to run all your tests at once). It could just work in many situations.

Chance are it won't work if your [Plone/CMF]TestCase test commits the following crimes against nature (check yourself if you've been wading in the z3 soup).

If your tests uses PloneTestCase and does the following:

1. uses Products.Five.zcml.load_string or Products.Five.zcml.load_config
   
   
2. uses aforementioned to load functions to load zcml that never gets loaded on Zope site loading
   


3. calls provideAdapter or other convenience registration functions
   


4. calls zope.testing.placelesssetup.tearDown or zope.testing.cleanup.cleanUp
   

1-3 will result in erratic behavior for tests within the ZCMLLayer layer. #4 will result in ComponentLookUpErrors due to the clearing of the registries of the component architecture.

Never fear: just following these easy refactorings.

You must(fixes in order of issues):

1. Delete it. you don't need it anymore
   
   
2. Slap yourself for doing this. Refactor your test into units and integration tests.
   
   
3. Repeat fix for #2
   
   
4. Delete it. you don't need it.
   
   

If you find that you are slapping yourself alot, you are feeling layers are in forcing us to use better testing practices.

You may even find you need a special layer of specific zcml loads. Mostly, you should just be deleting now useless boilerplate and loving it. If not, let me know.

You may notice that layers force us to divide our tests into more appropriate groupings of unit tests(atoms of code), and groups of p[roper integration tests(framework w/ code units).

It is important to understand that with this strategy all the active zcml is loaded for a site. This means if some product or package overrides zcml effecting your test, your test may behave differently than you expect(hopefully, your test coverage will actually act like a canary in the coal mine in this situation).

Note:for working ZTC doctest layer support, a fresh checkout of the Zope 2.9 branch is required.

Coming soon: The Future!!!!