#!bootstrap/environment

summary: Eggs are changing how things are done around these parts. This entry will cover how to use Ian Bicking's workenv with Zope2 and take a sideway glance at zc.buildout.

Hello Eggs

One of the first things plone developers noticed about eggs was the funny way they (by default) wriggle into your python's site-packages. Next thing noticed: the zope testrunner didn't know where to find them without symlinking them into Zope's lib/python.

Zope(2) classically did not handle it's mountains of python like the rest of the python world did, so real surprise that zope did not jive with the new setuptools regime. But things are getting better...

Building an Environment

Jim Fulton, pope of zope, has been working on a system for composing large complex systems called zc.buildout . As part of this recipe driven strategy it can install eggs into a local buildout directory(along with the rest of the buildout "parts"). Ostensibly, we will probably see recipes that can construct a particular type of development environment for any variety of plonehead in the not too distant future.

zc.buildout is a bit industrial to the first blush. Nothing wrong with that, just a bit of framework comprehension and possibly simple programming required(if the current menu of recipes doesn't fit your bill). zc.buildout looks very powerful and simple for deployment tasks, including building systems beyond python like squid or Apache. But maybe not the tool for playing around, testing and discarding your python ideas (yet).

note: If you are interested in zc.buildout for zope2, take a look at topp.buildout, an initial experimental buildout attempt for opencore stack. For pedagologicaly purposes only. Ian says this is a bad buildout example. look at zc.sharing.

Ian's workingenv is a bit to the other side of the same spectrum. It does a bit of the same thing but is more focussed on the python developer and python packages (as opposed to say a zope deployista with a host of other applications). It is more concerned with the problem of keeping those snakes on the plane than the care and feeding of your squid.

workingenv allows you to create a temporarily constrained environment for python, substituting a localized work area for your site-packages. It plays nice with setuptools and is generally helpful.

the self contained script runs like so::

$ workingenv.py mysandbox

This bootstraps a local installation of setuptools to 'mysandbox', which now has it's own bin, lib/python, conf and src directories.

To activate the environment(using bash)::

$ source mysandbox/bin/activate

Once activated, you can install eggs like so (mysandbox/bin/easy_install is now part of your path)::

$ easy_install zc.buildout

All scripts are installed to the local bin. Likewhise, any source packages you download and install via 'python setup.py develop' are installed in the local python directory. For example::

$ cd mysandbox/src
$ svn co https://svn.openplans.org/svn/tagger/trunk tagger
$ cd tagger
$ python setup.py develop

This puts an entry in mysandbox/lib/python/easy_install.pth pointing to my development directory in src/tagger.

Even more convenient, workingenv.py will take a -r argument for an egg reciper. This recipe may either be a url or a file. To bootstrap a local turbogears development environment::

$ workingenv.py mytgsandbox -r http://svn.colorstudy.com/home/ianb/workingenv/tg-0.9.txt

Pretty nifty, eh? You may also embed -r into the recipes allowing recipes to be aggregates.

workingenv and Zope 2

For working with zope2, we can use workingenv's --env flag to set the INSTANCE_HOME and SOFTWARE_HOME environmental variables(needed for running tests without zopectl). Zope install also expects the disutils --home flag, so we add it too.

In this example, I create the environment first, pointing it at a zope src directory I have not created yet, then checking out the zope and installing zope that directory.

$ workingenv.py myzopesandbox --home --env=INSTANCE_HOME:~/mysandbox/zope --env=SOFTWARE_HOME:~/mysandbox/zope/lib/python
$ cd myzopesandbox; source bin/activate
$ svn co svn://svn.zope.org/repos/main/Zope/branches/2.9 zope
$ cd zope
$ make inplace; make instance

This almost works. For zopectl to see the lib/python inside the workingenv sandbox, you have to edit line 8 to read::

PYTHONPATH="$SOFTWARE_HOME:$PYTHONPATH"

Now you have a fully functional constrained python environment for zope. To deactivate::

$ deactivate

Python and shell return to their normal search path.

The Zope Situation

Not too shabby but not a great situation with zope. Obviously the match is a bit rough right now. Editing a script is easy to forget, and all of Zope's legacy codebase exists as products, not eggs.

This particular place where the road turns to dirt is where I think this may zc.buildout and workingenv might be able to work together for best effect. zc.buildout lets us write recipes for zope2 installation and for handling products. The recipes could recognize when they are executed in a workenv setting(workingenv.py sets an environmental variable) and smooth such rough spots as adding the current PYTHONPATH to zopectl's PYTHONPATH.

TestRunner

And maybe recipes could set the testrunner up to find the right eggs. Currently, I have only found one sure fire way to have my testrunner run tests for a "development" egg (an egg installed via python setup.py develop)::

For the tagger install example above(if it were installed into the zope sandbox)::

$ cd myzopesandbox
$ zope/bin/zopectl
zopectl> test -s tagger --test-path=/myhome/myzopesandbox/src/tagger/src/tagger

Part of the issue is the zope testrunner isn't really egg/setuptools aware yet(so it doesn't see those nifty pth files and follow them). The other part of the issue, is that having the testrunner search the entire site-packages is slow. This is the part workingenv solves; it's ok to search all available python pkgs because the environment is constrained to what you are working on.

Final Thoughts

I'd like to see the testrunner handle setuptools better and I see potential in using zc.buildout to handle the ugly bits of bootstrapping one's environment. I'd like to see both of this tools take advantage of workenv and it's developer and setuptools friendly way of doing it's job.

I look forward to more recipes for zc.buildout and a decent tutorial(nouri? bitte?). Unlike workingenv, it's harder to get zc.buildout to doing something useful from the get go in the raw dev environment.

My response to workingenv and zc.buildout is mostly viceral though; in the land of thousand frameworks, a "script that just works " like a cool glass of water. We have deployment needs at work, but more commonly we have the need for bootstrapping consistent development environments. workingenv plays nicely with existing tools like easy_install amd setup.py whereas zc.buildout requires everything to go through the buildout script.

Right now, I prefer workingenv for developing and we are using zc.buildout for our buildbot. Still would like convergence though. Seems possible too.