We’ve used Windmill in our Launchpad buildbots for a while now, and it’s actually worked out quite well. I was afraid that we would have a lot of fallout, since in the beginning Windmill was fragile and caused a lot of intermittent test failure. However, so far I’d said that we’ve had very little problems. There was one intermittent failure, but it was known from the beginning that it would fail eventually. Apart from that we’ve had only one major issue, and that’s that something is using 100% CPU when our combined Javascript file is bigger than 512 000 bytes. This stopped people from landing Javascript additions for a while, and we still haven’t resolved this issue, apart from making the file smaller.

There are some things that would be nice to improve with regards to Windmill. The most important thing is to make sure that launchpad.js can be bigger than 512 000 bytes:

• https://launchpad.net/bugs/519744

It would also be nice to make the test output nicer. At the moment Windmill logs quite a lot to stderr, making it look like the test failed, even though it didn’t. We don’t want Windmill to log anything really, unless it’s a critical failure:

• https://launchpad.net/bugs/494519

I was going to say that we also have some problems related to logging in (because we look at a possibly stale page to decide whether the user is logged in), but it seems like Salgado already fixed it!

• https://launchpad.net/bugs/515494

It would also be nice to investigate whether the problem with asserting a node directly after waiting for it sometimes fails. We had problems like that before; code was waiting for an element, and when using assertNode directly after the wait, the node still didn’t exist. I haven’t seen any test fail like that lately, so it might have been fixed somehow:

• https://launchpad.net/bugs/487666

There are some other things I could think of that would be nice to have. I haven’t found any bugs filed for them, but I’ll list them here.

• Don’t run the whole test suite under xvfb-run. It’d be better to start xvfb only for the Windmill tests.
• Use xvfb by default for Windmill tests. When running the Windmill tests it’s quite annoying to have Firefox pop up now and then. It’d be better to run them headless by default.
• Switches for making debugging easier. Currently we shut down Firefox after running the Windmill tests. It should be possible to have Firefox remain running after the test has finished running, so that you can manually poke around if you want to. If we use xvfb by default, we also need a switch for not using it.

I can’t stress enough how important it is to automatically block, stop the line, when a regression occurs. Forcing someone to take action. Don’t think it’s enough to have tests to catch regressions. It won’t help you much, unless you run them automatically, and most importantly, block on test failures, forcing someone to fix them.

In Launchpad we have been quite good at this in the past. Already from the beginning we ran the whole test suite before every commit. If a test failed, the commit wasn’t made, and the committer had to make the test pass before being able to commit those changes to the mainline. Now we have something similar. For performance reasons, instead of running the tests before the commit, we run them after. If a test fail, we enter testfix mode, blocking all commits to mainline, until the test passes again.

But, when we decided to bring in AJAX into the equation, we failed to do the same for the new test infrastructure we added. We use Windmill to test our AJAX functionality, and since it was a bit flaky, and it wasn’t trivial to integrate it into our existing test suite, we thought it was enough to be able to run the tests manually to avoid regressions. This was a big mistake. Not many people are going to run the tests manually, so regressions are bound to sneak in without you noticing it. Believe me, I know. I integrated the Windmill tests into our normal zope.testing test runner. When I did this, I found out that a lot of our Windmill tests were actually failing. We set up a buildbot builder to run the Windmill automatically, hoping that it would make regressions less likely to be introduced without us knowing about it. It helped a bit, we actually did catch a few regressions, but it was hard manual work. It required someone (me!) to keep an eye on the buildbot runs, looking through the test log, and chase people to fix it. This led to having not all the tests passing most of the time, which made it even harder to notice new regressions. So while simply having the tests run automatically helps a bit, it still requires a lot of discipline and manual work to prevent regressions from being unnoticed.

That’s why I’m pleased to announce, that Windmill tests are now included in the regular Launchpad test suite, which means that when a Windmill test fails, we will enter testfix mode and we’re forced to take action. It will be a bit painful in the beginning. I’m sure that we will see some spurious test failures. However, I’m sure it will be less painful that it has been to keep the current Windmill test suite under control.

The next time you add new testing infrastructure, let’s include it in the regular test suite from the beginning, OK?

When doing work on something that is supposed to be used by others, don’t forget to think about how it’s actually going to be used. Not only to think about it, but to actually try it out, to confirm that it works nicely when integrated, and that it’s easy to integrate it. And let’s not forget to document how to integrate it, and ideally to test it as well.

As an example, in Launchpad we use lazr-js for our Javascript infrastructure. We recently changed the way it’s integrated into Launchpad, giving it a proper setup.py file, so that we can generate an egg and depend on it through Buildout. The integration issue was of course taken into account there, making sure it was easy to build lazr-js both standalone, and when used in another project, like Launchpad. There was one command to build everything, which is simple enough. However, one thing wasn’t done. It wasn’t documented how you should use lazr-js in another project. Therefore, when people continued to develop lazr-js, adding more features, and making the build system more complicated, there wasn’t much thought about keeping it easy to build lazr-js in other projects. The build process became more complicated, multiple commands had to be executed. This is fine when building lazr-js by itself, since all you have to do is make build. However, when using lazr-js as an egg, you don’t have access the Makefile, which means that you have to duplicate the build steps. Therefore, having the build to be more than one command, makes it harder to use elsewhere. In fact, the build process of lazr-js changed so much, that we didn’t know anymore, how to properly use the latest version of lazr-js in Launchpad.

This is just one example of integrating external libraries. But the same is true for code internal to the project. When developing code that is to be used by multiple call sites, it’s important to think about how it’s actually going to be used. It’s easy to get carried away, developing a, what you think, really nice API. But then when people start to use it, it turns out that it’s not so nice.

What can be done to avoid integration issue? Ideally, you should document and test how the integration is supposed to work. By doing this, you get a feel of how to use the API. Doctests are actually quite nice for this purpose. If you manage to produce a readable doctest, it’s quite probable that your API is easy enough to use.

Sometimes adding tests for integration isn’t feasible. For example, in the case of lazr-js is not that easy. What I do when I develop on lazr-js is to have a throw-away Launchpad branch, where I use my lazr-js branch and manually make sure that it works nicely when integrated. Take a look at how it looks when integrated. What steps do you have to do to use it? Is that something that you will want to do for every call site? Are people likely to copy and paste an existing example to use your code? If the answer to the last question is yes, it’s not easy to use your code.

In PyGarmin, I used PyUSB to implement USB support, and I was struck with one odd error. Sometimes, a USBError was raised, with the error message “No error”. I couldn’t find any documentation for this, and I still don’t understand why an error was raised, saying that there was no error.

The error happenend when trying to read from the bus for the first time, after trying to send two packet without any errors. After some testing, I found out that simply trying to read again from the bus didn’t work, but if I send the two packets again, everything works without any errors.

For those interested, I fixed this in revision 91 of PyGarmin.

I have been working on a program that can download data from my Garmin Edge 305 GPS in Linux, and now I’ve finally got something that’s work. The program is called GarminSync, and you can download it from Launchpad.

Please try it out, but don’t expect too much from it. My main focus has been to get something working, now I’m going to focus more on getting it somewhat stable. There are a tons of bugs, for example it sometimes doesn’t work to download the data, and you have to simply try to run it again, and/or reconnect the GPS. But at least it’s able to download the runs from my GPS and export them to TCX files. It should be possible to such files to MotionBased, but it seems like the format is slightly wrong. At the moment I can’t get the GPS to work in VMWare, so I haven’t been able to get a correct file to compare with.

GarminSync is written in Python, so it probably works on other platforms than Linux as well, but I haven’t tried yet.

While extending pygarmin to support my Garmin Edge 305, I ran into a problem; pygarmin uses the serial protocol, and some packets that the Edge sends are too big for the serial protocol to handle. So when you request some information from the GPS, you don’t get any response back. The solution to this was to add native USB support to pygarmin, making it possible to use the USB protocol to talk to the GPS.

This turned out to be a quite fun task, actually, pyusb makes it easy to communicate to USB devices using Python. Of course it wasn’t entirely trivial, since the Garmin protocol specification seems to be out of date, so not everything the Edge 305 does is documented. But last night I finally managed to get something working, I can now specify "usb:" as the device to use, so that the USB protocol will be used instead of the serial one. It actually seems to work quite well, although there are some bugs still.

For those of you brave enough, I’d love to get feedback whether it works for you, especially if you have some other Garmin GPS than the Edge 305. You can test it by getting my edge-305 branch of pygarmin and try the following (which should write out some information about your GPS):

./pygarmin -p "usb:"