This paper discuss the technical and integration choices of the continuous integration solution made for NETASQ.
Please understand that some of the choices we made were dicted or implicitly determined by the system we wished to set up.
The aim of this project was to, daily, validate code commits and thus helping us to discover bugs, regressions and performance or conformance issues.
Therefore, this documentation attemps to explain the technical aspects involved by this test architecture and brings some ideas and « proof of concept » which may or may not be retained.
It describes real-life examples: the first part focus on the ASQ case while the second one is tproxyd centric.
The software engineering process is a perpetual restarting.
A project borns from an idea, then, during its life, it grows up.
In the past, we have noticed that testing and written documentation generally do not get enough attention from the developers, omitting potential side effects and making things harder to support because of the lack of proper documentation.
As the above diagram shows, coming from the design, going to the release: four paramount stages appear where testing & documentation are central parts of the process.
As it, they must be completed with a special care.
Indeed, they have such an influence on a product quality that they are essential parts of it.
Implementing automated builds, configurations, tests... is not a new concept, major opensource projects used this concept before eXtreme Programming made it popular.
Those principles are now better known as « continuous integration ».
Indeed, with time, build farm have evolved to become more usefull to programers but much more complexe too.
That's why, even if a majority of programmers agreed about the necessity of continuous automation, only a few have implemented it; probably because it is really a nightmare to make it works in a usefull and reliable way.
NetASQ incoming projects like the ASQShield need portability testing on various operating system.
Moreover, Quality Assurance is really important for us and our customers; this is especially true for products on the computer security market.
Major new features, small release cycles, the next major NSBSD release etc... mean lot of changes making things harder and longer to evaluate.
To be pragmatic, I would say that we must find solutions that could help us to reduce the time spent on testing, that's where continuous integration enter the game.
Continuous integration can identify issues with little or none human interventions or, proactively, find regressions before they reach the customer.
These approaches want to be, in both cases, more cost-effective.
To stick with NETASQ specific needs, some brainstorming sessions were necessary to understand each stage implied before getting the expected result: an autonomous plateform for testing & benchmarking our system.
I would like to present some other continuous integration softwares, as an overview of what is today's « State of Art ».
Many solutions exist, nonetheless, they use self-made automation process.
Those ones are generally specific and thus not suitable to other projects.
Buildbot [PROJ02] & bitten [PROJ02] are two interesting projects written in Python, the first one is used by some major open-source projects: vlc, ethereal, python, zope, kde, etc...
The second was a Google summer of code project, the goal was to bring continuous integration to trac, a great project management user interface that uses subversion as a backend, in a fully integrated manner.
As of today, I encourage you to visit the Pikefarm [PROJ03].
Part of the Pike developper community network, based on xenofarm and code librarian, it is the project which gives the best results I have seen.
The
following functionnal diagram describes the xenofarm process and how
it can be used in another project with some modifications.
This comparison was required because Dart1 was the most widely used Dashboard when we started this project.
We need to be sure we made the right choice, by comparing their activities and status.
|
Features |
Dart |
Dart2 |
|
|
Yes, clean |
Destroyed by DartServer |
|
|
Well presented |
awfull, many « hits » before accessing the results |
|
|
Yes |
no |
|
|
yes |
no |
|
|
yes |
no |
|
|
+ |
- |
|
|
last commits at the end of 'O4 |
3 pre-releases in 15 days (0.4-0.6) then nothing since september '05 |
|
|
full-features |
Continuation; even if this project is not very active, dart2 will probably be improved in the futur |
|
|
Hard to install ? Many technologies & computer language involved (java,cgi,php,XSL etc...) |
buggy |
Dart2 is a « work in progress » thus it looks somewhat unfinished and this was really annoying.
To be clear, the first time I took a look at it, I was not satisfied by its behaviour, and I think everyone who will have to use it would feel the same.
To quote Armijn Hemel ( [3] §3.1.3):
« Good log processing and visualization of logs is perhaps the most important part of a build farm. Without good visualization of logs the build farm is of no use to developers. »
Sad to say, but this was the case for Dart2...
Anyway, we believe that Dart2 is moving in the right way and we hope to see new features and bugfixes in a near future. That's why we finally decided to use it.
A dashboard is primary an user interface intended to present the results submitted by a multitude of hosts running heterogeneous environments.
The point of interest resides in its ability to quickly show an overview of those results.
Dart2 is written in the Java programing language, using many functionabilities from other Open-Source projects, as the following diagram explains.
Actually, DartServer, have three different kinds of clients: ctest, DartClient.jar or a python script (Dart.pdf §4.1.4).
Dart2 uses a special XML file format (§6: Custom Test Results) explained in details in the documentation.
You are really encouraged to use this native format, doing the opposite would imply some Digestor modifications...
First you need a workstation with jdk-1.4 installed. This part is not discussed here, just keep the linux_emu layer on the host because we will need it later.
We have only tested Sun Microsystems® JVM natively build for FreeBSD.
Quick steps to install a Dart2 (v0.6.0 & pre-1.0) server:
$ java -jar DartServer.jar --createserver NSServer
$ java -jar DartServer.jar --initializeserver NSServer
$ java -jar DartServer.jar --create ASQShield
$ java -jar DartServer.jar --create another_project_using_ctest
$ java -jar DartServer.jar --initialize --refresh --refreshServer \ NSServer ASQShield another_project_using_ctest
- If the database schema has changed between releases it will be necessary to use the -u switch to upgrade the project db.
- 'refresh' copies the project resources into the Project/ directory and thus, is only required the first time you launch the server with its associated projects. See below for a proper script.
Optionally, you can build Dart2 from a fresh svn checkout (what was done), with the ant tool:
$ svn co http://svn.na-mic.org:8000/svn/Dart/trunk Dart
$ cd Dart ; ant jar
$ cd ../ ; tar cfz Dart-pre1.0.tar.gz Dart/
Dart-pre1.0.tar.gz already includes the NetASQ template, configuration and some modifications (see below).
The following script do the trick (of course this script should be run as the www user).
#!/bin/sh cd /home/www/Dart/ # Run java with a 256M stack size nohup /usr/local/bin/java -Xmx256M -jar DartServer.jar NSServer ASQShield \ another_project_using_ctest &
To get fine grained-authentification, we put an Apache2 webserver in front of the Jetty Servlet engine thanks to mod_proxy.
This was mandatory since parts of the source code are accessible through those webservices (see the chapter related to Gcov results presentation below).
Another consideration was to restrain direct access to the Jetty server; keeping in mind that the TCP port is used for both results submissions & presentation, so it was necessary to filter access without breaking xml-rpc submissions.
To achieve this, we tried to deny the usage of the HTTP GET and POST methods except for localhost (incarned by mod_proxy).
The configuration is mainly carried out by two files: Server.xml & Project.xml respectively for the global and per project configuration.
These files are easy to understand thanks to explicit <tags>.
We will only discuss changes made from the default Dart2 configuration files.
Nothing has to be changed in Server.xml, the following mods concern exclusively Project.xml.
- To set locales [11], put <#setting locale="fr_FR"> in the templates pages.
This will enable continental style date format.
- By default, the Quartz scheduler process tasks every 10 minutes, this must be revised to 1 min.
<!-- Process queued tasks every 1 minute -->
<Type>dart.server.task.QueueManager</Type>
<Schedule>0 */1 * * * ?</Schedule>
- Adding Mantis (BugTracking system) and ViewVC (SVN Web interface) links:
Gcov
indentation <Property name="Menu">
<![CDATA[
<ul>
<li><a href="#">Repository ></a>
<ul>
<li><a href="https://labo.netasq.com/viewvc/">Trunk</a></li>
</ul>
<li>
<li><a href="https://intranet.netasq.com/applications/mantis/my_view_page.php">Bugs</a></li>
<li><a href="#">Documentation ></a>
<ul>
<li><a href="https://labo2.netasq.com/Doxygen/html/">Doxygen (API)</a></li>
<li><a href="https://labo2.netasq.com/"> Wiki</a></li>
</ul>
</li>
<li><a href="http://www.netasq.com/">Home</a></li>
</ul>
]]>
</Property>
- Email notification: starting with Dart post March '06, two ListenerManager have been implemented: dart.server.listener.MissingSubmissionListener & dart.server.listener.SubmissionErrorsListener.
We have not been able to make them send the e-mails, we will wait for a proper documentation.
Currently, it is not really an issue because we have not so many computers submitting results but the results older than 6 months no longer present any interest and removing them will avoid to run out of disk space.
Archiving will help us too if you need to move Dart to another server or simply to backup the data.
Indeed, when you initialize the server and the project, current path is put in <BaseDirectory> and other hardcoded stuff are being processed.
This means, that you cannot simply tar(1) the Dart/ directory and move it to another box, it will not work !
For further reading, we recommand the Dart.pdf documentation that explains this part in details, including the average disk space size required per submission.
In Dart2, the results presentations are based on the FreeMarker [12] template engine.
The idea behind the MVC (Model View Controller) is to offer the possibility to separate the web design from the java code, this way, web-designers can modify the layout without relying on programmers.
The Firefox webdeveloper extension [FOX01] was used to quickly bring some motifications to the css files.
