The MonetDB project runs since early 1993 when the first lines of code were written. Initially developed for Unix only, it became clear that moving MonetDB into the open-source community in 2003 comes with the obligation to make it available on many platforms. For two decades, the code has been built and tested nightly on ~20 different platforms. Its functionality has increased significantly and its maturity has been brought to a level that more enterprise-class companies are using MonetDB in their core business offerings today.
A critical component in this process is the collection of tools we use to build the system from source, finding and linking the libraries needed, and to cope with the exceptions raised by the Linux (different flavours), Unix, macOS and Windows platforms. We started in 1996 with our own tool autogen to handle this.
This small tool generated automake and autoconf files for Unix systems, and Microsoft Visual Studio (nmake) files and batch scripts to build on Windows systems. It started out as a small, self-documented Python program. Over the years, it grew into a large program which required constant laborious attention to follow the OS versions and to support the ever-growing collection of linked libraries.
Automake and friends have been very stable over the years. However, with the experience of a fork of MonetDB, called program MonetDBLite using CMake, it became clear that this ‘oldy’ had to jump the fence. A major step was taken by Pedro Ferreira with the creation of the first CMake infrastructure for MonetDB. It was an excellent learning step and convinced us to prepare for a big jump. Finally, this is achieved in June 2020, when the code base has become fully controlled by CMake.
The main benefit is that CMake does all steps, including the configure phase (detect what a platform has to offer in terms of libraries and functions), the build phase, the test phase and the packaging phase (rpm, deb, tarball, etc). All of this for all platforms supported (Linux, Windows, macOS).The commands are the same for each platform so that a developer can work on any platform.
Although not a goal of this change, we observed some faster builds, mostly on Windows and macOS, where the old autogen system lacked smart dependency handling. Illustrative are the numbers obtained on a development laptop with Intel(R) Core(TM) i7-6560U, 16G RAM running ArchLinux. Compiling the “default” branch of MonetDB for debugging is a common action.
| autogen/ autotools | CMake (Makefiles) | CMake (Ninja) | |
|---|---|---|---|
| bootstrap+configure/cmake -DCMAKE_BUILD_TYPE=Debug | 0:10.82 + 0:20.60 | 0:13.20 | 0:11.22 |
| make/cmake –build . | 2:31.83 | 1:58.97 | 1:03.60 |
| make install/cmake –build . –target install | 0:03.95 | 0:01.64 | 0:00.13 |
A facelift like this requires all involved in the development and maintenance of the code to recognise its necessity. Despite the fact that CMake has already been around since 2000, the proper working of the old setup has hindered such a step for a long time. But a new decade calls for change, so we took off our old jacket and embraced the new build infrastructure.
The most difficult step is ongoing, i.e. cleaning up the interfaces between all layers in MonetDB, e.g. which functions need to be exposed to upper layers and which should not. A clean API makes it easier for the build system (CMake or others) to handle building and linking.
Every tool has its limitations in terms of functionality. A complementary part that calls for attention is to upgrade the packaging of MonetDB on all supported platforms, e.g. Ubuntu, Centos, Homebrew and RPMs. This definitely will require further stitching the CMake code. All this will come together in the next official feature release of MonetDB later in 2020.
We are happy that the facelift has worked out well. The build infrastructure is now more accessible for developers. Ready for the coming decade.