MonetDB is used for doing database research, but is also widely used in production for a large set of different applications. One of them is astronomy. The fact that MonetDB is a column store makes it very well suited for the large analytical workloads of modern astronomy. So it is not surprising that MonetDB was chosen to be part of the software pipeline of the LOFAR radio telescope. In the previous blogpost “Time-domain radio astronomy with MonetDB” by Bart Scheers, a more detailed explanation can be found.
But the work on the successor of LOFAR is already started. The Square Kilometre Array (SKA) telescope is going to be build in South-Africa and Australia. When it becomes operational it is expected to create more data than the current internet creates in one year! To make this possible, a lot of new software is needed. The SKA organization is working on this, in cooperation with Amazon Web Services (AWS). Last year, the grants program AstroCompute was announced to stimulate the development of this software.
Since we were already working on similar problems, we thought that we could make a contribution. Fortunately our proposal was accepted. One aspect of our project is continuous integration. In the blogpost “Hidden in the Manual” by Arjen de Rijke, we extensively explain what kind of problems we need to solve in order to use a CI system for such a large project.