Troubleshooting

Troubleshooting giulia Tue, 02/18/2020 - 16:51

Throughout your experience with MonetDB, you might incur in bugs and issues that are not expected. 

While we cannot give a solution to that right away, there are some tools that can help the debugging and tracing process to ease yoor experience in fixing issues.

Debugging

Debugging mk Mon, 03/02/2020 - 14:07

In most cases, the system produces  informative error messages. However, there are situations where MonetDB enters an area not covered by the test suite or previous use and a segmentation fault occurs. These cases are hard to analyze outside the development lab. To isolate and resolve the issue we need at least the following information.

Sent us the result of the command mserver5 --version --dbname=<databasename> or the equivalent using monetdb --version <databasename>

Is the error reproducible with a small (5-10 line) script/query/program? Trim your experiment to the minimal size that demonstrates the erroneous behavior. Such a script is the best we can hope for, because it will end up in the nightly testing.

In addition, follow the steps, assuming that you are logged onto the same Linux (!) machine as where the server will run:

  1. Sent the tail of the merovingian log (100 lines) located in the dbfarm directory.
  2. Start the database server either with monetdb  or as a full command line mserver --dbpath="..." .
  3. Lookup the process id for this server using the command  ps -al |grep mserver5
  4. Attach the debugger with gdb mserver5 <processid>
  5. Run your experiment again until it hits the segmentation fault, which you see emerging in the window.
  6. Then obtain a stack trace with the command where. It shows the context of the crash using the list command and the variables used in the context of the crash.
  7. Collect the status of all threads using the command thr app all bt.

 

Tracer

Tracer giulia Mon, 03/02/2020 - 14:08

Tracer is the general logging system for the MonetDB stack, modelled after other well-known logging schemes (e.g: Python). It provides a number of logging levels and options to increase or reduce the verbosity either of individual parts of the code or of the codebase as a whole. It allows users to focus on logging messages related to certain steps of execution, in order to aid debugging. The behaviour of Tracer can be controlled at runtime using a SQL API designed for that purpose.


LOG LEVELS

Tracer supports the following logging levels with the following order: CRITICAL > ERROR > WARNING > INFO > DEBUG


COMPONENTS

MonetDB is built upon three different layers SQL, MAL and GDK. Each layer consists of certain components. For instance, SQL layer involves “SQL MVC”, “SQL optimizer” and many more. However, not all the components need to belong to a single layer. There are components that target the whole stack, such as memory allocation, I/O and threads or MonetDB modules like lidar, fits and bam, that fall under a more general category. Tracer introduces the concept of “component logging” where each component can be in a different logging level. The list of the available components is the following:

  • ACCELERATOR

  • ALGO

  • ALLOC

  • BAT_

  • CHECK_

  • DELTA

  • HEAP

  • IO_

  • PAR

  • PERF

  • TEM

  • THRD

  • GEOM

  • FITS

  • SHP

  • SQL_PARSER

  • SQL_TRANS

  • SQL_REWRITER

  • SQL_EXECUTION

  • SQL_STORE

  • MAL_WLC

  • MAL_REMOTE

  • MAL_MAPI

  • MAL_SERVER

  • MAL_OPTIMIZER

  • GDK


LAYERS

Tracer provides a way for 1) manipulating massively the logging level of the components depending on the layer they belong to and 2) setting all the components to a certain level of verbosity. In the following list you can find the available layers that Tracer supports:

  • SQL_ALL      Sets SQL layer

  • MAL_ALL      Sets MAL layer

  • GDK_ALL     Sets GDK layer

  • MDB_ALL     Sets all the components (MonetDB codebase)


ADAPTERS

Tracer introduces the concept of adapters in MonetDB stack. Simply, an adapter determines where the logging messages will be output. MonetDB, for now, supports only one adapter, but more will come in the future. Below you can find the supported adapters along with a brief explanation.

BASIC: Writes all the logging messages produced to the log file “mdbtrace.log”. The file is created by default in the DB path directory, however, as it’s mentioned later on you have the option to specify the desired path on startup.


LOG ROTATION

Tracer is compatible with logrotate utility. Sending a SIGHUP to “mserver5” will cause Tracer to reopen “mdbtrace.log” if the BASIC adapter is used.


SETTING CUSTOM DIRECTORY FOR TRACES

Using directly mserver5: In order to use a different directory than the DB path in order to store the produced traces, you can simply start mserver5 with the option “--dbtrace=<path>”.

Using merovingian (monetdbd): In order to use a different directory than the DB path in order to store the produced traces, you can use ‘monetdb’ program and set the variable ‘dbtrace’. For instance: ‘monetdb set dbtrace /tmp/mydb_trace.log mydb’.


TRACER API

Tracer comes with a number of possible SQL calls at the user level. Using the mclient program, you can simply execute ‘CALL logging.*(...)’ in order to interact with the logging system.

  • logging.flush();

    Tracer uses a buffer under the hood where it stores all the logs until reaching 64KB. After that point, a flush buffer operation is performed automatically. However, in case your messages remain in the buffer because e.g: you have too few logging messages, you can always perform this call in order to flush the buffer explicitly. The log messages are getting outputted to the active adapter at that moment.

  • logging.setcomplevel(str comp, str level);

    Sets the log level for a specific component.

  • logging.resetcomplevel(str comp)

    Resets the log level for a specific component back to the default (ERROR).

  • logging.setlayerlevel(str layer, str level)

    Sets the log level for a specific layer

  • logging.resetlayerlevel(str layer)

    Resets the log level for a specific layer back to the default (ERROR).

  • logging.setflushlevel(str level)

    Sets the flush level. A flush level determines the importance of a certain logging level. When an important message comes to the logger which has the same logging level as the flush level, Tracer will be triggered to flush the buffer. Tracer by default will always decide to flush CRITICAL and ERROR messages no matter the flush level is set.

  • logging.resetflushlevel()

    Resets the flush level back to the default (INFO).

  • logging.setadapter(str adapter)

    Sets the adapter that Tracer is going to use in order to output the logging messages.

  • logging.resetadapter()

    Resets the adapter back to the default (BASIC).

  • SELECT * FROM logging.compinfo();

    Executing this query will trigger Tracer to dump all the available components along with their id and the logging level that has been set.


DEVELOPERS

If you are a MonetDB developer/enthusiast and you would like to contribute in the logging system of MonetDB by either introducing new layers, components or logging levels, then 'gdk/gdk_tracer.h' and 'gdk/gdk_tracer.c' are the files you are looking for!