Re: MonetDB error while restarting the database

Hi Ashish, What happened there is monetdbd send a TERM signal to the mserver process (the actual MonetDB database process), which killed all running threads. The messages "#GDKexit: killing thread" means the the MonetDB kernel (GDK) killed a thread still working at the time the process was terminated. This is most likely an indication that the database was still processing something at that time. Depending on what it was actually doing, data corruption might occur. This can explain why monetdbd reports an internal error why truing to start the mserver process again. Check the merovingian.log file again for the events when monetdbd reported the internal error. There should be more information logged on why mserver cannot start up any more. Best regards, Dimitar

On 2014-Oct-27, at 15:55 , Ashish Kumar Singh wrote:

Thanks Dimitar,

Yes we are restarting the data base and as part of that we are seeing this error.

I am more worried about the error below:

2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread

On java client side it says

monetdbd: internal error while starting mserver, please refer to the logs

Regards, Ashish

-----Original Message----- From: Dimitar Nedev Reply-To: "users-list@monetdb.org" Date: Monday, 27 October 2014 8:15 pm To: "users-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

Hi Ashish,

The exit timeout only tells monetdbd how long should it wait when trying to gracefully shut down a mserver process. According to the log, the mserver process was shut down within the limit, since there are no messages like "timeout of <exittimeout> seconds expired, sending process <PID> the KILL signal".

Now, the interesting entry in the log is the control process message: 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb'

This one would indicate that someone (or something) gracefully stopped the database using the monetdbd client. Probably with the following command line call like: 'monetdb stop pbsworksdb'. Do not be confused by the order of logged events - monetdbd will first stop the process and later log that the call to stop the a database has been successful.

Best regards, Dimitar

...

On 2014-Oct-27, at 14:49 , Ashish Kumar Singh wrote:

Thanks!

Appreciate your response so there are few more people getting the same error :). Any suggestions from dev team on this issue? Can monetdbd exittimeout be also in picture for this?

Regards, Ashish

From: Pierre-Adrien Coustillas Date: Monday, 27 October 2014 6:51 pm To: "users-list@monetdb.org" Cc: "developers-list@monetdb.org" , Ashish Singh Subject: Re: MonetDB error while restarting the database

...

Hello

(google translate)

I receive this error when there is no specific spacedisk. Monetdb sometimes uses a lot of temporary diskspace (500GBisnormalfor 1TBof data), which arereleasedat the end oftreatment or after acrash

Error last week : 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 (database 'lemonde') the TERM signal 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: 42014-10-22 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread

Pierre

-- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

De: "Ashish Kumar Singh" À: "Ashish Kumar Singh" , users-list@monetdb.org Cc: developers-list@monetdb.org Envoyé: Lundi 27 Octobre 2014 13:49:19 Objet: Re: MonetDB error while restarting the database

Guys,

Any help with this will be really helpful for us?

Regards, Ashish

From: Ashish Singh Date: Monday, 27 October 2014 4:06 pm To: "users-list@monetdb.org" Subject: MonetDB error while restarting the database

...

All,

We are facing a new issue with monetdb where monetdb connections and process is being killed very frequently. Is this a known issue or any suggestion in getting more details will be helpful.

Regards, Ashish

Mervogian.log:

2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor ksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw orksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module loaded 2014-10-27 09:01:00 MSG control[23583]: (local): started 'pbsworksdb' 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client localhost.localdomain:53844 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 (database 'pbsworksdb') the TERM signal 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' (6351) has exited with exit status 0 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' has shut down 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb' 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: starting database 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 0.00 (648-648=0) 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor ksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw orksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module loaded 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client localhost.localdomain:53853 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:32 MSG c

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list