Re: MonetDB error while restarting the database

Hi All, I have filled an issue Bug 3610: exittimeout property not being honoured against monetdb. It looks like GDKExit error is coming too frequently due to all this and its not even waiting for default value of 60 seconds. Regards, Ashish -----Original Message----- From: Ashish Singh Date: Thursday, 30 October 2014 8:24 pm To: "developers-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

Thanks All!

we made sure now that database connection pool has released all connections before we shutdown monetdb.

But still infrequently we are seeing this error popping up and we have double checked that client has returned all the connections.

Before stopping the database a data load process happened from the client to monetdb, can it be contributed to that??

Regards, Ashish

-----Original Message----- From: Sjoerd Mullender Reply-To: "developers-list@monetdb.org" Date: Monday, 27 October 2014 11:07 pm To: "developers-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

On 27/10/14 17:42, Dimitar Nedev wrote:

...

Hi Ashish,

What happened there is monetdbd send a TERM signal to the mserver process (the actual MonetDB database process), which killed all running threads. The messages "#GDKexit: killing thread" means the the MonetDB kernel (GDK) killed a thread still working at the time the process was terminated. This is most likely an indication that the database was still processing something at that time. Depending on what it was actually doing, data corruption might occur. This can explain why monetdbd reports an internal error why truing to start the mserver process again.

Data corruption should *never* occur. If you have proof of (and a recipe for) data corruption, please file a bug report. Of course, if a thread was still working, the transaction it was working on will most likely not get committed.

Otherwise the above is correct.

...

Check the merovingian.log file again for the events when monetdbd reported the internal error. There should be more information logged on why mserver cannot start up any more.

Best regards, Dimitar

...

On 2014-Oct-27, at 15:55 , Ashish Kumar Singh wrote:

Thanks Dimitar,

Yes we are restarting the data base and as part of that we are seeing this error.

I am more worried about the error below:

2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread

On java client side it says

monetdbd: internal error while starting mserver, please refer to the logs

Regards, Ashish

-----Original Message----- From: Dimitar Nedev Reply-To: "users-list@monetdb.org" Date: Monday, 27 October 2014 8:15 pm To: "users-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

Hi Ashish,

The exit timeout only tells monetdbd how long should it wait when trying to gracefully shut down a mserver process. According to the log, the mserver process was shut down within the limit, since there are no messages like "timeout of <exittimeout> seconds expired, sending process <PID> the KILL signal".

Now, the interesting entry in the log is the control process message: 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb'

This one would indicate that someone (or something) gracefully stopped the database using the monetdbd client. Probably with the following command line call like: 'monetdb stop pbsworksdb'. Do not be confused by the order of logged events - monetdbd will first stop the process and later log that the call to stop the a database has been successful.

Best regards, Dimitar

...

On 2014-Oct-27, at 14:49 , Ashish Kumar Singh wrote:

Thanks!

Appreciate your response so there are few more people getting the same error :). Any suggestions from dev team on this issue? Can monetdbd exittimeout be also in picture for this?

Regards, Ashish

From: Pierre-Adrien Coustillas Date: Monday, 27 October 2014 6:51 pm To: "users-list@monetdb.org" Cc: "developers-list@monetdb.org" , Ashish Singh Subject: Re: MonetDB error while restarting the database

> Hello > > (google translate) > > I receive this error when there is no specific spacedisk. > Monetdb sometimes uses a lot of temporary diskspace >(500GBisnormalfor > 1TBof data), which arereleasedat the end oftreatment or after >acrash > > Error last week : > 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 > (database 'lemonde') the TERM signal > 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: >42014-10-22 > 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread > > Pierre > > -- > 1G6 > 52 route de bischwiller > 67300 Schiltigheim > Société de Services et de Formations en Logiciels Libres > http://1g6.biz > Tél : 06 64 63 70 35 > > De: "Ashish Kumar Singh" > À: "Ashish Kumar Singh" , > users-list@monetdb.org > Cc: developers-list@monetdb.org > Envoyé: Lundi 27 Octobre 2014 13:49:19 > Objet: Re: MonetDB error while restarting the database > > Guys, > > Any help with this will be really helpful for us? > > > > Regards, > Ashish > > From: Ashish Singh > Date: Monday, 27 October 2014 4:06 pm > To: "users-list@monetdb.org" > Subject: MonetDB error while restarting the database > >> >> All, >> >> >> >> We are facing a new issue with monetdb where monetdb connections >>and >> process is being killed very frequently. Is this a known issue or >>any >> suggestion in getting more details will be helpful. >> >> >> >> Regards, >> Ashish >> >> Mervogian.log: >> >> >> >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: >> >>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>5 >> >>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>s >>wor >> ksdata/pbsworksdb --set >> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >> mapi_open=false --set mapi_port=0 --set >> >>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>p >>bsw >> orksdata/pbsworksdb/.mapi.sock --set >> >>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>e >>tdb >> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >> max_clients=64 --set sql_optimizer=default_pipe --set >>monet_daemon=yes >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server >> v11.17.17 "Jan2014-SP2" >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database >> 'pbsworksdb', using 4 threads >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for >> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB >> available main-memory. >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) >>1993-July >> 2008 CWI. >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August >> 2008-2014 MonetDB B.V., all rights reserved >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit >> http://www.monetdb.org/ for further information >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX >>domain >> connection requests on >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module >>loaded >> 2014-10-27 09:01:00 MSG control[23583]: (local): started >>'pbsworksdb' >> 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client >> localhost.localdomain:53844 for database 'pbsworksdb' to >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >> 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is >>on >> local UNIX domain socket, passing on filedescriptor instead of >>proxying >> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>list >> 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 >> (database 'pbsworksdb') the TERM signal >> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: >> 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing >>thread >> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >> (6351) has exited with exit status 0 >> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >>has >> shut down >> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >> 'pbsworksdb' >> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>list >> 2014-10-27 09:01:26 MSG merovingian[23583]: starting database >> 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 >>0.00 >> (648-648=0) >> 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: >> >>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>5 >> >>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>s >>wor >> ksdata/pbsworksdb --set >> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >> mapi_open=false --set mapi_port=0 --set >> >>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>p >>bsw >> orksdata/pbsworksdb/.mapi.sock --set >> >>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>e >>tdb >> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >> max_clients=64 --set sql_optimizer=default_pipe --set >>monet_daemon=yes >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server >> v11.17.17 "Jan2014-SP2" >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database >> 'pbsworksdb', using 4 threads >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for >> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB >> available main-memory. >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) >>1993-July >> 2008 CWI. >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August >> 2008-2014 MonetDB B.V., all rights reserved >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit >> http://www.monetdb.org/ for further information >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX >>domain >> connection requests on >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module >>loaded >> 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client >> localhost.localdomain:53853 for database 'pbsworksdb' to >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >> 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is >>on >> local UNIX domain socket, passing on filedescriptor instead of >>proxying >> 2014-10-27 09:01:32 MSG c >> >> >> >> >> > > _______________________________________________ > users-list mailing list > users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > _______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- Sjoerd Mullender