On 27/10/14 17:42, Dimitar Nedev wrote:

Hi Ashish,

What happened there is monetdbd send a TERM signal to the mserver process (the actual MonetDB database process), which killed all running threads. The messages "#GDKexit: killing thread" means the the MonetDB kernel (GDK) killed a thread still working at the time the process was terminated. This is most likely an indication that the database was still processing something at that time. Depending on what it was actually doing, data corruption might occur. This can explain why monetdbd reports an internal error why truing to start the mserver process again.

Data corruption should *never* occur. If you have proof of (and a recipe for) data corruption, please file a bug report. Of course, if a thread was still working, the transaction it was working on will most likely not get committed. Otherwise the above is correct.

Check the merovingian.log file again for the events when monetdbd reported the internal error. There should be more information logged on why mserver cannot start up any more.

Best regards, Dimitar

...

On 2014-Oct-27, at 15:55 , Ashish Kumar Singh wrote:

Thanks Dimitar,

Yes we are restarting the data base and as part of that we are seeing this error.

I am more worried about the error below:

2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread

On java client side it says

monetdbd: internal error while starting mserver, please refer to the logs

Regards, Ashish

-----Original Message----- From: Dimitar Nedev Reply-To: "users-list@monetdb.org" Date: Monday, 27 October 2014 8:15 pm To: "users-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

Hi Ashish,

The exit timeout only tells monetdbd how long should it wait when trying to gracefully shut down a mserver process. According to the log, the mserver process was shut down within the limit, since there are no messages like "timeout of <exittimeout> seconds expired, sending process <PID> the KILL signal".

Now, the interesting entry in the log is the control process message: 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb'

This one would indicate that someone (or something) gracefully stopped the database using the monetdbd client. Probably with the following command line call like: 'monetdb stop pbsworksdb'. Do not be confused by the order of logged events - monetdbd will first stop the process and later log that the call to stop the a database has been successful.

Best regards, Dimitar

...

On 2014-Oct-27, at 14:49 , Ashish Kumar Singh wrote:

Thanks!

Appreciate your response so there are few more people getting the same error :). Any suggestions from dev team on this issue? Can monetdbd exittimeout be also in picture for this?

Regards, Ashish

From: Pierre-Adrien Coustillas Date: Monday, 27 October 2014 6:51 pm To: "users-list@monetdb.org" Cc: "developers-list@monetdb.org" , Ashish Singh Subject: Re: MonetDB error while restarting the database

...

Hello

(google translate)

I receive this error when there is no specific spacedisk. Monetdb sometimes uses a lot of temporary diskspace (500GBisnormalfor 1TBof data), which arereleasedat the end oftreatment or after acrash

Error last week : 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 (database 'lemonde') the TERM signal 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: 42014-10-22 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread

Pierre

-- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35

De: "Ashish Kumar Singh" À: "Ashish Kumar Singh" , users-list@monetdb.org Cc: developers-list@monetdb.org Envoyé: Lundi 27 Octobre 2014 13:49:19 Objet: Re: MonetDB error while restarting the database

Guys,

Any help with this will be really helpful for us?

Regards, Ashish

From: Ashish Singh Date: Monday, 27 October 2014 4:06 pm To: "users-list@monetdb.org" Subject: MonetDB error while restarting the database

...

All,

We are facing a new issue with monetdb where monetdb connections and process is being killed very frequently. Is this a known issue or any suggestion in getting more details will be helpful.

Regards, Ashish

Mervogian.log:

2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor ksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw orksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module loaded 2014-10-27 09:01:00 MSG control[23583]: (local): started 'pbsworksdb' 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client localhost.localdomain:53844 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 (database 'pbsworksdb') the TERM signal 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' (6351) has exited with exit status 0 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' has shut down 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb' 2014-10-27 09:01:26 MSG control[23583]: (local): served status list 2014-10-27 09:01:26 MSG merovingian[23583]: starting database 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 0.00 (648-648=0) 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: /opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver5 --dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbswor ksdata/pbsworksdb --set merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set mapi_open=false --set mapi_port=0 --set mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pbsw orksdata/pbsworksdb/.mapi.sock --set monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server v11.17.17 "Jan2014-SP2" 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database 'pbsworksdb', using 4 threads 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB available main-memory. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) 1993-July 2008 CWI. 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August 2008-2014 MonetDB B.V., all rights reserved 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit http://www.monetdb.org/ for further information 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX domain connection requests on mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module loaded 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client localhost.localdomain:53853 for database 'pbsworksdb' to mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2014-10-27 09:01:32 MSG c

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- Sjoerd Mullender

Hi All, I have filled an issue Bug 3610: exittimeout property not being honoured against monetdb. It looks like GDKExit error is coming too frequently due to all this and its not even waiting for default value of 60 seconds. Regards, Ashish -----Original Message----- From: Ashish Singh Date: Thursday, 30 October 2014 8:24 pm To: "developers-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

Thanks All!

we made sure now that database connection pool has released all connections before we shutdown monetdb.

But still infrequently we are seeing this error popping up and we have double checked that client has returned all the connections.

Before stopping the database a data load process happened from the client to monetdb, can it be contributed to that??

Regards, Ashish

-----Original Message----- From: Sjoerd Mullender Reply-To: "developers-list@monetdb.org" Date: Monday, 27 October 2014 11:07 pm To: "developers-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

On 27/10/14 17:42, Dimitar Nedev wrote:

...

Hi Ashish,

What happened there is monetdbd send a TERM signal to the mserver process (the actual MonetDB database process), which killed all running threads. The messages "#GDKexit: killing thread" means the the MonetDB kernel (GDK) killed a thread still working at the time the process was terminated. This is most likely an indication that the database was still processing something at that time. Depending on what it was actually doing, data corruption might occur. This can explain why monetdbd reports an internal error why truing to start the mserver process again.

Data corruption should *never* occur. If you have proof of (and a recipe for) data corruption, please file a bug report. Of course, if a thread was still working, the transaction it was working on will most likely not get committed.

Otherwise the above is correct.

...

Check the merovingian.log file again for the events when monetdbd reported the internal error. There should be more information logged on why mserver cannot start up any more.

Best regards, Dimitar

...

On 2014-Oct-27, at 15:55 , Ashish Kumar Singh wrote:

Thanks Dimitar,

Yes we are restarting the data base and as part of that we are seeing this error.

I am more worried about the error below:

2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing thread

On java client side it says

monetdbd: internal error while starting mserver, please refer to the logs

Regards, Ashish

-----Original Message----- From: Dimitar Nedev Reply-To: "users-list@monetdb.org" Date: Monday, 27 October 2014 8:15 pm To: "users-list@monetdb.org" Subject: Re: MonetDB error while restarting the database

...

Hi Ashish,

The exit timeout only tells monetdbd how long should it wait when trying to gracefully shut down a mserver process. According to the log, the mserver process was shut down within the limit, since there are no messages like "timeout of <exittimeout> seconds expired, sending process <PID> the KILL signal".

Now, the interesting entry in the log is the control process message: 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database 'pbsworksdb'

This one would indicate that someone (or something) gracefully stopped the database using the monetdbd client. Probably with the following command line call like: 'monetdb stop pbsworksdb'. Do not be confused by the order of logged events - monetdbd will first stop the process and later log that the call to stop the a database has been successful.

Best regards, Dimitar

...

On 2014-Oct-27, at 14:49 , Ashish Kumar Singh wrote:

Thanks!

Appreciate your response so there are few more people getting the same error :). Any suggestions from dev team on this issue? Can monetdbd exittimeout be also in picture for this?

Regards, Ashish

From: Pierre-Adrien Coustillas Date: Monday, 27 October 2014 6:51 pm To: "users-list@monetdb.org" Cc: "developers-list@monetdb.org" , Ashish Singh Subject: Re: MonetDB error while restarting the database

> Hello > > (google translate) > > I receive this error when there is no specific spacedisk. > Monetdb sometimes uses a lot of temporary diskspace >(500GBisnormalfor > 1TBof data), which arereleasedat the end oftreatment or after >acrash > > Error last week : > 2014-10-22 22:51:46 MSG merovingian[18865]: sending process 30608 > (database 'lemonde') the TERM signal > 2014-10-22 22:51:46 ERR merovingian[18865]: unknown state: >42014-10-22 > 22:51:46 ERR lemonde[30608]: #GDKexit: killing thread > > Pierre > > -- > 1G6 > 52 route de bischwiller > 67300 Schiltigheim > Société de Services et de Formations en Logiciels Libres > http://1g6.biz > Tél : 06 64 63 70 35 > > De: "Ashish Kumar Singh" > À: "Ashish Kumar Singh" , > users-list@monetdb.org > Cc: developers-list@monetdb.org > Envoyé: Lundi 27 Octobre 2014 13:49:19 > Objet: Re: MonetDB error while restarting the database > > Guys, > > Any help with this will be really helpful for us? > > > > Regards, > Ashish > > From: Ashish Singh > Date: Monday, 27 October 2014 4:06 pm > To: "users-list@monetdb.org" > Subject: MonetDB error while restarting the database > >> >> All, >> >> >> >> We are facing a new issue with monetdb where monetdb connections >>and >> process is being killed very frequently. Is this a known issue or >>any >> suggestion in getting more details will be helpful. >> >> >> >> Regards, >> Ashish >> >> Mervogian.log: >> >> >> >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: arguments: >> >>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>5 >> >>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>s >>wor >> ksdata/pbsworksdb --set >> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >> mapi_open=false --set mapi_port=0 --set >> >>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>p >>bsw >> orksdata/pbsworksdb/.mapi.sock --set >> >>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>e >>tdb >> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >> max_clients=64 --set sql_optimizer=default_pipe --set >>monet_daemon=yes >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB 5 server >> v11.17.17 "Jan2014-SP2" >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Serving database >> 'pbsworksdb', using 4 threads >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Compiled for >> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Found 15.675 GiB >> available main-memory. >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) >>1993-July >> 2008 CWI. >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Copyright (c) August >> 2008-2014 MonetDB B.V., all rights reserved >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Visit >> http://www.monetdb.org/ for further information >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # Listening for UNIX >>domain >> connection requests on >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock >> 2014-10-27 09:01:00 MSG pbsworksdb[6351]: # MonetDB/SQL module >>loaded >> 2014-10-27 09:01:00 MSG control[23583]: (local): started >>'pbsworksdb' >> 2014-10-27 09:01:00 MSG merovingian[23583]: proxying client >> localhost.localdomain:53844 for database 'pbsworksdb' to >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >> 2014-10-27 09:01:00 MSG merovingian[23583]: target connection is >>on >> local UNIX domain socket, passing on filedescriptor instead of >>proxying >> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>list >> 2014-10-27 09:01:26 MSG merovingian[23583]: sending process 6351 >> (database 'pbsworksdb') the TERM signal >> 2014-10-27 09:01:26 ERR merovingian[23583]: unknown state: >> 42014-10-27 09:01:26 ERR pbsworksdb[6351]: #GDKexit: killing >>thread >> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >> (6351) has exited with exit status 0 >> 2014-10-27 09:01:26 MSG merovingian[23583]: database 'pbsworksdb' >>has >> shut down >> 2014-10-27 09:01:26 MSG control[23583]: (local): stopped database >> 'pbsworksdb' >> 2014-10-27 09:01:26 MSG control[23583]: (local): served status >>list >> 2014-10-27 09:01:26 MSG merovingian[23583]: starting database >> 'pbsworksdb', up min/avg/max: 5s/14m/1h, crash average: 0.00 0.00 >>0.00 >> (648-648=0) >> 2014-10-27 09:01:27 MSG pbsworksdb[6575]: arguments: >> >>/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/bin/mserver >>5 >> >>--dbpath=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/pb >>s >>wor >> ksdata/pbsworksdb --set >> merovingian_uri=mapi:monetdb://blrec12-vm6:9200/pbsworksdb --set >> mapi_open=false --set mapi_port=0 --set >> >>mapi_usock=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/monetdb/ >>p >>bsw >> orksdata/pbsworksdb/.mapi.sock --set >> >>monet_vault_key=/opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mon >>e >>tdb >> /pbsworksdata/pbsworksdb/.vaultkey --set gdk_nr_threads=4 --set >> max_clients=64 --set sql_optimizer=default_pipe --set >>monet_daemon=yes >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB 5 server >> v11.17.17 "Jan2014-SP2" >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Serving database >> 'pbsworksdb', using 4 threads >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Compiled for >> x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Found 15.675 GiB >> available main-memory. >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) >>1993-July >> 2008 CWI. >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Copyright (c) August >> 2008-2014 MonetDB B.V., all rights reserved >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Visit >> http://www.monetdb.org/ for further information >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # Listening for UNIX >>domain >> connection requests on >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock >> 2014-10-27 09:01:32 MSG pbsworksdb[6575]: # MonetDB/SQL module >>loaded >> 2014-10-27 09:01:32 MSG merovingian[23583]: proxying client >> localhost.localdomain:53853 for database 'pbsworksdb' to >> >>mapi:monetdb:///opt/pbsworks/12.4_rc3.1_live/portal/thirdparty/mone >>t >>db/ >> pbsworksdata/pbsworksdb/.mapi.sock?database=pbsworksdb >> 2014-10-27 09:01:32 MSG merovingian[23583]: target connection is >>on >> local UNIX domain socket, passing on filedescriptor instead of >>proxying >> 2014-10-27 09:01:32 MSG c >> >> >> >> >> > > _______________________________________________ > users-list mailing list > users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list > _______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

_______________________________________________ developers-list mailing list developers-list@monetdb.org https://www.monetdb.org/mailman/listinfo/developers-list

-- Sjoerd Mullender