[MonetDB-users] Database Failure
Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see this: 2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm 2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP connections on 0.0.0.0:50000 2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000 2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control 2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79 2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s) 2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal 2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8) 2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey 2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI. 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information 2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ 2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded 2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables 2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql' I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get this database back to a consistent state so that it is usable? 73, Matthew W. Jones (KI4ZIB) http://matburt.net
A little more information... when I start the server without sql and try to
initialize it from MAL I get more data:
mal>include sql;
MAPI = datawarehouse@localhost:50000
QUERY = include sql;
ERROR = !SQLException:SQLinit:Catalogue initialization failed
!ERROR: Incompatible database version 000000, this server supports
version 050000
!ERROR: Please move away
/var/MonetDB5/dbfarm/reporting_primary/sql_logs/sql/ and its corresponding
dbfarm.
#function user.main():void;
# sql.prelude();
#end main;
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 10:43 AM, Matthew Jones
Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see this:
2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm
2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP connections on 0.0.0.0:50000
2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000
2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control
2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79
2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s)
2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal
2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8)
2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI.
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded
2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables
2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities
The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql'
I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get this database back to a consistent state so that it is usable?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
Am I right in assuming that this is a bad transaction log file? I see where
these messages are emitted by gdk_logger.c Is there a way around replaying
this logfile?
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 11:01 AM, Matthew Jones
A little more information... when I start the server without sql and try to initialize it from MAL I get more data:
mal>include sql; MAPI = datawarehouse@localhost:50000 QUERY = include sql; ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away /var/MonetDB5/dbfarm/reporting_primary/sql_logs/sql/ and its corresponding dbfarm. #function user.main():void; # sql.prelude(); #end main;
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 10:43 AM, Matthew Jones
wrote: Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see this:
2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm
2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP connections on 0.0.0.0:50000
2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000
2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control
2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79
2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s)
2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal
2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8)
2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI.
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded
2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables
2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities
The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql'
I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get this database back to a consistent state so that it is usable?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
Hi,
I am not knowledgeable in these issues, but did you by any chance
upgraded monet during your uptime?
what version are you using?
lefteris
On Mon, Nov 23, 2009 at 5:01 PM, Matthew Jones
A little more information... when I start the server without sql and try to initialize it from MAL I get more data:
mal>include sql; MAPI = datawarehouse@localhost:50000 QUERY = include sql; ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away /var/MonetDB5/dbfarm/reporting_primary/sql_logs/sql/ and its corresponding dbfarm. #function user.main():void; # sql.prelude(); #end main;
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 10:43 AM, Matthew Jones
wrote: Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see this:
2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm
2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP connections on 0.0.0.0:50000
2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000
2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control
2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79
2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s)
2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal
2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8)
2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI.
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded
2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables
2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities
The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql'
I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get this database back to a consistent state so that it is usable?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No I have not, I deployed August 20009 SP2 as a new install and have not
upgraded it. It looks like Monet is looking at a transaction logfile when
it starts up... I can see this file in my sql_logs/sql directory and when
examining it, it looks like it has valid records in it but when the code in
gdk_logger.c:logger_new() calls check_version() it must return this bad
version (000000). Since that is the first thing that logger_new does when
examining existing transaction log files it returns an error (rightfully so)
before trying to read in the contents of the logfile. Unfortunately this
prevents the server from starting this database (or at least... it prevents
it from starting the 'sql' system), so I'm wondering if there is any way
around this? Obviously moving that transaction logfile out of the way
causes it to not be able to start either... so barring that, is there a way
to clean up this logfile to allow the database to function?
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 1:10 PM, Lefteris
Hi,
I am not knowledgeable in these issues, but did you by any chance upgraded monet during your uptime?
what version are you using?
lefteris
A little more information... when I start the server without sql and try to initialize it from MAL I get more data:
mal>include sql; MAPI = datawarehouse@localhost:50000 QUERY = include sql; ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away /var/MonetDB5/dbfarm/reporting_primary/sql_logs/sql/ and its corresponding dbfarm. #function user.main():void; # sql.prelude(); #end main;
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 10:43 AM, Matthew Jones
wrote: Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see
2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm
2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP
connections
on 0.0.0.0:50000
2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000
2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control
2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79
2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s)
2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal
2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8)
2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI.
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded
2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables
2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities
The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql'
I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get
On Mon, Nov 23, 2009 at 5:01 PM, Matthew Jones
wrote: this: this database back to a consistent state so that it is usable?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Matthew Jones wrote:
A little more information... when I start the server without sql and try to initialize it from MAL I get more data:
mal>include sql; MAPI = datawarehouse@localhost:50000 QUERY = include sql; ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away /var/MonetDB5/dbfarm/reporting_primary/sql_logs/sql/ and its corresponding dbfarm. #function user.main():void; # sql.prelude(); #end main;
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards. The feature to make such changes transparent to end-users is high on our list of todos. regards, Martin
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 10:43 AM, Matthew Jones
mailto:mat@matburt.net> wrote: Hey folks, had a database running for several weeks without any problems but early this morning anytime a client would try to connect to it the connection would just hang. After shutting down and restarting I see this:
2009-11-23 09:38:06 MSG merovingian[28036]: starting Merovingian 1.1 for dbfarm /opt/MonetDB5/dbfarm
2009-11-23 09:38:06 MSG merovingian[28036]: listening for TCP connections on 0.0.0.0:50000 http://0.0.0.0:50000
2009-11-23 09:38:06 MSG discovery[28036]: listening for UDP messages on 0.0.0.0:50000 http://0.0.0.0:50000
2009-11-23 09:38:06 MSG merovingian[28036]: handling commands over UNIX socket /opt/MonetDB5/dbfarm/.merovingian_control
2009-11-23 09:38:06 MSG discovery[28036]: new neighbour 10.1.0.79
2009-11-23 09:38:07 MSG discovery[28036]: new database mapi:monetdb:// hsdw01:50000/reporting_primary (ttl=660s)
2009-11-23 09:39:45 MSG merovingian[28036]: starting database 'reporting_primary' due to control signal
2009-11-23 09:39:45 MSG merovingian[28036]: database 'reporting_primary' has crashed after start on 2009-11-23 09:36:59, attempting restart, up min/avg/max: 2m/2d/1w, crash average: 1.00 0.60 0.27 (18-10=8)
2009-11-23 09:39:46 MSG reporting_primary[28067]: arguments: /usr/bin/mserver5 --config=/etc/monetdb5.conf --dbname=reporting_primary --dbinit=include sql; --set monet_daemon=yes --set mapi_open=false --set mapi_autosense=true --set mapi_port=50001 --set monet_vault_key=/opt/MonetDB5/dbfarm/reporting_primary /.vaultkey
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB server v5.14.2, based on kernel v1.32.2
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Serving database 'reporting_primary', using 8 threads
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) 1993-July 2008 CWI.
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Copyright (c) August 2008-2009 MonetDB B.V., all rights reserved
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Visit http://monetdb.cwi.nl / for further information
2009-11-23 09:39:46 MSG reporting_primary[28067]: # Listening for connection requests on mapi:monetdb://127.0.0.1:50001/ http://127.0.0.1:50001/
2009-11-23 09:39:46 MSG reporting_primary[28067]: # MonetDB/SQL module v2.32.2 loaded
2009-11-23 09:39:46 ERR reporting_primary[28067]: !mvc_init: unable to create system tables
2009-11-23 09:39:55 ERR merovingian[28036]: failed to fork mserver: database 'reporting_primary' has inconsistent state (running but dead), review merovingian's logfile for any peculiarities
The database marks itself as running but any attempt to connect to it returns the following error: monetdb ERROR MALException:setScenario:Scenario not initialized 'sql'
I've read other problems reports that had a similar issue and the suggestion was to wait for the sql module to allow connections but even after 5 minutes this is all that is returned. Is there any way to get this database back to a consistent state so that it is usable?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
------------------------------------------------------------------------
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Martin, this database was created with the August 2009 SP2 release, it was
never upgraded from an old release. In fact... this database has been
running since it was created and started with the August 2009 SP2 release...
there has never been any downtime until the whole thing deadlocked last
night.
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
It looks actually that the error is coming from store_init()
Would it be possible for you to run it with gdb and see the values of
the parameters of
mvc_init(int debug, store_type store, backend_stack stk)
I know thats too much to ask, and I am not en expert on this subject,
but it is an interesting situation:) Others may have a better idea
than me on this one.
lefteris
On Mon, Nov 23, 2009 at 7:37 PM, Matthew Jones
Martin, this database was created with the August 2009 SP2 release, it was never upgraded from an old release. In fact... this database has been running since it was created and started with the August 2009 SP2 release... there has never been any downtime until the whole thing deadlocked last night.
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten
wrote: Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
Hi Matthew,
I think I've seen such an error before, but I wasn't able to reproduce
it, and therefore did not report it. I think I deleted the old dbfarm
and started from scratch as my data was easy to reproduce. I assume
less destructive actions are feasible... perhaps deleting the contents
in the log folder might help too, but I don't want to be held
responsible.
Good luck,
Wouter
2009/11/23 Matthew Jones
Martin, this database was created with the August 2009 SP2 release, it was never upgraded from an old release. In fact... this database has been running since it was created and started with the August 2009 SP2 release... there has never been any downtime until the whole thing deadlocked last night.
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten
wrote: Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Nov 23, 2009 at 07:59:11PM +0100, Wouter Alink wrote:
Hi Matthew,
I think I've seen such an error before, but I wasn't able to reproduce it, and therefore did not report it. I think I deleted the old dbfarm and started from scratch as my data was easy to reproduce. I assume less destructive actions are feasible... perhaps deleting the contents in the log folder might help too, but I don't want to be held responsible.
Could you send us the content of your $prefix/dbfarm/$dbname/sql_logs/sql/log file? Somehow it got corrupted, we are unsure to why this happens. Niels
Good luck, Wouter
2009/11/23 Matthew Jones
: Martin, this database was created with the August 2009 SP2 release, it was never upgraded from an old release. In fact... this database has been running since it was created and started with the August 2009 SP2 release... there has never been any downtime until the whole thing deadlocked last night.
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten
wrote: Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
Hi Niels, interestingly enough.... the 'log' file is totally empty. So I
went back and examined a backup from a few days ago and the contents of the
file are:
"""
050000
17623
"""
Which corresponded with a 'log.17623' file that was in the directory at the
same time as the backup.
So it looks like maybe the new transaction log file that was hanging
around: log.22420
Can't even be picked up because it is referenced by the 'log' file at the
top of sql_logs/sql
Can I recreate this file with the right contents?
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 2:09 PM, Niels Nes
On Mon, Nov 23, 2009 at 07:59:11PM +0100, Wouter Alink wrote:
Hi Matthew,
I think I've seen such an error before, but I wasn't able to reproduce it, and therefore did not report it. I think I deleted the old dbfarm and started from scratch as my data was easy to reproduce. I assume less destructive actions are feasible... perhaps deleting the contents in the log folder might help too, but I don't want to be held responsible.
Could you send us the content of your $prefix/dbfarm/$dbname/sql_logs/sql/log file?
Somehow it got corrupted, we are unsure to why this happens.
Niels
Good luck, Wouter
2009/11/23 Matthew Jones
: Martin, this database was created with the August 2009 SP2 release, it
was
never upgraded from an old release. In fact... this database has been running since it was created and started with the August 2009 SP2 release... there has never been any downtime until the whole thing deadlocked last night.
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten
wrote:
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels http://www.cwi.nl/%7Eniels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Nov 23, 2009 at 02:19:33PM -0500, Matthew Jones wrote:
Hi Niels, interestingly enough.... the 'log' file is totally empty. So I went back and examined a backup from a few days ago and the contents of the file are:
""" 050000
17623 """ That looks indeed like a correct file. All the latest changes were in the 17623 log file.
By now probably we move on to 22420. Whats the file size of log.22420? It seems we got corruption somehow when we wrote the new log file. Is the log.22420 the only line (besides the version number) in the log file? Niels
Which corresponded with a 'log.17623' file that was in the directory at the same time as the backup.
So it looks like maybe the new transaction log file that was hanging around: log.22420
Can't even be picked up because it is referenced by the 'log' file at the top of sql_logs/sql
Can I recreate this file with the right contents?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 2:09 PM, Niels Nes
wrote: On Mon, Nov 23, 2009 at 07:59:11PM +0100, Wouter Alink wrote: > Hi Matthew, > > I think I've seen such an error before, but I wasn't able to reproduce > it, and therefore did not report it. I think I deleted the old dbfarm > and started from scratch as my data was easy to reproduce. I assume > less destructive actions are feasible... perhaps deleting the contents > in the log folder might help too, but I don't want to be held > responsible.
Could you send us the content of your $prefix/dbfarm/$dbname/sql_logs/sql/log file?
Somehow it got corrupted, we are unsure to why this happens.
Niels > > Good luck, > Wouter > > > > 2009/11/23 Matthew Jones
: > > Martin, this database was created with the August 2009 SP2 release, it was > > never upgraded from an old release. In fact... this database has been > > running since it was created and started with the August 2009 SP2 release... > > there has never been any downtime until the whole thing deadlocked last > > night. > > > > 73, > > Matthew W. Jones (KI4ZIB) > > http://matburt.net > > > > > > On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten < Martin.Kersten@cwi.nl> > > wrote: > >> > >> Indeed, this indicates that you have installed a new version of > >> MonetDB. One for which a warning was sent out to first dump > >> your database, before installing the new version and loading > >> the database again afterwards. > >> > >> The feature to make such changes transparent to end-users is > >> high on our list of todos. > >> > >> regards, Martin > >> > > > > > > ------------------------------------------------------------------------------ > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > > trial. Simplify your report design, integration and deployment - and focus > > on > > what you do best, core application coding. Discover what's new with > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
That actually worked... I recreated that file with the contents that I
mentioned below and everything came back up. Very interesting that that
file was empty after the crash.
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, Nov 23, 2009 at 2:19 PM, Matthew Jones
Hi Niels, interestingly enough.... the 'log' file is totally empty. So I went back and examined a backup from a few days ago and the contents of the file are:
""" 050000
17623 """
Which corresponded with a 'log.17623' file that was in the directory at the same time as the backup.
So it looks like maybe the new transaction log file that was hanging around: log.22420
Can't even be picked up because it is referenced by the 'log' file at the top of sql_logs/sql
Can I recreate this file with the right contents?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 2:09 PM, Niels Nes
wrote: On Mon, Nov 23, 2009 at 07:59:11PM +0100, Wouter Alink wrote:
Hi Matthew,
I think I've seen such an error before, but I wasn't able to reproduce it, and therefore did not report it. I think I deleted the old dbfarm and started from scratch as my data was easy to reproduce. I assume less destructive actions are feasible... perhaps deleting the contents in the log folder might help too, but I don't want to be held responsible.
Could you send us the content of your $prefix/dbfarm/$dbname/sql_logs/sql/log file?
Somehow it got corrupted, we are unsure to why this happens.
Niels
Good luck, Wouter
2009/11/23 Matthew Jones
: Martin, this database was created with the August 2009 SP2 release,
it was
never upgraded from an old release. In fact... this database has been running since it was created and started with the August 2009 SP2 release... there has never been any downtime until the whole thing deadlocked last night.
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten < Martin.Kersten@cwi.nl> wrote:
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
The feature to make such changes transparent to end-users is high on our list of todos.
regards, Martin
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels http://www.cwi.nl/%7Eniels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Nov 23, 2009 at 02:30:14PM -0500, Matthew Jones wrote:
That actually worked... I recreated that file with the contents that I mentioned below and everything came back up. Very interesting that that file was empty after the crash.
Great, now I just have to find out why the file was empty. Niels
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 2:19 PM, Matthew Jones
wrote: Hi Niels, interestingly enough.... the 'log' file is totally empty. So I went back and examined a backup from a few days ago and the contents of the file are:
""" 050000
17623 """
Which corresponded with a 'log.17623' file that was in the directory at the same time as the backup.
So it looks like maybe the new transaction log file that was hanging around: log.22420
Can't even be picked up because it is referenced by the 'log' file at the top of sql_logs/sql
Can I recreate this file with the right contents?
73, Matthew W. Jones (KI4ZIB) http://matburt.net
On Mon, Nov 23, 2009 at 2:09 PM, Niels Nes
wrote: On Mon, Nov 23, 2009 at 07:59:11PM +0100, Wouter Alink wrote: > Hi Matthew, > > I think I've seen such an error before, but I wasn't able to reproduce > it, and therefore did not report it. I think I deleted the old dbfarm > and started from scratch as my data was easy to reproduce. I assume > less destructive actions are feasible... perhaps deleting the contents > in the log folder might help too, but I don't want to be held > responsible.
Could you send us the content of your $prefix/dbfarm/$dbname/sql_logs/sql/log file?
Somehow it got corrupted, we are unsure to why this happens.
Niels > > Good luck, > Wouter > > > > 2009/11/23 Matthew Jones
: > > Martin, this database was created with the August 2009 SP2 release, it was > > never upgraded from an old release. In fact... this database has been > > running since it was created and started with the August 2009 SP2 release... > > there has never been any downtime until the whole thing deadlocked last > > night. > > > > 73, > > Matthew W. Jones (KI4ZIB) > > http://matburt.net > > > > > > On Mon, Nov 23, 2009 at 1:33 PM, Martin Kersten < Martin.Kersten@cwi.nl> > > wrote: > >> > >> Indeed, this indicates that you have installed a new version of > >> MonetDB. One for which a warning was sent out to first dump > >> your database, before installing the new version and loading > >> the database again afterwards. > >> > >> The feature to make such changes transparent to end-users is > >> high on our list of todos. > >> > >> regards, Martin > >> > > > > > > ------------------------------------------------------------------------------ > > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > > trial. Simplify your report design, integration and deployment - and focus > > on > > what you do best, core application coding. Discover what's new with > > Crystal Reports now. http://p.sf.net/sfu/bobj-july > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > MonetDB-users mailing list > MonetDB-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users -- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
On Mon, Nov 23, 2009 at 08:35:49PM +0100, Niels Nes wrote:
On Mon, Nov 23, 2009 at 02:30:14PM -0500, Matthew Jones wrote:
That actually worked... I recreated that file with the contents that I mentioned below and everything came back up. Very interesting that that file was empty after the crash.
Great, now I just have to find out why the file was empty.
This couldn't be a case of bad memory could it? I'm more of a user of Postgres and it happens surprisingly often there---i.e. PG misbehaves, user goes away and runs a RAM tester and it reports some bad memory. Symptoms are about right; code running for a reasonable amount of time with what (in my experience, but not sure if this applies here) is normally a regular workload and spontaneously something breaks. -- Sam http://samason.me.uk/
Memory tests were run and no problems were reported. With MonetDB+bad
memory I would expect to lose some data and no data was lost here.
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Tue, Nov 24, 2009 at 8:59 AM, Sam Mason
On Mon, Nov 23, 2009 at 08:35:49PM +0100, Niels Nes wrote:
On Mon, Nov 23, 2009 at 02:30:14PM -0500, Matthew Jones wrote:
That actually worked... I recreated that file with the contents that I mentioned below and everything came back up. Very interesting that that file was empty after the crash.
Great, now I just have to find out why the file was empty.
This couldn't be a case of bad memory could it? I'm more of a user of Postgres and it happens surprisingly often there---i.e. PG misbehaves, user goes away and runs a RAM tester and it reports some bad memory.
Symptoms are about right; code running for a reasonable amount of time with what (in my experience, but not sure if this applies here) is normally a regular workload and spontaneously something breaks.
-- Sam http://samason.me.uk/
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Tue, Nov 24, 2009 at 09:42:08AM -0500, Matthew Jones wrote:
Memory tests were run and no problems were reported. With MonetDB+bad memory I would expect to lose some data and no data was lost here.
You lost the contents of one file didn't you? Hits on memory from cosmic rays cause random single bit flips at a small but measurable rate--though ECC memory should help. These could cause something in Linux's page buffer to go awry and hence ending up with an empty file when unexpected. The bug rate of normal software means that that these sorts of errors are almost completely masked, however when code matures these sorts of errors start to become important. I'm not sure if Monet is stable enough to think about this yet, but if you've got a stable access pattern that only hits exactly the same code paths then it shouldn't matter about the remaining bugs. Higher assurance software tends to include checksums and other simple invariants that are checked at various places in code to make sure that errors like this aren't propagated too far. -- Sam http://samason.me.uk/
I tend to look at software problems before I look at hardware problems...
I'm much more inclined to believe that the deadlock was a mutex issue and
the timing of it happening caused the transaction log record file to be
truncated. Hits on memory from cosmic rays? That's rather grandiose... I
think you should go re-read Occam's Razor.
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Tue, Nov 24, 2009 at 9:55 AM, Sam Mason
On Tue, Nov 24, 2009 at 09:42:08AM -0500, Matthew Jones wrote:
Memory tests were run and no problems were reported. With MonetDB+bad memory I would expect to lose some data and no data was lost here.
You lost the contents of one file didn't you? Hits on memory from cosmic rays cause random single bit flips at a small but measurable rate--though ECC memory should help. These could cause something in Linux's page buffer to go awry and hence ending up with an empty file when unexpected. The bug rate of normal software means that that these sorts of errors are almost completely masked, however when code matures these sorts of errors start to become important. I'm not sure if Monet is stable enough to think about this yet, but if you've got a stable access pattern that only hits exactly the same code paths then it shouldn't matter about the remaining bugs.
Higher assurance software tends to include checksums and other simple invariants that are checked at various places in code to make sure that errors like this aren't propagated too far.
-- Sam http://samason.me.uk/
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On 23-11-2009 19:33:06 +0100, Martin Kersten wrote:
Matthew Jones wrote:
ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
It looks more like something got corrupt ( :( ), since version 000000 most probably isn't the previous one to 050000.
On Mon, Nov 23, 2009 at 08:11:54PM +0100, Fabian Groffen wrote:
On 23-11-2009 19:33:06 +0100, Martin Kersten wrote:
Matthew Jones wrote:
ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
It looks more like something got corrupt ( :( ), since version 000000 most probably isn't the previous one to 050000.
how about version 000000 is actually not read from the log, but the initial value in the code; the same error message is printed both when a version number cannot be read (correctly) from the log file and when the read version number does not match the supported one; cf. MonetDB/src/gdk/gdk_logger.mx ======== static int check_version(logger *lg, FILE *fp) { int version = 0; if (fscanf(fp, "%6d", &version) != 1 || version != lg->version) { GDKerror("Incompatible database version %06d, " "this server supports version %06d\n" "Please move away %s and its corresponding dbfarm.", version, lg->version, lg->dir); return -1; } fgetc(fp); /* skip \n */ fgetc(fp); /* skip \n */ return 0; } ======== Maybe, splitting the error message in two, one for fscanf() failure and one for non-matching version number might help to avoid confusion in future cases ... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Mon, Nov 23, 2009 at 08:41:11PM +0100, Stefan Manegold wrote:
On Mon, Nov 23, 2009 at 08:11:54PM +0100, Fabian Groffen wrote:
On 23-11-2009 19:33:06 +0100, Martin Kersten wrote:
Matthew Jones wrote:
ERROR = !SQLException:SQLinit:Catalogue initialization failed !ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away
Indeed, this indicates that you have installed a new version of MonetDB. One for which a warning was sent out to first dump your database, before installing the new version and loading the database again afterwards.
It looks more like something got corrupt ( :( ), since version 000000 most probably isn't the previous one to 050000.
how about version 000000 is actually not read from the log, but the initial value in the code; the same error message is printed both when a version number cannot be read (correctly) from the log file and when the read version number does not match the supported one; cf. MonetDB/src/gdk/gdk_logger.mx ======== static int check_version(logger *lg, FILE *fp) { int version = 0;
if (fscanf(fp, "%6d", &version) != 1 || version != lg->version) { GDKerror("Incompatible database version %06d, " "this server supports version %06d\n" "Please move away %s and its corresponding dbfarm.", version, lg->version, lg->dir);
return -1; } fgetc(fp); /* skip \n */ fgetc(fp); /* skip \n */ return 0; } ========
Maybe, splitting the error message in two, one for fscanf() failure and one for non-matching version number might help to avoid confusion in future cases ...
Indeed a good idea, but still we need to find out why the file was empty. Niels
Stefan
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- Niels Nes, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room C0.02/M3.46, phone ++31 20 592-4098 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl
participants (8)
-
Fabian Groffen
-
Lefteris
-
Martin Kersten
-
Matthew Jones
-
Niels Nes
-
Sam Mason
-
Stefan Manegold
-
Wouter Alink