Scale Factor 2000 TPCH Database?
Has anyone successfully created a scale factor 2000 TPCH database with MonetDB? We're having some problems trying to do so. We don't have enough disk space to hold both the data (created by dbgen) and the MonetDB database, so we have to create the database in two steps. (Create the first half of the data as dbgen table files, load it, delete the dbgen files, create the second half of the data as dbgen table files, load it, then delete those files.) The problem shows up when loading the second half of the data: the mserver crashes and won't restart. This is what the dbfarm/merovingian.log says: 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' (39920) has crashed (dumped core) 2013-08-02 18:42:19 ERR merovingian[39857]: client error: database 'q1_s2000_n2' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' has crashed after start on 2013-08-02 18:42:19, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.50 0.17 (5-0=5) Other databases on the same system are still running. We're running Feb13 SP1 with minor local modifications that are not coming into play at this point. We're running on a large Linux server. Is there anything we can look at to help diagnose the problem? Thanks, Steve Morgan
Hi Did you load all tables with/without key/foreign constraints? Did the system crash due to disk space limitations? (OS logs) regards, Martin On 8/2/13 9:50 PM, Stephen P. Morgan wrote:
Has anyone successfully created a scale factor 2000 TPCH database with MonetDB? We're having some problems trying to do so. We don't have enough disk space to hold both the data (created by dbgen) and the MonetDB database, so we have to create the database in two steps. (Create the first half of the data as dbgen table files, load it, delete the dbgen files, create the second half of the data as dbgen table files, load it, then delete those files.) The problem shows up when loading the second half of the data: the mserver crashes and won't restart. This is what the dbfarm/merovingian.log says:
2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' (39920) has crashed (dumped core) 2013-08-02 18:42:19 ERR merovingian[39857]: client error: database 'q1_s2000_n2' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' has crashed after start on 2013-08-02 18:42:19, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.50 0.17 (5-0=5)
Other databases on the same system are still running. We're running Feb13 SP1 with minor local modifications that are not coming into play at this point. We're running on a large Linux server.
Is there anything we can look at to help diagnose the problem?
Thanks,
Steve Morgan
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
Is your disk full when the crash occurs? That is a well-known cause of crashes. Note that when you load into an existing non-empty table, data needs to be copied, so you need more space than you would need if you did it in a single operation. On 2013-08-02 21:53, Martin Kersten wrote:
Hi
Did you load all tables with/without key/foreign constraints? Did the system crash due to disk space limitations? (OS logs)
regards, Martin
On 8/2/13 9:50 PM, Stephen P. Morgan wrote:
Has anyone successfully created a scale factor 2000 TPCH database with MonetDB? We're having some problems trying to do so. We don't have enough disk space to hold both the data (created by dbgen) and the MonetDB database, so we have to create the database in two steps. (Create the first half of the data as dbgen table files, load it, delete the dbgen files, create the second half of the data as dbgen table files, load it, then delete those files.) The problem shows up when loading the second half of the data: the mserver crashes and won't restart. This is what the dbfarm/merovingian.log says:
2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' (39920) has crashed (dumped core) 2013-08-02 18:42:19 ERR merovingian[39857]: client error: database 'q1_s2000_n2' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' has crashed after start on 2013-08-02 18:42:19, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.50 0.17 (5-0=5)
Other databases on the same system are still running. We're running Feb13 SP1 with minor local modifications that are not coming into play at this point. We're running on a large Linux server.
Is there anything we can look at to help diagnose the problem?
Thanks,
Steve Morgan
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- Sjoerd Mullender
That seems to be the issue, thanks.
________________________________ From: Sjoerd Mullender
To: users-list@monetdb.org Sent: Friday, August 2, 2013 1:15 PM Subject: Re: Scale Factor 2000 TPCH Database? Is your disk full when the crash occurs? That is a well-known cause of crashes. Note that when you load into an existing non-empty table, data needs to be copied, so you need more space than you would need if you did it in a single operation.
On 2013-08-02 21:53, Martin Kersten wrote:
Hi
Did you load all tables with/without key/foreign constraints? Did the system crash due to disk space limitations? (OS logs)
regards, Martin
On 8/2/13 9:50 PM, Stephen P. Morgan wrote:
Has anyone successfully created a scale factor 2000 TPCH database with MonetDB? We're having some problems trying to do so. We don't have enough disk space to hold both the data (created by dbgen) and the MonetDB database, so we have to create the database in two steps. (Create the first half of the data as dbgen table files, load it, delete the dbgen files, create the second half of the data as dbgen table files, load it, then delete those files.) The problem shows up when loading the second half of the data: the mserver crashes and won't restart. This is what the dbfarm/merovingian.log says:
2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' (39920) has crashed (dumped core) 2013-08-02 18:42:19 ERR merovingian[39857]: client error: database 'q1_s2000_n2' has crashed after starting, manual intervention needed, check monetdbd's logfile for details 2013-08-02 18:42:19 MSG merovingian[39857]: database 'q1_s2000_n2' has crashed after start on 2013-08-02 18:42:19, attempting restart, up min/avg/max: 0s/0s/0s, crash average: 1.00 0.50 0.17 (5-0=5)
Other databases on the same system are still running. We're running Feb13 SP1 with minor local modifications that are not coming into play at this point. We're running on a large Linux server.
Is there anything we can look at to help diagnose the problem?
Thanks,
Steve Morgan
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- Sjoerd Mullender
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Martin Kersten
-
Sjoerd Mullender
-
Stephen P. Morgan