[MonetDB-users] mserver memory grows, system thrashes and hangs

newer
[MonetDB-users] multi-users and...

Bostr

12 Dec 2008 12 Dec '08

10:28 p.m.

Hello MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array. I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows. The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below) Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions? Cheers Bob PS More detailed account of the problem below: Everything starts off fine, files load in 8-10 seconds, queries quite fast. As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases. At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable. Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too. top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it.. So I blew away, and recreated the database. This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO ' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. -- View this message in context: http://www.nabble.com/mserver-memory-grows%2C-system-thrashes-and-hangs-tp20... Sent from the monetdb-users mailing list archive at Nabble.com.

Attachments:

attachment.html (text/html — 3.3 KB)

Show replies by date

Arjen de Rijke

15 Dec 15 Dec

9:37 a.m.

New subject: [MonetDB-users] mserver memory grows, system thrashes and hangs

Bostr writes:

...

Hello

MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array.

I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows.

The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below)

Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions?

Cheers Bob

Hello Bob, I am experiencing similar problems. The testcase is different, but the symptoms are the same. In my case, the symptoms go away when i drop all constraints, suggesting that the problem is related to that. Is it possible for you to run your test without any primary/foreign key constraints on the database? Arjen de Rijke -- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ====================

...

PS More detailed account of the problem below:

Everything starts off fine, files load in 8-10 seconds, queries quite fast.

As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases.

At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable.

Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too.

top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal

The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it..

So I blew away, and recreated the database.

This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO

' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/ thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ View this message in context: mserver memory grows, system thrashes and hangs Sent from the monetdb-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Strahan, Bob

17 Dec 17 Dec

3:15 p.m.

New subject: [MonetDB-users] mserver memory grows, system thrashes and hangs

Hello Arjen, Thank you for your reply. I tried recreating the table with no constraints specified. Unfortunately, the memory growth still happened to the point where the system went into heavy swapping/thrashing. However, this time I was able to shut down and restart the database without seeing the corruption problem that I saw before.. And I was able to restart loads without getting foreign key violations (now there are no foreign keys). So this is a step in the right direction. However, the memory growth still worries me.. Do you think there might be a memory leak in the monetdb server.. Should we log a defect in Tracker? (I'm not sure what is the correct process) I am not sure if this memory growth is caused by the data loading, or by the queries, as I had both continually / concurrently running. So I have just started another test, this time doing data loads only (no concurrent queries), so we'll see if the problem re-occurs. Cheers Bob -----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Monday, December 15, 2008 4:38 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs Bostr writes:

...

Hello

MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array.

I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows.

The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below)

Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions?

Cheers Bob

...

PS More detailed account of the problem below:

Everything starts off fine, files load in 8-10 seconds, queries quite fast.

As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases.

At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable.

Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too.

top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal

The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it..

So I blew away, and recreated the database.

This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO

' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/ thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ View this message in context: mserver memory grows, system thrashes and hangs Sent from the monetdb-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Arjen de Rijke

18 Dec 18 Dec

8:36 a.m.

New subject: [MonetDB-users] mserver memory grows, system thrashes and hangs

"Strahan, Bob" writes:

...

Hello Arjen, Thank you for your reply.

I tried recreating the table with no constraints specified. Unfortunately, the memory growth still happened to the point where the system went into heavy swapping/thrashing.

However, this time I was able to shut down and restart the database without seeing the corruption problem that I saw before.. And I was able to restart loads without getting foreign key violations (now there are no foreign keys). So this is a step in the right direction.

However, the memory growth still worries me.. Do you think there might be a memory leak in the monetdb server.. Should we log a defect in Tracker? (I'm not sure what is the correct process)

Hello Bob, You can add a bugreport at sourceforge, http://sourceforge.net/tracker/?atid=482468&group_id=56967&func=browse Especially if your tests used to work in an earlier version.

...

I am not sure if this memory growth is caused by the data loading, or by the queries, as I had both continually / concurrently running. So I have just started another test, this time doing data loads only (no concurrent queries), so we'll see if the problem re-occurs.

Well, only loading the data should not cause these problems. We have a database with a table that has more than 13 billion records and loading that one worked without crashing the system. I assume you also removed the primary key on the table, so loading the data should work for you. if indeed this problem is caused by running queries while loading, there might be other explanations other than memory leaks. For example, transaction management might be involved. Please let us know the results of your test. Arjen de Rijke

...

Cheers Bob

-----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Monday, December 15, 2008 4:38 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs

Bostr writes:

...
Hello

MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array.

I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows.

The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below)

Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions?

Cheers Bob

Hello Bob,

I am experiencing similar problems. The testcase is different, but the symptoms are the same. In my case, the symptoms go away when i drop all constraints, suggesting that the problem is related to that.

Is it possible for you to run your test without any primary/foreign key constraints on the database?

Arjen de Rijke

-- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ====================

...
PS More detailed account of the problem below:

Everything starts off fine, files load in 8-10 seconds, queries quite fast.

As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases.

At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable.

Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too.

top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal

The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it..

So I blew away, and recreated the database.

This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO

' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/ thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ View this message in context: mserver memory grows, system thrashes and hangs Sent from the monetdb-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Strahan, Bob

6:26 p.m.

New subject: [MonetDB-users] mserver memory grows, system thrashes and hangs

Hi Arjen I ran data load tests again.. loaded 500,000 records (68MB) files every 20 seconds, and noticed that memory use gradually but steadily increased until eventually the system started swapping, and then load times (and everything else) rapidly degraded. This time I had no concurrent queries running. It seems that the LOAD is the culprit.

...

...
I assume you also removed the primary key on the table, so loading the data should work for you.

Correct. There is no primary key, and no foreign key constraints. However, I do have 'NOT NULL' specified for some fact table columns.. could that be a problem? Also, I have not modified any of the default gdk_memsize settings in the monetdb5.conf file.. Do you think that setting any of these would limit memory growth?

...

You can add a bugreport at sourceforge, http://sourceforge.net/tracker/?atid=482468&group_id=56967&func=browse Especially if your tests used to work in an earlier version.

This version is my first experience of monetdb, so I cannot say if this issue existed before.

...

...
if indeed this problem is caused by running queries while loading, there might be other explanations other than memory leaks. For example, transaction management might be involved.

Your comment on transaction management made me wonder. My loads are repeated statement executed by mclient.. as follows mclient -lsql --database=demo -s COPY 500000 RECORDS INTO fact from 'fact.csv' DELIMITERS ',' NULL AS '' This should automatically commit, right? Or is there any chance I'm accumulating uncommitted transactions here? Or, another thought, do I need to do anything to truncate transaction logs, or anything of that nature? Thanks in advance for your continued help. Cheers Bob -----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Thursday, December 18, 2008 3:36 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs "Strahan, Bob" writes:

...

Hello Arjen, Thank you for your reply.

I tried recreating the table with no constraints specified. Unfortunately, the memory growth still happened to the point where the system went into heavy swapping/thrashing.

However, this time I was able to shut down and restart the database without seeing the corruption problem that I saw before.. And I was able to restart loads without getting foreign key violations (now there are no foreign keys). So this is a step in the right direction.

However, the memory growth still worries me.. Do you think there might be a memory leak in the monetdb server.. Should we log a defect in Tracker? (I'm not sure what is the correct process)

Hello Bob, You can add a bugreport at sourceforge, http://sourceforge.net/tracker/?atid=482468&group_id=56967&func=browse Especially if your tests used to work in an earlier version.

...

I am not sure if this memory growth is caused by the data loading, or by the queries, as I had both continually / concurrently running. So I have just started another test, this time doing data loads only (no concurrent queries), so we'll see if the problem re-occurs.

...

Cheers Bob

-----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Monday, December 15, 2008 4:38 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs

Bostr writes:

...
Hello

MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array.

I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows.

The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below)

Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions?

Cheers Bob

Hello Bob,

I am experiencing similar problems. The testcase is different, but the symptoms are the same. In my case, the symptoms go away when i drop all constraints, suggesting that the problem is related to that.

Is it possible for you to run your test without any primary/foreign key constraints on the database?

Arjen de Rijke

-- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ====================

...
PS More detailed account of the problem below:

Everything starts off fine, files load in 8-10 seconds, queries quite fast.

As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases.

At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable.

Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too.

top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal

The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it..

So I blew away, and recreated the database.

This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO

' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/ thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ View this message in context: mserver memory grows, system thrashes and hangs Sent from the monetdb-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ==================== ------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

Niels Nes

7:55 p.m.

New subject: [MonetDB-users] mserver memory grows, system thrashes and hangs

On Thu, Dec 18, 2008 at 06:26:48PM +0000, Strahan, Bob wrote:

...

Hi Arjen

I ran data load tests again.. loaded 500,000 records (68MB) files every 20 seconds, and noticed that memory use gradually but steadily increased until eventually the system started swapping, and then load times (and everything else) rapidly degraded.

This time I had no concurrent queries running. It seems that the LOAD is the culprit.

...
...
I assume you also removed the primary key on the table, so loading the data should work for you.

Correct. There is no primary key, and no foreign key constraints. However, I do have 'NOT NULL' specified for some fact table columns.. could that be a problem?

Also, I have not modified any of the default gdk_memsize settings in the monetdb5.conf file.. Do you think that setting any of these would limit memory growth?

...
You can add a bugreport at sourceforge, http://sourceforge.net/tracker/?atid=482468&group_id=56967&func=browse Especially if your tests used to work in an earlier version.

This version is my first experience of monetdb, so I cannot say if this issue existed before.

...
...
if indeed this problem is caused by running queries while loading, there might be other explanations other than memory leaks. For example, transaction management might be involved.

Your comment on transaction management made me wonder. My loads are repeated statement executed by mclient.. as follows

mclient -lsql --database=demo -s COPY 500000 RECORDS INTO fact from 'fact.csv' DELIMITERS ',' NULL AS ''

This should automatically commit, right? Or is there any chance I'm accumulating uncommitted transactions here? Or, another thought, do I need to do anything to truncate transaction logs, or anything of that nature?

-s is a single statement. And the default mode of mclient is indeed auto-commit. Niels

...

Thanks in advance for your continued help.

Cheers Bob

-----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Thursday, December 18, 2008 3:36 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs

"Strahan, Bob" writes:

...
Hello Arjen, Thank you for your reply.

I tried recreating the table with no constraints specified. Unfortunately, the memory growth still happened to the point where the system went into heavy swapping/thrashing.

However, this time I was able to shut down and restart the database without seeing the corruption problem that I saw before.. And I was able to restart loads without getting foreign key violations (now there are no foreign keys). So this is a step in the right direction.

However, the memory growth still worries me.. Do you think there might be a memory leak in the monetdb server.. Should we log a defect in Tracker? (I'm not sure what is the correct process)

Hello Bob,

You can add a bugreport at sourceforge, http://sourceforge.net/tracker/?atid=482468&group_id=56967&func=browse Especially if your tests used to work in an earlier version.

...
I am not sure if this memory growth is caused by the data loading, or by the queries, as I had both continually / concurrently running. So I have just started another test, this time doing data loads only (no concurrent queries), so we'll see if the problem re-occurs.

Well, only loading the data should not cause these problems. We have a database with a table that has more than 13 billion records and loading that one worked without crashing the system. I assume you also removed the primary key on the table, so loading the data should work for you.

if indeed this problem is caused by running queries while loading, there might be other explanations other than memory leaks. For example, transaction management might be involved.

Please let us know the results of your test.

Arjen de Rijke

...
Cheers Bob

-----Original Message----- From: Arjen de Rijke [mailto:Arjen.de.Rijke@cwi.nl] Sent: Monday, December 15, 2008 4:38 AM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] mserver memory grows, system thrashes and hangs

Bostr writes:

...
Hello

MonetDB Nov2008 release, built with all standard options on RedHat EL 5 (2.6.18-92.el5 #1 SMP), on HP Proliant DL380 8-way, 16GB RAM / 16GB swap, with dbfarm on a RAID0 internal 7-disk array.

I'm doing bulk load tests (using mclient to run COPY INTO table FROM file), repeatedly loading 500,000 row (68M) CSV files into a fact table, while periodically running a suite of test queries in a separate process to test query times as the fact table grows.

The problem is that as the fact table grows to around 80Million records, the database starts to consume all available system memory and swap, the system thrashes and comes to its knees. I also get database corruption causing foreign key violations. (More details below)

Has anyone else seen this? Any ideas if it's a bug, memory leak? Or could this be caused by operator error? Suggestions?

Cheers Bob

Hello Bob,

I am experiencing similar problems. The testcase is different, but the symptoms are the same. In my case, the symptoms go away when i drop all constraints, suggesting that the problem is related to that.

Is it possible for you to run your test without any primary/foreign key constraints on the database?

Arjen de Rijke

-- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ====================

...
PS More detailed account of the problem below:

Everything starts off fine, files load in 8-10 seconds, queries quite fast.

As the table starts to fill, the queries (not unexpectedly) start to slow. But (unexpectedly) the memory consumption of the mserver5 process steadily increases.

At some point, around 80Million records, the system performance rapidly starts to degrade. Query times dramatically increase, load time also dramatically increases, and the entire system thrashes and becomes unusable.

Running 'top' shows that mserver5 has consumed vurtually all physical memory, and most of the swap too.

top - 16:33:34 up 28 days, 23:52, 8 users, load average: 1.68, 1.51, 1.96 Tasks: 224 total, 1 running, 223 sleeping, 0 stopped, 0 zombie Cpu(s): 0.1%us, 1.5%sy, 0.0%ni, 85.9%id, 12.4%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 16439196k total, 16298404k used, 140792k free, 2300k buffers Swap: 18876364k total, 13315960k used, 5560404k free, 9824352k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32380 monetdb 18 0 54.6g 14g 9.4g S 12 94.9 107:19.46 mserver5 29217 bostr 15 0 290m 5552 2264 S 1 0.0 12:01.37 gnome-terminal

The first time this happened, I stopped the load scripts, restarted the database, and that helped things for a little while, but it wasn't long before the memory was all consumed again, and performance went to the dogs. Then loads started failing with foreign-key violation errors.. Inspection showed a corrupted dimension table that somehow ended up with a bunch of bad values in it..

So I blew away, and recreated the database.

This time I tried adding the line 'gdk_nr_threads=1' to the monetdb5.conf file - having seen a defect report related to 'COPY INTO

' being buggy and this being the solution. But the same thing happened again.. it ran fine again until I had about 100Million rows, then memory problems/ thrashing. Restarted the server again, and shortly after the data loads failed again with foreign-key violations and a corrupted dimension table.. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ View this message in context: mserver memory grows, system thrashes and hangs Sent from the monetdb-users mailing list archive at Nabble.com.

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/... MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- ==================================================================== CWI, Kamer C0.03 Centrum voor Wiskunde en Informatica Kruislaan 413 Email: arjen.de.rijke@cwi.nl 1098 SJ Amsterdam tel: +31-(0)20-5924305 Nederland +31-(0)6-51899284 fax: +31-(0)20-5924312 ===================== http://www.cwi.nl/~rijke/ ====================

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

------------------------------------------------------------------------------ SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada. The future of the web can't happen without you. Join us at MIX09 to help pave the way to the Next Web now. Learn more and register at http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/ _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users

-- Niels Nes, Centre for Mathematics and Computer Science (CWI) Kruislaan 413, 1098 SJ Amsterdam, The Netherlands room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 url: http://www.cwi.nl/~niels e-mail: Niels.Nes@cwi.nl

5854

Age (days ago)

5860

Last active (days ago)

List overview

Download

5 comments

4 participants

participants (4)

Arjen de Rijke
Bostr
Niels Nes
Strahan, Bob