Re: Monetdb bulk copy crash

Medha, some of the MonetDB internal virtual memory mamagement has changed since version 11.9.7 (Apr2012-SP2), mostly to clean things up and --- for the workloads we tested --- improve scalability. During which part of you extended load process does MonetDB get killed by the OS? Already during the initial copy into, or only during on of the insert statements? BTW, instead of create table lubm1b_p1(sub int, obj int); insert into lubm1b_p1 select sub, obj from lubm1b where pred=1 order by sub, obj; you could also simply use create table lubm1b_p1 as select sub, obj from lubm1b where pred=1 order by sub, obj with data; Stefan On Sat, Nov 17, 2012 at 05:19:44PM -0500, Medha Atre wrote:

Actually the reason I found it mysterious is because if I load exact same dataset using same SQL commands with MonetDB 11.9.7 (I have both compiled on same machine), it works fine! OS doesn't kill it. The peak resident memory utilization goes up to 6.3 GB or so, and then drops below, with swap remaining pretty much empty (this is observed with MonetDB 11.9.7).

I have 8 GB of RAM and 4 GB of Swap space.

If you could please confirm that the data format remains the same and I can use 11.9.7 for loading the data and 11.13.3 for querying, it would be great (as I need to run some experiments which cannot wait).

On Sat, Nov 17, 2012 at 4:40 PM, Fabian Groffen wrote:

...

On 17-11-2012 16:35:38 -0500, Medha Atre wrote:

...

dataset (in integer format) of 1.33 billion triples. /var/log/syslog shows mserver5 killed by the kernel "oom_signal (Out Of Memory)". ... So I wonder what changed from 11.9.7 to 11.13.3 that the bulk loader started crashing on one dataset of 1.33 billion triples, but works fine for 800 million triples.

It doesn't crash. Your operating system kills it. MonetDB can't do anything to a decision like that taken by your OS. Your OS probably feels like MonetDB is using too much memory. You could try adding some swap. A good rule of thumb is having twice as much swap as main memory.

Fabian

_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list