[Monetdb-developers] mclient running out of memory, crashing mserver5

newer
[Monetdb-developers] Results of...

sylver_b

20 Jan 2009 20 Jan '09

11:47 p.m.

Hi, I've finally found a better way to bulk load records via perl & odbc - a lot more stable. However, I've noticed many times mclient crashing mserver5 when loading records via the interactive interface. For instance, I tried to load about 130 inserts from a sql file using this command : mclient -lsql -d mydb < public_sysuser.sql After few seconds I started to see : #BBPTRIM: memtarget=17978753 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=273741960 [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] #BBPTRIM_ENTER: memsize=275281456,vmsize=275166768 #BBPTRIM: memtarget=19273201 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=275158816 [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] [ 1 ] #BBPTRIM_ENTER: memsize=276161752,vmsize=276047064 #BBPTRIM: memtarget=20153497 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=276056368 [ 1 ] #BBPTRIM_ENTER: memsize=276322608,vmsize=276207920 #BBPTRIM: memtarget=20314353 vmtarget=0 #BBPTRIM_EXIT: memsize=24100864,vmsize=276198064 Top is showing : top - 23:26:14 up 249 days, 9:57, 5 users, load average: 0.00, 0.20, 0.45 Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 2433324k used, 1592276k free, 191788k buffers Swap: 9502408k total, 92k used, 9502316k free, 1993392k cached I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ? Please advise, Thank you SB

Attachments:

attachment.html (text/html — 3.0 KB)

Show replies by date

Stefan de Konink

21 Jan 21 Jan

1:09 a.m.

New subject: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...

I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago).

So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to. Stefan

sylver_b

10:07 a.m.

New subject: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

Hi Interesting.. Even worst, this morning i found out that all the inserts of last night failed !! :( #BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached Is there anything to do to prevent this ? should the DB swap at all ? ________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5 sylver_b wrote:

...

I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

Stefan Manegold

10:28 a.m.

New subject: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...

Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed? How big is/was your mserver5 when running or just before crashing? Your machine and OS seem to be 64-bit, right? What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ? How much free space is on your disk partition where your dbfarm is located?

...

Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files. Stefan

...

________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

sylver_b

11:12 a.m.

New subject: [Monetdb-developers] Re : Re : mclient running out of memory, crashing mserver5

Hi Stephan, I will reply below : ________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...

...
Yes

How big is/was your mserver5 when running or just before crashing?

...

...
missed to check that, but this is what i have at the moment : USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | \_ mserver5 --dbinit=include sql;

Your machine and OS seem to be 64-bit, right?

...

...
32-bit

What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?

...

...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld

...

...
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 212G 180G 22G 90% / varrun 2.0G 56K 2.0G 1% /var/run varlock 2.0G 0 2.0G 0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb 10M 52K 10M 1% /proc/bus/usb udev 10M 52K 10M 1% /dev devshm 2.0G 0 2.0G 0% /dev/shm

...

Is there anything to do to prevent this ? should the DB swap at all ?

...

________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

...

Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

Stefan Manegold

11:48 a.m.

New subject: [Monetdb-developers] Re : Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:

...

Hi Stephan, I will reply below :

________________________________

De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...
...
Yes

How big is/was your mserver5 when running or just before crashing?

...
...
missed to check that, but this is what i have at the moment : USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | \_ mserver5 --dbinit=include sql;

This while being idle or while loading data? In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).

...

Your machine and OS seem to be 64-bit, right?

...
...
32-bit

...

What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?

...
...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld

Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB. The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted. As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data. Hope this helps ... Stefan

...

...
...
# df -h Filesystem            Size Used Avail Use% Mounted on /dev/sda1             212G 180G   22G 90% / varrun                2.0G   56K 2.0G   1% /var/run varlock               2.0G     0 2.0G   0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb             10M   52K   10M   1% /proc/bus/usb udev                   10M   52K   10M   1% /dev devshm                2.0G     0 2.0G   0% /dev/shm

...
Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.

Stefan

...
________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...
Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total,   1 running, 85 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:   4025600k total, 3633228k used,   392372k free,   107544k buffers Swap: 9502408k total,       92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

sylver_b

12:12 p.m.

New subject: [Monetdb-developers] Re : Re : Re : mclient running out of memory, crashing mserver5

Hi Stefan, Very interesting. Thanks for your detailed reply. I will go for the 64 bit architecture and re-run the trial. Thanks again SB ________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5 On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:

...

Hi Stephan, I will reply below :

________________________________

De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...
...
Yes

How big is/was your mserver5 when running or just before crashing?

...
...
missed to check that, but this is what i have at the moment : USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | \_ mserver5 --dbinit=include sql;

...

Your machine and OS seem to be 64-bit, right?

...
...
32-bit

...

What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?

...
...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld

...

...
...
# df -h Filesystem            Size Used Avail Use% Mounted on /dev/sda1             212G 180G   22G 90% / varrun                2.0G   56K 2.0G   1% /var/run varlock               2.0G     0 2.0G   0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb             10M   52K   10M   1% /proc/bus/usb udev                   10M   52K   10M   1% /dev devshm                2.0G     0 2.0G   0% /dev/shm

...
Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.

Stefan

...
________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...
Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total,   1 running, 85 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:   4025600k total, 3633228k used,   392372k free,   107544k buffers Swap: 9502408k total,       92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

sylver_b

12:16 p.m.

New subject: [Monetdb-developers] Re : Re : Re : mclient running out of memory, crashing mserver5

By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ? ________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5 On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:

...

Hi Stephan, I will reply below :

________________________________

De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...
...
Yes

How big is/was your mserver5 when running or just before crashing?

...
...
missed to check that, but this is what i have at the moment : USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ 10:05 0:00 | \_ mserver5 --dbinit=include sql;

...

Your machine and OS seem to be 64-bit, right?

...
...
32-bit

...

What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ?

...
...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking : /usr/bin/ld

...

...
...
# df -h Filesystem            Size Used Avail Use% Mounted on /dev/sda1             212G 180G   22G 90% / varrun                2.0G   56K 2.0G   1% /var/run varlock               2.0G     0 2.0G   0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb             10M   52K   10M   1% /proc/bus/usb udev                   10M   52K   10M   1% /dev devshm                2.0G     0 2.0G   0% /dev/shm

...
Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.

Stefan

...
________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...
Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total,   1 running, 85 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:   4025600k total, 3633228k used,   392372k free,   107544k buffers Swap: 9502408k total,       92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Stefan Manegold

12:40 p.m.

New subject: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 04:16:00AM -0800, sylver_b wrote:

...

By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ?

MonetDB will use virtual memory, mostly memory mapped files, partly swap --- until the available disk capacity or address space is exceeded --- whatever limit comes first (holds for both 32- & 64-bit systems). Stefan

...

________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:

...
Hi Stephan, I will reply below :

________________________________

De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...
...
Yes

How big is/was your mserver5 when running or just before crashing?

...
...
missed to check that, but this is what i have at the moment : USER       PID     %CPU %MEM    VSZ        RSS     TTY      STAT START   TIME COMMAND root          31679 0.0       3.2          1950816 131056 pts/3     Sl+    10:05      0:00 |   \_ mserver5 --dbinit=include sql;

This while being idle or while loading data?

In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).

...
Your machine and OS seem to be 64-bit, right?

...
...
32-bit

...
What about your MonetDB installation? Could you please provide us with the output of     `mserver5 --version` ?

...
...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking    : /usr/bin/ld

Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.

The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.

As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.

Hope this helps ...

Stefan

...
...
...
# df -h Filesystem            Size Used Avail Use% Mounted on /dev/sda1             212G 180G   22G 90% / varrun                2.0G   56K 2.0G   1% /var/run varlock               2.0G     0 2.0G   0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb             10M   52K   10M   1% /proc/bus/usb udev                   10M   52K   10M   1% /dev devshm                2.0G     0 2.0G   0% /dev/shm

...
Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.

Stefan

...
________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...
Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total,   1 running, 85 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:   4025600k total, 3633228k used,   392372k free,   107544k buffers Swap: 9502408k total,       92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

sylver_b

1:32 p.m.

New subject: [Monetdb-developers] Re : Re : mclient running out of memory, crashing mserver5

Thanks.. ________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 12h40mn 10s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5 On Wed, Jan 21, 2009 at 04:16:00AM -0800, sylver_b wrote:

...

By the way ... what will happen if i only have 4GB of memory on a 64-bit system and table size over 10GB (for example) ?

...

________________________________ De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s Objet : Re : [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote:

...
Hi Stephan, I will reply below :

________________________________

De : Stefan Manegold À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s Objet : Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

...
...
Yes

How big is/was your mserver5 when running or just before crashing?

...
...
missed to check that, but this is what i have at the moment : USER       PID     %CPU %MEM    VSZ        RSS     TTY      STAT START   TIME COMMAND root          31679 0.0       3.2          1950816 131056 pts/3     Sl+    10:05      0:00 |   \_ mserver5 --dbinit=include sql;

This while being idle or while loading data?

In any case, with almost 2 GB vitual size, your mserver5 has obviously reached the addressable limit on a 32-bit system (well, theoretically, 32-bit systems can address 4 GB, but in practice processes are often only allowed to address 2 GB).

...
Your machine and OS seem to be 64-bit, right?

...
...
32-bit

...
What about your MonetDB installation? Could you please provide us with the output of     `mserver5 --version` ?

...
...
# mserver5 --version MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008- MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/for further information Configured for prefix: /root/MonetDB Libraries: openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May 2006) Compiled by: root@ulys Compilation: gcc -O2 -std=c99 Linking    : /usr/bin/ld

Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB at a time. Th efollwoing two limitations apply: each column of your tables (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, i.e., the addressable data size on 32-bit systems. When bulkloading data, all columns (BATs) of a table are active concurrently, hence, the total table size cannot exceed 2 GB.

The GDKmmap failure in you previous posting suggetst that MonetDB cannot allocate more memory (or to be more precise: address space), most probably as the available 2 GB address space is already exhausted.

As Stefan (de Konink) suggested, you can test whether loading your data succeeds without constraints (primary keys, foreign keys, etc.) defined on your table(s). If not, your data size might just be too large for a 32-bit MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or consider fragmenting your data.

Hope this helps ...

Stefan

...
...
...
# df -h Filesystem            Size Used Avail Use% Mounted on /dev/sda1             212G 180G   22G 90% / varrun                2.0G   56K 2.0G   1% /var/run varlock               2.0G     0 2.0G   0% /var/lock

How much free space is on your disk partition where your dbfarm is located? procbususb             10M   52K   10M   1% /proc/bus/usb udev                   10M   52K   10M   1% /dev devshm                2.0G     0 2.0G   0% /dev/shm

...
Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files.

Stefan

...
________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...
I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago). So, is there a way to increase the memory limit or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

...
Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, 0.04 Tasks: 86 total,   1 running, 85 sleeping,   0 stopped,   0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:   4025600k total, 3633228k used,   392372k free,   107544k buffers Swap: 9502408k total,       92k used, 9502316k free, 1520104k cached

Is that after mserver5 had crashed?

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5768

Age (days ago)

5769

Last active (days ago)

List overview

Download

9 comments

3 participants

participants (3)

Stefan de Konink
Stefan Manegold
sylver_b