Re: [Monetdb-developers] Re : mclient running out of memory, crashing mserver5

On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote:

Hi

Interesting..

Even worst, this morning i found out that all the inserts of last night failed !! :(

#BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 #BBPTRIM: memtarget=819717481 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 #BBPTRIM: memtarget=849042689 vmtarget=0 #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 #BBPTRIM: memtarget=0 vmtarget=19052584 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 #BBPTRIM: memtarget=0 vmtarget=35173384 #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 #BBPTRIM: memtarget=0 vmtarget=65975328 #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 #BBPTRIM: memtarget=0 vmtarget=2536432 #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 #BBPTRIM: memtarget=0 vmtarget=9483224 #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 #GDKmmap(148111360) fail => BBPtrim(enter) usage[mem=1143633920,vm=1496757208] #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 #BBPTRIM: memtarget=0 vmtarget=1073741824 #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 #GDKmmap(148111360) fail => BBPtrim(ready) usage[mem=1143633920,vm=1496757208] !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap !OS: Cannot allocate memory !ERROR: GDKload: failed name=32/3267, ext=theap

top - 10:00:34 up 249 days, 20:31,  4 users,  load average: 0.28, 0.12, 0.04 Tasks:  86 total,   1 running,  85 sleeping,   0 stopped,   0 zombie Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st Mem:   4025600k total,  3633228k used,   392372k free,   107544k buffers Swap:  9502408k total,       92k used,  9502316k free,  1520104k cached

Is that after mserver5 had crashed? How big is/was your mserver5 when running or just before crashing? Your machine and OS seem to be 64-bit, right? What about your MonetDB installation? Could you please provide us with the output of `mserver5 --version` ? How much free space is on your disk partition where your dbfarm is located?

Is there anything to do to prevent this ? should the DB swap at all ?

MonetDB does in-memory processing, i.e., all columns that are active at a time need to be in the processes address space, either being loaded or being memory mapped; so yes, with huge amount of data MonetDB will also use virtual memory, either as swap or as memory-mapped files. Stefan

________________________________ De : Stefan de Konink À : sylver_b Cc : monetdb-developers@lists.sourceforge.net Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s Objet : Re: [Monetdb-developers] mclient running out of memory, crashing mserver5

sylver_b wrote:

...

I don't think i'm running out of memory. I had to stop and restart mserver5, flush the table, then re-run the command - all the records were inserted. Also I noticed this type of messages on the mserver5 console when piping records through the mclient (over 1million records) . It seems that the mclient is quiet instable (cf my post about "COPY, terminating connection" few months ago).   So, is there a way to increase the memory limit  or make sure mserver5 won't crash when running bash inserts via a cron job ?

The only thing I was able to do against that; more swap for the job. And of course the standard stuff as loading CSV files, preferably no indices on the table you are loading huge amounts of data to.

Stefan

------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |