Memory allocations in MonetDB
Hi Team, I am running monetdb with default settings. After profiling the application instructions, I have noticed that all the memory allocation are currently being performed by the following functions: GDKmalloc, GDKmallocmax, GDKzalloc, HEAPalloc So mmap and malloc are being used with in all GDK* and HEAPalloc. I have three questions please: 1) Which memory allocations of monetdb are performed through mmap and which memory allocation are performed through malloc. I know this is very open ended question but little bit info about this would be helpful. Are the BAT structures allocated through mmap or malloc? 2) One thing I noticed that, when the DB is running and even if I am not running any queries, still the DB is constantly issuing load/store instructions to Main memory. I assume that some thread in monetdb is doing bookkeeping or persistence related stuff. Can you please point out which part of the code is responsible for this? I want to exclude that part of the code from profiling 3) When we start monetdb with multiple threads, whether multiple threads use the same common virtual address memory region to allocate memory? Or each thread has their own distinct memory region? Thanks. Kind Regards, Ahmad
On Mon, Feb 04, 2013 at 01:41:47PM +0000, Hassan, Ahmad wrote:
Hi Team,
I am running monetdb with default settings. After profiling the application instructions, I have noticed that all the memory allocation are currently being performed by the following functions:
GDKmalloc, GDKmallocmax, GDKzalloc, HEAPalloc
So mmap and malloc are being used with in all GDK* and HEAPalloc. I have three questions please:
1) Which memory allocations of monetdb are performed through mmap and which memory allocation are performed through malloc. I know this is very open ended question but little bit info about this would be helpful. Are the BAT structures allocated through mmap or malloc?
In short, small chunks are malloced, larger chunks are mmapped; the details are in the code of the functions you mentioned above, and the code is the most concices description of them.
2) One thing I noticed that, when the DB is running and even if I am not running any queries, still the DB is constantly issuing load/store instructions to Main memory. I assume that some thread in monetdb is doing bookkeeping or persistence related stuff. Can you please point out which part of the code is responsible for this? I want to exclude that part of the code from profiling
Here's what an "idle" mserver5 does: (gdb) thread apply all bt Thread 5 (Thread 0x7fffee602700 (LWP 3852)): #0 0x0000003cf30ea643 in select () from /lib64/libc.so.6 #1 0x00007ffff72c22e3 in MT_sleep_ms (ms=50) at.../MonetDB/gdk/gdk_posix.c:1003 #2 0x00007fffef529ded in store_manager () at.../MonetDB/sql/storage/store.c:1533 #3 0x00007fffef4b6813 in mvc_logmanager () at.../MonetDB/sql/server/sql_mvc.c:149 #4 0x0000003cf3c07d90 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003cf30f119d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fffee803700 (LWP 3851)): #0 0x0000003cf30ea643 in select () from /lib64/libc.so.6 #1 0x00007ffff7bf8fc5 in SERVERlistenThread (Sock=0x194ae50) at.../MonetDB/monetdb5/modules/mal/mal_mapi.c:209 #2 0x0000003cf3c07d90 in start_thread () from /lib64/libpthread.so.0 #3 0x0000003cf30f119d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ffff0527700 (LWP 3850)): #0 0x0000003cf30ea643 in select () from /lib64/libc.so.6 #1 0x00007ffff72c22e3 in MT_sleep_ms (ms=1000) at.../MonetDB/gdk/gdk_posix.c:1003 #2 0x00007ffff7a63414 in profilerHeartbeat (dummy=0x0) at.../MonetDB/monetdb5/mal/mal_profiler.c:1431 #3 0x0000003cf3c07d90 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003cf30f119d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ffff0728700 (LWP 3849)): #0 0x0000003cf30ea643 in select () from /lib64/libc.so.6 #1 0x00007ffff72c22e3 in MT_sleep_ms (ms=50) at.../MonetDB/gdk/gdk_posix.c:1003 #2 0x00007ffff71ff84c in GDKvmtrim (limit=0x7ffff78e50f8) at.../MonetDB/gdk/gdk_utils.c:918 #3 0x0000003cf3c07d90 in start_thread () from /lib64/libpthread.so.0 #4 0x0000003cf30f119d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffff6ccd800 (LWP 3822)): #0 0x0000003cf3c0e62d in read () from /lib64/libpthread.so.0 #1 0x0000003cfe82a131 in rl_getc () from /lib64/libreadline.so.6 #2 0x0000003cfe82a8e9 in rl_read_key () from /lib64/libreadline.so.6 #3 0x0000003cfe815b11 in readline_internal_char () from /lib64/libreadline.so.6 #4 0x0000003cfe816065 in readline () from /lib64/libreadline.so.6 #5 0x00007ffff7a6424d in getConsoleInput (c=0x647100, prompt=0x63b920 ">", linemode=0, exit_on_error=1) at.../MonetDB/monetdb5/mal/mal_readline.c:329 #6 0x00007ffff7a6486a in readConsole (cntxt=0x647100) at.../MonetDB/monetdb5/mal/mal_readline.c:473 #7 0x00007ffff7a74a42 in MALreader (c=0x647100) at.../MonetDB/monetdb5/mal/mal_session.c:491 #8 0x00007ffff7a73473 in runPhase (c=0x647100, phase=0) at.../MonetDB/monetdb5/mal/mal_scenario.c:522 #9 0x00007ffff7a73572 in runScenarioBody (c=0x647100) at.../MonetDB/monetdb5/mal/mal_scenario.c:552 #10 0x00007ffff7a7377f in runScenario (c=0x647100) at.../MonetDB/monetdb5/mal/mal_scenario.c:586 #11 0x00007ffff7a747f8 in MSserveClient (dummy=0x647100) at.../MonetDB/monetdb5/mal/mal_session.c:431 #12 0x0000000000403760 in main (argc=1, av=0x7fffffffd758) at.../MonetDB/tools/mserver/mserver5.c:622 (gdb)
3) When we start monetdb with multiple threads, whether multiple threads use the same common virtual address memory region to allocate memory? Or each thread has their own distinct memory region?
Well, they are threads, hence, they share the same address space. Stefan
Thanks.
Kind Regards, Ahmad
_______________________________________________ users-list mailing list users-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
participants (2)
-
Hassan, Ahmad -
Stefan Manegold