- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_instruction.mx, , 1.372, 1.373
by Stefan Manegold 26 Nov '08

26 Nov '08

On Wed, Nov 26, 2008 at 08:21:17PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13401 > > Modified Files: > mal_instruction.mx > Log Message: > An experiment for the nightly tests to address e.g. Darwin color test. > Should be handled correctly by GDK. > > > U mal_instruction.mx > Index: mal_instruction.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_instruction.mx,v > retrieving revision 1.372 > retrieving revision 1.373 > diff -u -d -r1.372 -r1.373 > --- mal_instruction.mx 23 Nov 2008 15:48:14 -0000 1.372 > +++ mal_instruction.mx 26 Nov 2008 20:21:14 -0000 1.373 > @@ -1,4 +1,4 @@ > -@' The contents of this file are subject to the MonetDB Public License > +' The contents of this file are subject to the MonetDB Public License > @' Version 1.1 (the "License"); you may not use this file except in > @' compliance with the License. You may obtain a copy of the License at > @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > @@ -1715,7 +1715,6 @@ > if( vr->vtype == type) > return 0; > if( vr->vtype == TYPE_str){ > -#ifdef STRUCT_ALIGNED > int ll = 0; > ptr d = NULL; > > @@ -1734,6 +1733,8 @@ > VALset(vr, type, d); > if (ATOMextern(type) == 0) > GDKfree(d); > +#ifdef STRUCT_ALIGNED > + /* experiment against alternative 32 bit platforms */ The problems does not seem to be 32-bit specific, but possibly big-endian related; it occurs on both 32- & 64-bit big-endian platforms such as PowerPC http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… and SPARC http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… similar problems occur with other tests on the same set of platforms: http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… http://monetdb.cwi.nl/testing/projects/monetdb/Current/MonetDB5/.mTests103/… Stefan > #else > showException(SYNTAX, "convertConstant", "missing implementation"); > #endif > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_instruction.mx, , 1.372, 1.373
by Stefan Manegold 26 Nov '08

26 Nov '08

On Wed, Nov 26, 2008 at 08:21:17PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13401 > > Modified Files: > mal_instruction.mx > Log Message: > An experiment for the nightly tests to address e.g. Darwin color test. > Should be handled correctly by GDK. > > > U mal_instruction.mx > Index: mal_instruction.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_instruction.mx,v > retrieving revision 1.372 > retrieving revision 1.373 > diff -u -d -r1.372 -r1.373 > --- mal_instruction.mx 23 Nov 2008 15:48:14 -0000 1.372 > +++ mal_instruction.mx 26 Nov 2008 20:21:14 -0000 1.373 > @@ -1,4 +1,4 @@ > -@' The contents of this file are subject to the MonetDB Public License > +' The contents of this file are subject to the MonetDB Public License ^^ Was this (also) intended? Stefan > @' Version 1.1 (the "License"); you may not use this file except in > @' compliance with the License. You may obtain a copy of the License at > @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > @@ -1715,7 +1715,6 @@ > if( vr->vtype == type) > return 0; > if( vr->vtype == TYPE_str){ > -#ifdef STRUCT_ALIGNED > int ll = 0; > ptr d = NULL; > > @@ -1734,6 +1733,8 @@ > VALset(vr, type, d); > if (ATOMextern(type) == 0) > GDKfree(d); > +#ifdef STRUCT_ALIGNED > + /* experiment against alternative 32 bit platforms */ > #else > showException(SYNTAX, "convertConstant", "missing implementation"); > #endif > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools/Tests mclient-lmal--help.stable.out.SunOS5.11.STATIC, Nov2008, NONE, 1.1.2.1
by Stefan Manegold 25 Nov '08

25 Nov '08

On Tue, Nov 25, 2008 at 01:45:28PM +0000, Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools/Tests > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv12238 > > Added Files: > Tag: Nov2008 > mclient-lmal--help.stable.out.SunOS5.11.STATIC ^^^^^^^ IMHO, this will not work since Mtest.py considers ".STATIC" specific stable output only in case MonetDB is "statically linked", i.e., configures with "--disable-shared --enable-static", i.e., extension modules are statically linked into Mserver --- a "feature" which is (soon: was) only supported by MonetDB version 4, but is not (has never been) supported by MonetDB version 5). ... I guess, we should rather try to find out, why the compilation seems to find a usable readline on the very same machine (build-x86-sol10) when using GNU gcc (http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB5/.mTests103/G…) while is apparently does not when using the Sun Studio 12 compiler (http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB5/.mTests103/S…) ... (Mtest.py does not (yet) provide functionality to have *compiler*-specific stable output, mainly because is (still) uses the stabdard welcome message of both Mserver & mserver5 to get configuration information --- but only mserver5's --version option reveals the compiler name ...) http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB5/.GNU.32.32.d… says ======== checking readline/readline.h usability... yes checking readline/readline.h presence... yes checking for readline/readline.h... yes checking for readline in -lreadline... yes checking for rl_completion_matches in -lreadline... yes ======== and ======== configure:31480: checking readline/readline.h usability configure:31497: gcc -c -O2 -D__EXTENSIONS__ -Wall -Wextra -std=c99 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt conftest.c >&5 configure:31503: $? = 0 configure:31517: result: yes configure:31521: checking readline/readline.h presence configure:31536: gcc -E conftest.c configure:31542: $? = 0 configure:31556: result: yes configure:31589: checking for readline/readline.h configure:31596: result: yes configure:31601: checking for readline in -lreadline configure:31636: gcc -o conftest -O2 -D__EXTENSIONS__ -Wall -Wextra -std=c99 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt conftest.c -lreadline >&5 configure:31642: $? = 0 configure:31660: result: yes configure:31815: checking for rl_completion_matches in -lreadline configure:31850: gcc -o conftest -O2 -D__EXTENSIONS__ -Wall -Wextra -std=c99 -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt conftest.c -lreadline -lreadline >&5 configure:31856: $? = 0 configure:31874: result: yes ======== while http://monetdb.cwi.nl/testing/projects/monetdb/Stable/MonetDB5/.Sun.32.32.d… says ======== checking readline/readline.h usability... no checking readline/readline.h presence... no checking for readline/readline.h... no ======== and ======== configure:31480: checking readline/readline.h usability configure:31497: cc -c -D__EXTENSIONS__ -xc99 -xO5 conftest.c >&5 "conftest.c", line 104: cannot find include file: <readline/readline.h> cc: acomp failed for conftest.c configure:31503: $? = 2 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "MonetDB Server" | #define PACKAGE_TARNAME "MonetDB5-server" | #define PACKAGE_VERSION "5.8.1" | #define PACKAGE_STRING "MonetDB Server 5.8.1" | #define PACKAGE_BUGREPORT "monet(a)cwi.nl" | #define PACKAGE "MonetDB5-server" | #define VERSION "5.8.1" | #define HOST "i386-pc-solaris2.11" | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define SIZEOF_LONG 4 | #define _XPG6 1 | #define __EXTENSIONS__ 1 | #define HAVE_C99 1 | #define HAVE_DLFCN_H 1 | #define YYTEXT_POINTER 1 | #define TIME_WITH_SYS_TIME 1 | #define HAVE_DIRENT_H 1 | #define HAVE_ALLOCA_H 1 | #define HAVE_GETOPT_H 1 | #define HAVE_NETDB_H 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_TIMES_H 1 | #define _FILE_OFFSET_BITS 64 | #define RETSIGTYPE void | #define HAVE_PTRDIFF_T 1 | #define HAVE_SSIZE_T 1 | #define HAVE_LONG_LONG 1 | #define LLFMT "%lld" | #define ULLFMT "%llu" | #define SIZEOF_CHAR 1 | #define SIZEOF_SHORT 2 | #define SIZEOF_INT 4 | #define SIZEOF_LONG 4 | #define SIZEOF_VOID_P 4 | #define SIZEOF_SIZE_T 4 | #define SIZEOF_SSIZE_T 4 | #define SIZEOF_PTRDIFF_T 4 | #define SIZEOF_LONG_LONG 8 | #define SIZEOF___INT64 0 | #define HAVE_FTRUNCATE 1 | #define HAVE_GETTIMEOFDAY 1 | #define HAVE_OPENDIR 1 | #define HAVE_SYSCONF 1 | #define HAVE_TIMES 1 | #define HAVE_FTELLO 1 | #define HAVE_FSEEKO 1 | #define DIR_SEP '/' | #define DIR_SEP_STR "/" | #define PATH_SEP ':' | #define PATH_SEP_STR ":" | #define SO_PREFIX "lib" | #define SO_EXT ".so" | #define NDEBUG 1 | #define HAVE_PTHREAD_H 1 | #define HAVE_SEMAPHORE_H 1 | #define HAVE_SCHED_H 1 | #define HAVE_PTHREAD_KILL 1 | #define HAVE_PTHREAD_SIGMASK 1 | #define HAVE_PTHREAD_SETSCHEDPRIO 1 | #define HAVE_LIBPTHREAD 1 | /* end confdefs.h. */ | #include <stdio.h> | #ifdef HAVE_SYS_TYPES_H | # include <sys/types.h> | #endif | #ifdef HAVE_SYS_STAT_H | # include <sys/stat.h> | #endif | #ifdef STDC_HEADERS | # include <stdlib.h> | # include <stddef.h> | #else | # ifdef HAVE_STDLIB_H | # include <stdlib.h> | # endif | #endif | #ifdef HAVE_STRING_H | # if !defined STDC_HEADERS && defined HAVE_MEMORY_H | # include <memory.h> | # endif | # include <string.h> | #endif | #ifdef HAVE_STRINGS_H | # include <strings.h> | #endif | #ifdef HAVE_INTTYPES_H | # include <inttypes.h> | #endif | #ifdef HAVE_STDINT_H | # include <stdint.h> | #endif | #ifdef HAVE_UNISTD_H | # include <unistd.h> | #endif | #include <readline/readline.h> configure:31517: result: no configure:31521: checking readline/readline.h presence configure:31536: cc -E conftest.c "conftest.c", line 71: cannot find include file: <readline/readline.h> cc: acomp failed for conftest.c configure:31542: $? = 2 configure: failed program was: | /* confdefs.h. */ | #define PACKAGE_NAME "MonetDB Server" | #define PACKAGE_TARNAME "MonetDB5-server" | #define PACKAGE_VERSION "5.8.1" | #define PACKAGE_STRING "MonetDB Server 5.8.1" | #define PACKAGE_BUGREPORT "monet(a)cwi.nl" | #define PACKAGE "MonetDB5-server" | #define VERSION "5.8.1" | #define HOST "i386-pc-solaris2.11" | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define SIZEOF_LONG 4 | #define _XPG6 1 | #define __EXTENSIONS__ 1 | #define HAVE_C99 1 | #define HAVE_DLFCN_H 1 | #define YYTEXT_POINTER 1 | #define TIME_WITH_SYS_TIME 1 | #define HAVE_DIRENT_H 1 | #define HAVE_ALLOCA_H 1 | #define HAVE_GETOPT_H 1 | #define HAVE_NETDB_H 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_TIMES_H 1 | #define _FILE_OFFSET_BITS 64 | #define RETSIGTYPE void | #define HAVE_PTRDIFF_T 1 | #define HAVE_SSIZE_T 1 | #define HAVE_LONG_LONG 1 | #define LLFMT "%lld" | #define ULLFMT "%llu" | #define SIZEOF_CHAR 1 | #define SIZEOF_SHORT 2 | #define SIZEOF_INT 4 | #define SIZEOF_LONG 4 | #define SIZEOF_VOID_P 4 | #define SIZEOF_SIZE_T 4 | #define SIZEOF_SSIZE_T 4 | #define SIZEOF_PTRDIFF_T 4 | #define SIZEOF_LONG_LONG 8 | #define SIZEOF___INT64 0 | #define HAVE_FTRUNCATE 1 | #define HAVE_GETTIMEOFDAY 1 | #define HAVE_OPENDIR 1 | #define HAVE_SYSCONF 1 | #define HAVE_TIMES 1 | #define HAVE_FTELLO 1 | #define HAVE_FSEEKO 1 | #define DIR_SEP '/' | #define DIR_SEP_STR "/" | #define PATH_SEP ':' | #define PATH_SEP_STR ":" | #define SO_PREFIX "lib" | #define SO_EXT ".so" | #define NDEBUG 1 | #define HAVE_PTHREAD_H 1 | #define HAVE_SEMAPHORE_H 1 | #define HAVE_SCHED_H 1 | #define HAVE_PTHREAD_KILL 1 | #define HAVE_PTHREAD_SIGMASK 1 | #define HAVE_PTHREAD_SETSCHEDPRIO 1 | #define HAVE_LIBPTHREAD 1 | /* end confdefs.h. */ | #include <readline/readline.h> configure:31556: result: no configure:31589: checking for readline/readline.h configure:31596: result: no ======== apparently, readline/readline.h is in GNU gcc's default include path but not in Sun Studio cc's default include path on that machine ... Stefan > Log Message: > Attempt to approve no static readline library on SunOS 5.11 (when not Prefix-enabled) > > --- NEW FILE: mclient-lmal--help.stable.out.SunOS5.11.STATIC --- > stdout of test 'mclient-lmal--help` in directory 'src/tools` itself: > > > # 11:03:02 > > # 11:03:02 > Mtimeout -timeout 240 mserver5 "--config=/ufs/mk/monet5/Linux/etc/monetdb5.conf" --debug=10 --set "monet_mod_path=/ufs/mk/monet5//Linux/lib/MonetDB5:/ufs/mk/monet5//Linux/lib/MonetDB5/lib:/ufs/mk/monet5//Linux/lib/MonetDB5/bin" --set "gdk_dbfarm=/ufs/mk/monet5//Linux/var/MonetDB5/dbfarm" --set "sql_logdir=/ufs/mk/monet5//Linux/var/MonetDB5/sql_logs" --set "xquery_logdir=/ufs/mk/monet5//Linux/var/MonetDB5/xquery_logs" --set mapi_open=true --set xrpc_open=true --set mapi_port=36261 --set xrpc_port=45929 --set monet_prompt= --trace "--dbname=mTests_src_tools" --dbinit="" ; echo ; echo Over.. > # 11:03:02 > > > !WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. > !WARNING: BBPdir: initializing BBP. > # MonetDB server v5.1.0, based on kernel v1.19.0 > # Serving database 'mTests_src_tools' > # Compiled for x86_64-redhat-linux-gnu/64bit with 64bit OIDs dynamically linked > # Copyright (c) 1993-2007 CWI, all rights reserved > # Visit http://monetdb.cwi.nl/ for further information > # Listening for connection requests on mapi:monetdb://gio.ins.cwi.nl:36261/ > > Ready. > #function user.main():void; > # clients.quit(); > #end main; > > Over.. > > # 11:03:02 > > # 11:03:02 > Mtimeout -timeout 120 ./mclient-lmal--help.MAL.sh mclient-lmal--help > # 11:03:02 > > > > # 11:03:02 > > # 11:03:02 > Mtimeout -timeout 60 mclient -lmal -umonetdb -Pmonetdb --config=/ufs/mk/monet5/Linux/etc/monetdb5.conf --host=gio --port=36261 --help > # 11:03:02 > > > \? - show this message > ?pat - MAL function help. pat=[modnme[.fcnnme][(][)]] wildcard * > \<file - read input from file > \>file - save response in file, or stdout if no file is given > \|cmd - pipe result to process, or stop when no command is given > \t - toggle timer > \e - echo the query in sql formatting mode > \f - format using a built-in renderer {csv,tab,raw,sql,xml} > \w# - set maximal page width (-1=raw,0=no limit, >0 max char) > \r# - set maximum rows per page (-1=raw) > \L file - save client/server interaction > \X - trace mclient code > \q - terminate session > mal> > # 11:03:02 > > # 11:03:02 > Done. > # 11:03:02 > > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

[Monetdb-developers] Could not create hash table for key test
by Stefan de Konink 23 Nov '08

23 Nov '08

#GDKmalloc(14777222432) fail => BBPtrim(enter) usage[mem=14840896,vm=27527806976] #BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976 #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 #GDKmalloc(14777222432) fail => BBPtrim(ready) usage[mem=11538120,vm=27527806976] #BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key test #GDKmalloc(14777222432) fail => BBPtrim(enter) usage[mem=11537664,vm=27527806976] #BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976 #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 #GDKmalloc(14777222432) fail => BBPtrim(ready) usage[mem=10416208,vm=27527806976] #BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key test MAPI = monetdb@localhost:50000 QUERY = COPY 773410980 RECORDS INTO node_tags from '/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', ''''; ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes !ERROR: GDKload: failed name=12/1252, ext=thash !ERROR: GDKmallocmax: failed for 1892320544 bytes !ERROR: GDKload: failed name=12/1253, ext=thash What kind of voodoo is this? I'm trying to import a 42GB csv file using copy into, on a native system, 2GB of memory, 4GB of swap. Latest CVS version, this time compiled for CoreDuo, without debuging, with optimise. Stefan

3 10

Re: [Monetdb-developers] Monetdb-developers Digest, Vol 30, Issue 10
by Peter Boncz 23 Nov '08

23 Nov '08

Hi Stefan (the non-german .de one) > Maybe a very strange idea, could be your better solution. If by default > these operations are mmap'ed, they should be taken care of by a disk based > memory extension. Hence being independent of RAM/SWAP but having all pro's > of both. You are right. In fact, this is what MonetDB is supposed to do. Memory is organized in contiguous Heap objects. This does not only hold for BATs but also for Hash Tables. At some moment, I did add the possibility to use a real memory mapped files for their allocation, as a last resort. The problem was there that the Heap() API did not have a filename parameter, and a HEAPextend() thus would not know what file to use. In order not to break the API, I decided to initialize the filename fields at BAT creation time (this is why you may sometimes see .hhash and .thash files in a just stopped MonetDB repository -- even though hash tables are not persistent). With hindsight, this API decision was bad judgement; because the question whether real file mapping is used now depends on the question whether the filename was initialized at BAT or Hash table creation. Now, this should be the case, but it would be my first question to address in a debugging session. I also recall that in case of the Skyserver TB dataset, key checking was disabled to to similar problems. Of course, if we would find and fix this bug, life will not be rosy for you, doing random access (key uniqueness checking) into a 14GB heap that will never fit your 2GB RAM. That goes back to your question whether it is in fact reasonable to materialize 14GB. Questions like these led to a redesign that is much more careful with memory consumption (X100) whose spin-off development regrettably means that my time for such debugging sessions currently is limited. Peter PS The likely cause for the problem is a shortage of swap space. Increasing your swap file size (with a few 10s of GB) should be a workaround for the problem.

3 2

[Monetdb-developers] Constraint violations with all details
by Stefan de Konink 23 Nov '08

23 Nov '08

Hi, I cannot find it back in the trackers but I think it have asked this question before. Would it be possible to output the actual failing constraint details instead of 'just an error'. sql>alter table relation_members_way add constraint relation_to_way_fkey foreign key(to_way) references ways; !SQLException:assert:INSERT INTO: FOREIGN KEY constraint 'relation_members_way.relation_to_way_fkey' violated Inserting with INSERTS will give a nice error, because the query it self contains the problem. This is not true for a copy into, or in this respect just an alter table. Stefan ps. indeed I have to think too long about a query that will show me the non existing primary keys.

2 2

Re: [Monetdb-developers] Could not create hash table for key test
by Peter Boncz 23 Nov '08

23 Nov '08

Hi Stefan, Like Martin indicated, MonetDB runs out of memory here, when trying to obtain a contiguous area of 14GB. The error messages show that MonetDB is pulling all the stops to try to get this memory from the system. That is, a so-called "trim" buffer manager thread tries to unload all unpinned other tables. To no avail here. You see the actvity of the trim thread only when a GDKmalloc has failed already, and the system tries again after unloading what it can. The fact that though it tries to GDKmalloc you see the GDKload error message indicates that it actually tries to get the 14GB by creating a file and memory mapping on it. The cause for failure can thus be: (1) your file system is full (test just after the error message appears, because a MonetDB restarts first clears out all temp files it created on the last run) (2) you try to process such large loads on a 32-bts machine (naughty naughty) For these large sizes, and the way MonetDB manages mamory, you need a 64-bits OS. Peter > Message: 4 > Date: Sun, 23 Nov 2008 02:04:09 +0100 > From: Stefan de Konink <stefan(a)konink.de> > Subject: [Monetdb-developers] Could not create hash table for key test > To: monetdb-dev <monetdb-developers(a)lists.sourceforge.net> > Message-ID: <4928AC09.7010205(a)konink.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > #GDKmalloc(14777222432) fail => BBPtrim(enter) > usage[mem=14840896,vm=27527806976] > #BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976 > #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 > #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 > #GDKmalloc(14777222432) fail => BBPtrim(ready) > usage[mem=11538120,vm=27527806976] > #BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key > test > #GDKmalloc(14777222432) fail => BBPtrim(enter) > usage[mem=11537664,vm=27527806976] > #BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976 > #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 > #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 > #GDKmalloc(14777222432) fail => BBPtrim(ready) > usage[mem=10416208,vm=27527806976] > #BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key > test > MAPI = monetdb@localhost:50000 > QUERY = COPY 773410980 RECORDS INTO node_tags from > '/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', ''''; > ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes > !ERROR: GDKload: failed name=12/1252, ext=thash > !ERROR: GDKmallocmax: failed for 1892320544 bytes > !ERROR: GDKload: failed name=12/1253, ext=thash > > What kind of voodoo is this? > > I'm trying to import a 42GB csv file using copy into, on a native > system, 2GB of memory, 4GB of swap. Latest CVS version, this time > compiled for CoreDuo, without debuging, with optimise. > > > Stefan > > > > ------------------------------ > > Message: 5 > Date: Sun, 23 Nov 2008 03:02:18 +0100 > From: Stefan de Konink <stefan(a)konink.de> > Subject: Re: [Monetdb-developers] Could not create hash table for key > test > To: monetdb-dev <monetdb-developers(a)lists.sourceforge.net> > Message-ID: <4928B9AA.5060401(a)konink.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > #BBPTRIM_ENTER: memsize=180224,vmsize=14397997056 > #BBPTRIM: memtarget=0 vmtarget=0 > #BBPTRIM_EXIT: memsize=20365312,vmsize=14397997056 > #BBPTRIM_ENTER: memsize=2496368440,vmsize=16874000184 > #BBPTRIM: memtarget=778092856 vmtarget=0 > #BBPTRIM_EXIT: memsize=20365312,vmsize=10150287928 > Killed > > This one also doen't make me happy. The initial table that are inserted > without a problem are of 19GB and 1GB. > > The table that fails after it is 7GB. > > > Now obviously we are talking about a system that could use some more > memory. But I don't really see why it is inserting 20GB without a > problem, and then borks on the inserting of 7GB. A non string, table. > > > > Stefan > > > > ------------------------------ > > Message: 6 > Date: Sun, 23 Nov 2008 08:32:00 +0100 > From: Martin Kersten <Martin.Kersten(a)cwi.nl> > Subject: Re: [Monetdb-developers] Could not create hash table for key > test > To: Stefan de Konink <stefan(a)konink.de>, monetdb-dev > <monetdb-developers(a)lists.sourceforge.net> > Message-ID: <492906F0.5060209(a)cwi.nl> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Dear Stefan, > Let's explain as far as possible. > > Stefan de Konink wrote: >> #GDKmalloc(14777222432) fail => BBPtrim(enter) >> usage[mem=14840896,vm=27527806976] >> #BBPTRIM_ENTER: memsize=14840896,vmsize=27527806976 >> #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 >> #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 >> > The system is trying to malloc space for (in this case) a hash table, > but that instruction fails. Then it attempts to free up memory by > swapping out tables. > Given the fact that after this sweep it still fails, indicates that > your swap memory has become fragmented, such that there is no > consecutive area of 14 GB left. > > For large table loads (relative to the swap size) it is advisable to > load in multiple steps. > First load the data into the tables without (foreign) key checks. > After this step ALTER the tables to respect the (foreign) keys. > > Better solutions are being worked on. >> #GDKmalloc(14777222432) fail => BBPtrim(ready) >> usage[mem=11538120,vm=27527806976] >> #BATpropcheck: BAT tmp_1260(-688): could not allocate hash table for key >> test >> #GDKmalloc(14777222432) fail => BBPtrim(enter) >> usage[mem=11537664,vm=27527806976] >> #BBPTRIM_ENTER: memsize=11537664,vmsize=27527806976 >> #BBPTRIM: memtarget=4611686018427387904 vmtarget=0 >> #BBPTRIM_EXIT: memsize=19120128,vmsize=27527806976 >> #GDKmalloc(14777222432) fail => BBPtrim(ready) >> usage[mem=10416208,vm=27527806976] >> #BATpropcheck: BAT tmp_1135(-605): could not allocate hash table for key >> test >> MAPI = monetdb@localhost:50000 >> QUERY = COPY 773410980 RECORDS INTO node_tags from >> '/mnt/data2/csv/node_tags.csv' USING DELIMITERS ',', '\n', ''''; >> ERROR = !MALException: GDKmallocmax: failed for 1892320544 bytes >> !ERROR: GDKload: failed name=12/1252, ext=thash >> !ERROR: GDKmallocmax: failed for 1892320544 bytes >> !ERROR: GDKload: failed name=12/1253, ext=thash >> >> What kind of voodoo is this? >> >> I'm trying to import a 42GB csv file using copy into, on a native >> system, 2GB of memory, 4GB of swap. Latest CVS version, this time >> compiled for CoreDuo, without debuging, with optimise. >> >> >> Stefan >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great >> prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Monetdb-developers mailing list >> Monetdb-developers(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-developers >> > > > > > ------------------------------ > > Message: 7 > Date: Sun, 23 Nov 2008 09:32:30 +0100 > From: Martin Kersten <Martin.Kersten(a)cwi.nl> > Subject: Re: [Monetdb-developers] Could not create hash table for key > test > To: Stefan de Konink <stefan(a)konink.de>, monetdb-dev > <monetdb-developers(a)lists.sourceforge.net> > Message-ID: <4929151E.2040909(a)cwi.nl> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Stefan de Konink wrote: >> #BBPTRIM_ENTER: memsize=180224,vmsize=14397997056 >> #BBPTRIM: memtarget=0 vmtarget=0 >> #BBPTRIM_EXIT: memsize=20365312,vmsize=14397997056 >> #BBPTRIM_ENTER: memsize=2496368440,vmsize=16874000184 >> #BBPTRIM: memtarget=778092856 vmtarget=0 >> #BBPTRIM_EXIT: memsize=20365312,vmsize=10150287928 >> Killed >> > ^^^^^^^^ > This means the process is victimized by the Linux kernel. >> This one also doen't make me happy. The initial table that are inserted >> without a problem are of 19GB and 1GB. >> >> The table that fails after it is 7GB. >> >> >> Now obviously we are talking about a system that could use some more >> memory. But I don't really see why it is inserting 20GB without a >> problem, and then borks on the inserting of 7GB. A non string, table. >> >> >> >> Stefan >> >> ------------------------------------------------------------------------- >> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge >> Build the coolest Linux based applications with Moblin SDK & win great >> prizes >> Grand prize is a trip for two to an Open Source event anywhere in the world >> http://moblin-contest.org/redirect.php?banner_id=100&url=/ >> _______________________________________________ >> Monetdb-developers mailing list >> Monetdb-developers(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-developers >> > > > > > ------------------------------ > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > > ------------------------------ > > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > End of Monetdb-developers Digest, Vol 30, Issue 9 > ************************************************* >

2 1

[Monetdb-developers] MAPI protocol suggestion
by Stefan de Konink 22 Nov '08

22 Nov '08

Hi, With light on my recent philosophical question on doubles/uints for the lossy storage on coordinates. I wonder the following; Would it be possible to implement a binary MAPI protocol that can use the native types of the data that is stored within MonetDB? So for example; I create a table: CREATE TABLE helloword (hello varchar(13), world integer); INSERT INTO helloword (hello, world) VALUES ('hoi', 1); And fetch this table: SELECT hello, world FROM helloworld; And receive the MAPI data in the following (too simplistic) way: (len) (string) --> 5 hello 5 world --> 3 hoi 4 0x0 0x0 0x0 0x1 For several reasons the above will not work; the length field is too short (only up to 255) it is also not optimal. I wonder if it can trigger any idea to allow the end user to use native types instead of char[]'s. Stefan

2 4

Re: [Monetdb-developers] [Monetdb-checkins] java Makefile.ag, , 1.11, 1.12 build.properties, , 1.5, 1.6 release.txt, , 1.5, 1.6
by Stefan Manegold 18 Nov '08

18 Nov '08

Fabian, Sjoerd, shouldn't we then also increase the "version" of the java tarball, e.g., ======== $ sed -i -e 's|20080625|20081118|' java/configure.ag java/NT/rules.msc ======== $ cvs diff java/configure.ag java/NT/rules.msc -------- Index: java/configure.ag =================================================================== RCS file: /cvsroot/monetdb/java/configure.ag,v retrieving revision 1.16 diff -u -r1.16 configure.ag --- java/configure.ag 10 Oct 2008 08:41:25 -0000 1.16 +++ java/configure.ag 18 Nov 2008 10:36:12 -0000 @@ -20,7 +20,7 @@ # Process this file with autoconf to produce a configure script. AC_PREREQ(2.59) -AC_INIT([MonetDB Java], [20080625], [monet(a)cwi.nl] [MonetDB-java]) +AC_INIT([MonetDB Java], [20081118], [monet(a)cwi.nl] [MonetDB-java]) AC_CONFIG_AUX_DIR(conf) AM_INIT_AUTOMAKE AM_CONFIG_HEADER([monetdb_java_config.h]) Index: java/NT/rules.msc =================================================================== RCS file: /cvsroot/monetdb/java/NT/rules.msc,v retrieving revision 1.9 diff -u -r1.9 rules.msc --- java/NT/rules.msc 10 Oct 2008 08:41:25 -0000 1.9 +++ java/NT/rules.msc 18 Nov 2008 10:36:12 -0000 @@ -29,6 +29,6 @@ MONETDB_LIBS=/libpath:"$(MONETDB_PREFIX)\lib" pkg = MonetDB -VERSION=20080625 +VERSION=20081118 ODBC_LIBS = odbccp32.lib user32.lib ======== The java tarball has a date as "version number", since it contains JDBC & XRPC and hence can/should not have the JDBC version number. (The xrpc wrapper .jar files do not even have a versioning schema, yet ...) Stefan On Tue, Nov 18, 2008 at 10:27:33AM +0000, Fabian wrote: > Update of /cvsroot/monetdb/java > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv13749 > > Modified Files: > Makefile.ag build.properties release.txt > Log Message: > Bump version due to changed licence. Nothing has changed between 1.8 and 1.9, tagged suffix (Excelsa) with the current release (Nov2008). We should probably make a note of this in the general release notes. > > U Makefile.ag > Index: Makefile.ag > =================================================================== > RCS file: /cvsroot/monetdb/java/Makefile.ag,v > retrieving revision 1.11 > retrieving revision 1.12 > diff -u -d -r1.11 -r1.12 > --- Makefile.ag 10 Oct 2008 08:41:25 -0000 1.11 > +++ Makefile.ag 18 Nov 2008 10:27:31 -0000 1.12 > @@ -30,7 +30,7 @@ > ant_distjdbc = { > COND = HAVE_JAVAJDBC > DIR = datadir/MonetDB/lib > - FILES = monetdb-1.3-mcl.jar monetdb-1.8-jdbc.jar jdbcclient.jar > + FILES = monetdb-1.3-mcl.jar monetdb-1.9-jdbc.jar jdbcclient.jar > } > > ant_distxrpc = { > > U release.txt > Index: release.txt > =================================================================== > RCS file: /cvsroot/monetdb/java/release.txt,v > retrieving revision 1.5 > retrieving revision 1.6 > diff -u -d -r1.5 -r1.6 > --- release.txt 16 Jun 2008 12:46:39 -0000 1.5 > +++ release.txt 18 Nov 2008 10:27:31 -0000 1.6 > @@ -1,8 +1,8 @@ > RELEASE NOTES > -MonetDB JDBC driver version 1.8 (Excelsa/MCL-1.3) > +MonetDB JDBC driver version 1.9 (Excelsa-Nov2008/MCL-1.3) > Fabian Groffen <Fabian.Groffen(a)cwi.nl> > > -Release date: 2008-06-16 > +Release date: 2008-11-18 > > > This JDBC driver is designed for use with MonetDB, a main-memory > @@ -12,6 +12,8 @@ > interface are available. It is believed, however, that this > implementation is rich enough to be suitable for a majority of > application settings. > +Most notably, JDBC4 is not implemented at the moment. This disallows > +Java 1.6 and up to compile the driver from source. > > Below a list of supported features can be found. Please read this list > if you intend to use this driver. If you feel some features are > > U build.properties > Index: build.properties > =================================================================== > RCS file: /cvsroot/monetdb/java/build.properties,v > retrieving revision 1.5 > retrieving revision 1.6 > diff -u -d -r1.5 -r1.6 > --- build.properties 16 Jun 2008 12:46:39 -0000 1.5 > +++ build.properties 18 Nov 2008 10:27:31 -0000 1.6 > @@ -19,9 +19,9 @@ > # major release number > JDBC_MAJOR=1 > # minor release number > -JDBC_MINOR=8 > +JDBC_MINOR=9 > # an additional identifying string > -JDBC_VER_SUFFIX=Excelsa > +JDBC_VER_SUFFIX=Excelsa-Nov2008 > # the default port to connect on, if no port given when using SQL > JDBC_DEF_PORT=50000 > # the default port to connect on, if no port given when using XML:DB > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime/xrpc/admin/Tests db_env.stable.out, XQuery_0-24, 1.1.2.1, 1.1.2.2
by Stefan Manegold 18 Nov '08

18 Nov '08

On Mon, Nov 17, 2008 at 05:53:14PM +0000, Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/runtime/xrpc/admin/Tests > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv17083/admin/Tests > > Modified Files: > Tag: XQuery_0-24 Also this one: checked in in wrong branch! Stefan > db_env.stable.out > Log Message: > new output (with one additional bun) > > > > U db_env.stable.out > Index: db_env.stable.out > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/xrpc/admin/Tests/Attic/db_env.stable.out,v > retrieving revision 1.1.2.1 > retrieving revision 1.1.2.2 > diff -u -d -r1.1.2.1 -r1.1.2.2 > --- db_env.stable.out 17 Nov 2008 12:03:42 -0000 1.1.2.1 > +++ db_env.stable.out 17 Nov 2008 17:53:12 -0000 1.1.2.2 > @@ -125,6 +125,9 @@ > <bun head="xrpc_trusted"> > # http://monetdb.cwi.nl;http://127.0.0.1;http://localhost;/net/jarl/export/sc… > </bun> > + <bun head="xrpc_user"> > +# http://127.0.0.1;127.0.0.1;http://localhost;localhost > + </bun> > </XQueryResult> > > # 12:02:35 > > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0