- developers-list - monetdb.org

[Monetdb-developers] New stable branch created
by Sjoerd Mullender 13 Jan '10

13 Jan '10

I have created new stable branches for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. However, these are only allowed after consultation with me, and only before the first relese from this branch. All packages use the same name for the branch: Feb2010. So in each of your stable checkout directories you can do cvs update -rFeb2010 to update to this new branch. I have created branches for the following packages: MonetDB clients testing MonetDB4 MonetDB5 buildtools pathfinder sql geom java template TestTools. Note that datacell is missing from this list. It is not yet released and will not be released in this round. -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, , 1.10, 1.11
by Lefteris 06 Jan '10

06 Jan '10

That explains a lot:) Thank you! On Wed, Jan 6, 2010 at 10:34 AM, Fabian <mr-meltdown(a)users.sourceforge.net> wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv21323 > > Modified Files: > rdf_shredder.mx > Log Message: > maintainer timeout > > warning: dereferencing type-punned pointer will break strict-aliasing rules > > triple->{predicate,subject,object} are of type void*, first cast to > string pointer, then get the pointer address. > > > Index: rdf_shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v > retrieving revision 1.10 > retrieving revision 1.11 > diff -u -d -r1.10 -r1.11 > --- rdf_shredder.mx 4 Jan 2010 17:06:23 -0000 1.10 > +++ rdf_shredder.mx 6 Jan 2010 09:33:59 -0000 1.11 > @@ -148,7 +148,10 @@ > } > > @= rdf_tknzr_insert > -TKNZRdeposit(&bun,(str *)&(@1)); > +{ > + str t = @1; > + TKNZRdeposit(&bun,&t); > +} > > @= rdf_BUNappend > {@1 = BUNappend(@1, @2, TRUE);} > @@ -166,7 +169,7 @@ > > if (triple->subject_type == RAPTOR_IDENTIFIER_TYPE_RESOURCE > || triple->subject_type == RAPTOR_IDENTIFIER_TYPE_ANONYMOUS) { > - @:rdf_insert(graph[MAP_LEX],triple->subject)@ > + @:rdf_insert(graph[MAP_LEX],(str)triple->subject)@ > @:rdf_BUNappend(graph[S_sort], &bun)@ > bun = BUN_NONE; > } else { > @@ -174,7 +177,7 @@ > } > > if (triple->predicate_type == RAPTOR_IDENTIFIER_TYPE_RESOURCE) { > - @:rdf_insert(graph[MAP_LEX],triple->predicate)@ > + @:rdf_insert(graph[MAP_LEX],(str)triple->predicate)@ > @:rdf_BUNappend(graph[P_sort], &bun)@ > bun = BUN_NONE; > } else { > @@ -183,7 +186,7 @@ > > if (triple->object_type == RAPTOR_IDENTIFIER_TYPE_RESOURCE > || triple->object_type == RAPTOR_IDENTIFIER_TYPE_ANONYMOUS) { > - @:rdf_insert(graph[MAP_LEX],triple->object)@ > + @:rdf_insert(graph[MAP_LEX],(str)triple->object)@ > @:rdf_BUNappend(graph[O_sort], &bun)@ > bun = BUN_NONE; > } else if (triple->object_type == RAPTOR_IDENTIFIER_TYPE_LITERAL) { > > > ------------------------------------------------------------------------------ > This SF.Net email is sponsored by the Verizon Developer Community > Take advantage of Verizon's best-in-class app development support > A streamlined, 14 day to market process makes app distribution fast and easy > Join now and get one step closer to millions of Verizon customers > http://p.sf.net/sfu/verizon-dev2dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk.mx, , 1.294, 1.295 gdk_atoms.mx, , 1.168, 1.169 gdk_bbp.mx, , 1.267, 1.268 gdk_heap.mx, , 1.129, 1.130 gdk_search.mx, , 1.89, 1.90
by Stefan de Konink 05 Jan '10

05 Jan '10

Op 05-01-10 20:54, Sjoerd Mullender schreef: > There is also code in there to store the string length before the hash > value, but this is not enabled in this version. It would increase the > space requirements for probably very little (if any) gain. As in the feature request was suggested, the stringlength will significantly help LIKE operations, if a preselection is made based on this value. (Hence a like operation with % would always be equal or longer than the specified constant length + the single character modification count, if not present: equal) How can I enable the length storage? Stefan

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/plain aggr.mx, , 1.12, 1.13 ascii_io.mx, , 1.34, 1.35 bat.mx, , 1.37, 1.38 mkey.mx, , 1.17, 1.18 radix.mx, , 1.11, 1.12 xtables.mx, , 1.22, 1.23
by Sjoerd Mullender 04 Jan '10

04 Jan '10

I forgot to say, a full rebuild will be necessary (some essential macros deep in gdk were changed). Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/MonetDB4/src/modules/plain > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv28677/MonetDB4/src/modules/plain > > Modified Files: > aggr.mx ascii_io.mx bat.mx mkey.mx radix.mx xtables.mx > Log Message: > Changed the implementation of the offset heap for type string. > > Before, the offset heap would contain values of type var_t (typically > equal in size to oid), so if there are lots of entries but not that > many different strings, a lot of space was wasted, since only the > lower few bytes of the values in the offset heap would be used, the > rest being zero. > > In this version, the offset heap has a dynamic width. It starts off > as a single byte value, and when the offsets become too large to fit > in a single byte, the heap is upgraded to a short, then an int, and > finally (if the system uses 64 bit oids) to lng. > > There are tags to mark this change: > before_variable_string_offset and after_variable_string_offset. > > Only MonetDB* and pathfinder needed to be changes, so the tags are > only added there. -- Sjoerd Mullender

1 0

[Monetdb-developers] Open source database survey - response requested from a MonetDB developer!
by Selena Deckelmann 01 Jan '10

01 Jan '10

Hi! My name is Selena Deckelmann, and I'm giving a talk at LinuxConf AU 2010 called "A Survey of Open Source Databases". The goal of the talk is to give developers and software architects an overview of the open source tools that are out there, and a sense of which types of problems your software is intended to solve. I'm contacting over 30 projects. If there is a better person or mailing list to direct my questions to, please point the way! I've done some basic research, but it would very much help me to have answers to questions I've included below in your own words. Also, if a representative from your project has time for a short phone call, I'd be happy to get answers to these questions by voice (Skype/Jabber/regular old phone all works for me). Anyway - the survey is here: http://bit.ly/8yarGr Even just responding to the first few questions would be helpful, if you don't have time for them all. About me: I primarily volunteer with the PostgreSQL project for creation of user groups, advocacy and support our project's systems infrastructure, and I contribute to a replication system called Bucardo. I'm also President of Technocation, a 501(c)3 non-profit in the US that helps organize OpenSQLCamp, WhereCampPDX and other un-conference events, and distributes free video for open source technical events we're able to record. -- http://chesnok.com/daily - me http://endpoint.com - work

1 0

[Monetdb-developers] memory usage question
by Mark Bucciarelli 23 Dec '09

23 Dec '09

I'm seeing some behavior that makes me wonder if mserver5 leaks. I'm using Ubuntu 9.04 w/ 32-bit i386 and 500 MB of RAM. This is 2009Nov CVS. Does this sequence of steps and memory usage look right? 1. Reboot the server and start merovingian 0m -- mserver5 RSS (not running) 417m -- free memory reported by top 78m -- used 5m -- buffers 2. Destroy then create database 15m -- mserver5 RSS 392m -- free memory reported by top 3. Add tables to database 21m -- mserver5 RSS 378m -- free memory reported by top 4. Import a small table (38,000 rows, autoincr int + varchar(1024)) 29m -- mserver5 RSS 350m -- free memory reported by top 5. Repeat steps 2 through 4 three times. 29m -- mserver5 RSS !!! 225m -- free memory reported by top 6. Shut down merovingian !!! 251m -- free memory reported by top 242m -- used 136m -- buffers Why is top only showing 251m of free memory? Yesterday, the system started to swap when free memory, as reported by top, was exhausted. And I bet I could get it to swap if I ran lots more iterations in step 5 above. I know that having free close to zero is a good thing, as it means you are using your RAM, but that swapping is never good on a database server. :) I'm not using the server for anything else at the moment, other than SSH and transferring data from SQLite to MonetDB via mclient and files of SQL commands. As a related question, what is the minimum RAM need to run MonetDB? I'd like to get it running on an old slow computer running OpenBSD to check how well Monet handles memory pressure and to turn on OpenBSD's paranoid malloc options. Are there some knobs I can turn to make monetdb5 run when there is only 100MB of free memory after the OS starts up? That swapping yesterday got me a nervous about stability, especially since stopping and restarting merovingian didn't give the memory back. Thanks, m

1 0

[Monetdb-developers] Stable tree?
by Mark Bucciarelli 21 Dec '09

21 Dec '09

Hi, I've been using CVS HEAD to test against, but now I'm wondering if I should switch the Nov2009 branch. I'm going to use Monet DB in production for an in-house project, so I would prefer a reasonable level of stability. Which source tree should I track? Thanks! m

3 2

[Monetdb-developers] hashjoin and strHash
by Wouter Alink 19 Dec '09

19 Dec '09

Dear developers, I would like to propose a change in GDK and hear opinions. It is about the following issue: in the BATjoin code, if there is no possibility to do a fetch or merge join, a hashjoin is performed. A hashtable is created for the smallest BAT. The reasons (i could think of) for choosing the smallest BAT for the hashtable are that less space is required for the hashtable (which in turn causes less cache misses when doing a lookup) and also because the hashfunction used is assumed to be very inexpensive (it needs to be calculated for each item in the large bat each time a join is performed). I can see that the hashfunction can be very efficient for data types without indirection, but I feel that for data types like strings in some cases this is a little different. If a string BAT for example contains many different values (i.e. is not a bat which contains enumeration values) the hashfunction will not be inexpensive anymore (many cache misses), as each hashfunction call needs to hash a whole (arbitrary long) string at an arbitrary place in the heap. Is it perhaps possible to specify that, when a BAT of type 'str' has many different values a hashtable may be build on the large BAT instead of on the small BAT? Reason that I ask this: I was analysing costs of a query in which I had a few short strings (26 tuples, 1-column table: varchar) which I wanted to look up in a dictionary (9M tuples, 2-column table: int,varchar). "SELECT a.id FROM longlist AS a JOIN smalllist as b ON a.strvalue=b.strvalue;" The result is a small list of integers (26 or less tuples). This operation currently takes roughly 1.5 seconds for a hot run, mostly due to 9M strHash operations. By applying the patch below the execution time for a hot run dropped down to .01 seconds. The performance gain is caused by only having to perform strHash on the items in the small bat once the hashtable for the large bat has been created. Any suggestions whether such a change is useful? Which benchmarks will be influenced? I guess this code change is probably not useful for large string BATs with only few different values, but perhaps a guess could be made how diverse the strings in a bat are (by taking a sample or perhaps simply by looking at the ratio batsize/heapsize), and based on that determine whether to build it on the large or small BAT? Greetings, Wouter Index: src/gdk/gdk_relop.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_relop.mx,v retrieving revision 1.167.2.4 diff -u -r1.167.2.4 gdk_relop.mx --- src/gdk/gdk_relop.mx 20 Nov 2009 13:04:06 -0000 1.167.2.4 +++ src/gdk/gdk_relop.mx 18 Dec 2009 14:59:13 -0000 @@ -1232,7 +1232,12 @@ @- hash join: the bread&butter join of monet @c - /* Simple rule, always build hash on the smallest */ + /* Simple rule, always build hash on the smallest, + except when it is a string-join, then we do the opposite */ + if (swap && rcount < lcount && l->ttype == TYPE_str) { + ALGODEBUG THRprintf(GDKout, "#BATjoin: BATmirror(BAThashjoin(BATmirror(r), BATmirror(l)," BUNFMT "));\n", estimate); + return BATmirror(BAThashjoin(BATmirror(r), BATmirror(l), estimate)); + } if (swap && rcount > lcount) { ALGODEBUG THRprintf(GDKout, "#BATjoin: BATmirror(BAThashjoin(BATmirror(r), BATmirror(l)," BUNFMT "));\n", estimate);

5 8

[Monetdb-developers] Memory use
by Voornaam Achternaam 19 Dec '09

19 Dec '09

When I try to fill a database with the COPY INTO command, the data will (depending on the file used) either go: - Fully into Memory. - To HDD. - A combination of Memory and HDD space. Is there a way to configure MonetDB so that it always uses Memory only? Thanks in advance. -- View this message in context: http://old.nabble.com/Memory-use-tp26843876p26843876.html Sent from the monetdb-developers mailing list archive at Nabble.com.

2 1

[Monetdb-developers] Announcement: New Nov2009-SP1 bugfix release of MonetDB suite
by Sjoerd Mullender 18 Dec '09

18 Dec '09

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2009-SP1 bug fix release of the MonetDB suite of programs. More information on this release will be available at <http://monetdb.cwi.nl/Development/Releases/Nov2009/>. Fixes include: - The Windows installers for MonetDB/SQL in the Nov2009 release were incomplete. The missing files have been added. - Fixed performance issue with loading complex schemas in readonly mode in MonetDB/SQL. - Fixed performance issue with grouping in MonetDB/SQL when one of the columns is sorted. - Fixed problems with LIKE joins in MonetDB/SQL. - Load all files in a multi-file COPY INTO query. - Fix error reporting of unexpected input in XQuery queries to be Unicode aware. - Correctly handle circular dependencies in FD property check. (SF bug #2908615.) - Fixes throughout the code base to better cope with unexpected situations (such as memory shortage). -- Sjoerd Mullender

1 0