- developers-list - monetdb.org

[Monetdb-developers] thoughts on XQJ
by p.a.boncz 15 Aug '06

15 Aug '06

Hi Fabian, You report the following JDBC xquery error on an example from the XQJ spec: > monetdb-> import schema namespace foo='http://www.foo.com' > monetdb=> at 'http://www.fooschema.com'; > monetdb=> for $i in (foo:hatsize(1),foo:hatsize(4)) return $i > Error: protocol violation, unexpected line: xquery_prepare: missing ';' > after module import. > monetdb-> this error message is indeed wrong: for some reason, the query cache mechanism incorrectly classifies this query as importing a module. Instead, you should have received the following error: error in function application: at (3,12-3,25): reference to undefined function `foo:hatsize' # halted in /data/home/boncz/pathfinder/compiler/semantics/functions.c (check_fun_usage), line 291 I think this example is wrong, apparently the XQuery system the XQJ developers are using (DB2?) allows to import modules by just importing a schema, or allows schemas to define methods, because 'hatsize()' is not a built-in function, so it should be either defined in the XQuery itself or in a module. It seems most likely that 'schema' should be replaced by 'module'. The other question is how to handle prepared queries. It will be some work to support this, but it can be done. MonetDB/XQuery has a query cache that can accelerate queries if they consist of a single function call. That function must be defined in a module. Suppose an XQJ client has the query: "declare variable $x as xs:integer external; for $n in fn:doc('catalog.xml')//item where $n/price <= $x return fn:data($n/name)" At the prepare phase (either by a call to the PrepareQuery method, or implicitly in the ExecuteQuery method), this should be translated to: "module namespace tmpXXX_1 = "http://monetdb.cwi.nl/XQuery"; function tmpXXX_1:preparedQuery($x as xs:integer) as xs:anyNode* for $n in fn:doc('catalog.xml')//item where $n/price <= $x return fn:data($n/name)" Here XXX is some session-ID, that should be known to the client. Notice that the external variable declarations where "eaten away", and the query was wrapped in a module function, which as signature has the externally declared variables. The default return type is a sequence of items (xs:AnyNode*). This query can be executed immediately. At server-side, we should have a local directory, e.g. 'preparedXQuery/' available, and executing the module query will just put it in the server-side file 'preparedXQuery/tmpXXX_1.xq'. Now suppose variable $x is bound to 42 and the query is executed. XQJ sould generate the following query: "import module namespace tmpXXX_1='file://preparedXQuery/tmpXXX_1.xq'; tmpXXX_1:preparedQuery(42)" Note that in order to get the benefits of prepared queries, we do not need a prepare-bind-execute mechanism. You just need to define your query as a function in a module. The benefits can be very substantial, a factor 10 is often observed on small queries. But as you see, we can support prepare-bind-query with it as well. The question is whether we should, because the scenario is still suboptimal in case there are many sessions that use this same prepared query. All these sessions will pay prepare costs once, instead of just the first. In that sense, the module-approach is actually better. So maybe we should just take a simplictic approach to variable substitutions by generating the stand-alone query: "let $x := 42 return for $n in fn:doc('catalog.xml')//item where $n/price <= $x return fn:data($n/name)" I advise against trying to substitute $x in place, because lexical analysis of xqueries (which may contain XML and comments) is difficult, so you do not want to replicate all that in the XQJ layer at client-side. Thus, in this approach, all external variable declarations are tranformed into let-statements that define and binds the variables. Between the let's and the query, a 'return' keyword should be inserted. A final question is what should be done in sessions with multiple queries. Supporting this would not be difficult if each query is executed in a single transaction (autocommit on). Autocommit off is more work, because this means we have to preserve the "working-set" (database state) between queries, and postpone the commit until the last one in the session. Multi-query scenarios do raise the question what to do when a query returns XML nodes. If these values are fed as parameters into a next query, should we copy by reference, or by value? Currently we serialize nodes, so you can only get by value (i.e. nodes will take serialized form, which when passed in comes down to sequence construction). Passing by reference adds the implicit semantic that documents opened with fn:doc() stay "open" across queries in a single transaction. Like autocommit off, this also requires the working set to be preserved. Peter

1 0

[Monetdb-developers] XQ n00b XQuery question
by Fabian Groffen 14 Aug '06

14 Aug '06

While reading the JXQ draft specs, I stumbled upon the following query: (pegasus:sql/src/jdbc) fabian% JdbcClient -lxquery Welcome to the MonetDB interactive JDBC terminal! Type \q to quit, \h for a list of available commands auto commit mode: on monetdb-> import schema namespace foo='http://www.foo.com' monetdb=> at 'http://www.fooschema.com'; monetdb=> for $i in (foo:hatsize(1),foo:hatsize(4)) return $i monetdb=> Error: protocol violation, unexpected line: xquery_prepare: missing ';' after module import. monetdb-> Apart from the protocol violation (should be fixed, missing !) I was wondering whether the "at 'http://www.fooschema.com';" part is not supported by the compiler, or whether it may not reside on a new line. monetdb-> import schema namespace foo='http://www.foo.com' at 'http://www.fooschema.com'; monetdb=> for $i in (foo:hatsize(1),foo:hatsize(4)) return $i monetdb=> Error: protocol violation, unexpected line: xquery_prepare: missing ';' after module import. monetdb-> It seems not to matter. Not being an XQuery expert at all, I was wondering if someone could enlighten me on the validity of this statement. (PS: not a simple example from the JXQ draft works in MonetDB/XQuery, because we either don't support external variables, or their syntax is wrong... The notion of "prepared" xquery is quite central in JXQ, I was wondering whether this is available in MonetDB/XQuery, already or not.)

1 0

[Monetdb-developers] [PATCH] Mx Io.c more specific errors
by Fabian Groffen 09 Aug '06

09 Aug '06

Hi devs, I ran into a problem with Mx, where IoWriteToFile seemed to have a problem but didn't tell exactly what. With the attached patch, I was able to figure out what the problem was. I don't like to commit it, as the Mx code is completely unknown to me. Also, I have no ideas on the effects on this highly portable piece of code on Windows. If someone considers it useful to commit, please do.

3 5

[Monetdb-developers] Unable to shred 1gb xml file (OS: Not enough space)
by kla gw 08 Aug '06

08 Aug '06

Hi, I tried MonetDB/XQuery to shred a 1gb xml file, but it failed. Following is the error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856] I use windows XP Professional version 2002 service pack 2, Pentium 4 CPU, 2.40GHz, 512 MB of RAM. Previously I tried to shred 100mb xml file, and it took 18.532 sec. For this 1 gb file, I left it overnight so I don't know how long it takes till the error message occurs. Can please help me to solve this problem? Regards, Klarinda Below is the complete error message: MonetDB>shred_doc("D:xbench/output/DC1000catalog.xml", "DC1000catalog.xml"); !ERROR: MT_mmap: MapViewOfFile(6b4, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !GDKmmap(442499072) fail => BBPtrim(enter) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 #BBPTRIM_ENTER: memsize=101297568,vmsize=794361856 #BBPTRIM: memtarget=0 vmtarget=1073741824 #TRIMSCAN: mem=0 vm=1, start=1, limit=1 #TRIMSCAN: 145030 0=tmp_35 (#0) #TRIMSCAN: 145059 1=tmp_36 (#0) #TRIMSCAN: 145088 2=tmp_37 (#0) #TRIMSCAN: 145146 3=tmp_41 (#0) #TRIMSCAN: 149075 4=doc_query (#0) #TRIMSCAN: 149092 5=doc_sema (#0) #TRIMSCAN: 155215 6=tmp_374 (#0) #TRIMSCAN: 155218 7=prop_pre_39 (#0) #TRIMSCAN: 157895 8=tmp_533 (#0) #TRIMSCAN: 157898 9=prop_pre_310 (#0) #TRIMSCAN: end at 1 (size=628) #TRIMSELECT: dirty = 0 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: keep tmp_35 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: keep tmp_36 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: keep tmp_37 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: keep tmp_41 [224,0] bytes [224,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: keep doc_query [224,0] bytes [224,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: keep doc_sema [224,0] bytes [224,0] dirty target(mem=0 vm=107374182 4) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: keep tmp_374 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: keep prop_pre_39 [224,0] bytes [0,0] dirty target(mem=0 vm=10737418 24) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: keep tmp_533 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741824) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: keep prop_pre_310 [224,0] bytes [0,0] dirty target(mem=0 vm=1073741 824) #TRIMSELECT: end #TRIMSELECT: dirty = 1 #TRIMSELECT: candidate=tmp_35 BAT*=03D6A230 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145030,145030 ,145030) #TRIMSELECT: delete tmp_35 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_36 BAT*=03D66E60 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145059,145059 ,145059) #TRIMSELECT: delete tmp_36 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_37 BAT*=058080B0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145088,145088 ,145088) #TRIMSELECT: delete tmp_37 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_41 BAT*=058066F0 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=145146,145146 ,145146) #TRIMSELECT: delete tmp_41 from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_query BAT*=057F9D10 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149075,149075 ,149075) #TRIMSELECT: delete doc_query from trimlist (does not match trim needs) #TRIMSELECT: candidate=doc_sema BAT*=05845370 # (cnt=0, mode=1024, refs=0, wait=0, parent=0, lastused=149092,149092 ,149092) #TRIMSELECT: delete doc_sema from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_374 BAT*=03D722F0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155215,155215 ,155215) #TRIMSELECT: delete tmp_374 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_39 BAT*=05845CD0 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=155218,155218 ,155218) #TRIMSELECT: delete prop_pre_39 from trimlist (does not match trim needs) #TRIMSELECT: candidate=tmp_533 BAT*=0583C598 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157895,157895 ,157895) #TRIMSELECT: delete tmp_533 from trimlist (does not match trim needs) #TRIMSELECT: candidate=prop_pre_310 BAT*=057F7E70 # (cnt=0, mode=4096, refs=0, wait=0, parent=0, lastused=157898,157898 ,157898) #TRIMSELECT: delete prop_pre_310 from trimlist (does not match trim needs) #TRIMSELECT: end #BBPTRIM: no more unload candidates! #BBPTRIM_EXIT: memsize=95140356,vmsize=794361856 !GDKmmap(442499072) fail => BBPtrim(ready) usage[mem=101297568,vm=794361856] # !mallinfo.arena = 15613828 !mallinfo.ordblks = 46134 !mallinfo.smblks = 15492 !mallinfo.hblkhd = 0 !mallinfo.hblks = 0 !mallinfo.usmblks = 13718408 !mallinfo.fsmblks = 899720 !mallinfo.uordblks = 950740 !mallinfo.fordblks = 44960 !ERROR: MT_mmap: MapViewOfFile(6b0, 2, 0, 0, 442499072, 0) failed !OS: Not enough space !ERROR: GDKload: cannot mmap(): name=05\552, ext=theap.priv !OS: Not enough space !ERROR: GDKload failed: name=05\552, ext=theap.priv !ERROR: shredder.mx:append_str2bat: APPEND-STR[PROP_TEXT](final foxes since the silent, quick realms should breach never sheaves--ruthless, daring waters beneat h the close asymptotes c), BUNappend fails !ERROR: CMDshred2bats: operation failed. MonetDB>

2 3

[Monetdb-developers] M5 inspect05 test
by Fabian Groffen 02 Aug '06

02 Aug '06

Devs, I added sabaoth today, but could not approve inspect05 with manual patching, because there seems to have changed more than just some sabaoth signatures added. This is seems to be so for some time. Whoever knows if those changes are correct (something with sortT) and commits inspect05: the sabaoth additions are correct AFAIT.

1 0

[Monetdb-developers] Partitioning
by moredata＠fastmail.net 31 Jul '06

31 Jul '06

Hello, I've been following the developments of MonetDB off and on. It seems to me that there is a real emphasis on developing XML-related functionality, and less so for BI-related functionality. I wanted to ask if there are any plans to allow tables to be partitioned by some condition. This would have a couple of benefits. A table could be broken out into several BAT files, one for each condition met. That would allow you to potentially overcome the 2 GB BAT file limitation on 32 bit systems. And of course, when adding intelligence into the optimizer for this, it could examine the conditions of queries and potentially eliminate entire BAT files to scan. For example, assume you have a table with a column named sometype, and chose sometype for partitioning. Sometype has 10 possible values. Physically, MonetDB would create 10 different BAT files for each column, corresponding to each possible value. When a query is executed with a condition like "sometype = 1", the optimizer is smart enough to know it need only use 1 set of the bat files, and not all 10, significantly reducing the amount of data that needs to be examined. If you are familiar with PostgreSQL, they introduced something like this called Constraint Exclusion Partitioning (also, MySQL has some partitioning functionality in its new beta). Each partition is treated as a subtable of a main parent table. You are required to insert directly into the proper subtable, but querying on the main parent table will determine which subtables are required to be examined in processing the query. Their scheme is not entirely convenient with loading up data, but it is quite flexible in setting up arbitrary conditions on the subtables. Are there any such plans to do something similar? At what state is this in? I believe I saw something about Partitioning on the roadmap several months ago. The MonetDB home page mentions OLAP, and it would seem to me that a feature like this is critical if MonetDB really wants to handle large data volume Business Intelligence queries. Thanks. -- moredata(a)fastmail.net -- http://www.fastmail.fm - Choose from over 50 domains or use your own

1 0

[Monetdb-developers] CVS and Monet5
by Luis Zarrabeitia 29 Jul '06

29 Jul '06

Hi again. The checkout instructions on the website cvs -d:pserver:anonymous@monetdb.cvs.sourceforge.net:/cvsroot/monetdb checkout buildtools MonetDB sql checks out Monet4.1x rather than Monet5 / Monet4.99. How do I checkout the later? I made some changes to 4.1x that I'd like to try on 4.99 before submitting them (they are so short that I feel I made a MAJOR mistake somewhere, so I'll be testing them at least one more day). Btw, should I submit the .diffs to this list? Zarrabeitia.

2 1

[Monetdb-developers] Partitioning
by moredata＠fastmail.net 28 Jul '06

28 Jul '06

Hello, I've been following the developments of MonetDB off and on. It seems to me that there is a real emphasis on XML-related functionality, and less so for BI-related functionality. I wanted to ask if there are any plans to allow tables to be partitioned by some condition. This would have a couple of benefits. A table could be broken out into several BAT files, one for each condition met. That would allow you to potentially overcome the 2 GB BAT file limitation on 32 bit systems. And of course, when adding intelligence into the optimizer for this, it could examine the conditions of queries and potentially eliminate entire BAT files to scan. For example, assume you table tab with a column named sometype, and used sometype for partitioning. Sometype has 10 possible values. Physically, MonetDB would create 10 different BAT files for each column, corresponding to each possible value. When a query is executed with a condition like "sometype = 1", the optimizer is smart enough to know it need only use 1 set of the bat files, and not all 10, significantly reducing the amount of data that needs to be examined. If you are familiar with PostgreSQL, they introduced something like this called Constraint Exclusion Partitioning. Each partition is treated as a subtable of a main parent table. You are required to insert directly into the proper subtable, but querying on the main parent table will determine which subtables are required to be examined in processing the query. Their scheme is not entirely convenient with loading up data, but it is quite flexible in setting up arbitrary conditions on the subtables. Are there any such plans to do something similar? At what state is this in? I believe I saw something about Partitioning on the roadmap several months ago. The MonetDB home page mentions OLAP, and it would seem to me that a feature like this is critical if MonetDB really wants to handle Business Intelligence type of queries. Thanks! -- moredata(a)fastmail.net -- http://www.fastmail.fm - Faster than the air-speed velocity of an unladen european swallow

2 1

Re: [Monetdb-developers] join performance issue!
by Stefan Manegold 25 Jul '06

25 Jul '06

On Tue, Jul 25, 2006 at 10:02:16PM +0200, Arjen P. de Vries wrote: > Wasn't the property checking always determined by debug flag 8? in general: yes. for join: no, at least not any more since the following checkin: =================================================================== 2003/08/06 - sjoerd: src/gdk/gdk_relop.mx,1.38 DDI Merge. This is the big one! Before tag: BeforeDDImerge, after tag: AfterDDImerge. The DDI merge is now made the official main branch; the CWI_DDI_merge branch is not to be used anymore. =================================================================== Well, Sjoerd just did the checking; the origin is most probabaly Peter. As I said, there are indeed good reasons to do the BATpropcheck (even with debug=0) for *arbitrary* joins, where we can hardly predict any result properties --- maybe, we should consider to not do it in those cases (especially fetchjoin), where we can predict the join result properties pretty well... Stefan > On 25/07/06, Martin Kersten <Martin.Kersten(a)cwi.nl> wrote: > > > > > > > >Stefan Manegold wrote: > >> Dear all, > >> > >> > >> - We should consider making this BATpropcheck in BATjoin and BATleftjoin > >> configurable from the outside, e.g., by adding an extra optional > >argument > >> to join() to request to skipping the BATpropcheck. > >> > >Instead of pollution of the calling interface, I think we should inject > >an explicit request to check/enforce properties. This one might also be > >focussed on those properties of interest according to the optimizer at > >that point. > > > >In M5: > > b:= algebra.join(BAT,BAT); > > bat.setProperties(b); > > > > ... > > bat.checkDenseProperty(b); > > bat.checkOrderProperty(b); > > bat.checkKeyProperty(b); > > > >But first, assess the use of the join(bat,bat) in our controlled > >code base. Please report on this. > > > >regards, Martin > > > > > > > -- > ==================================================================== > CWI, room C1.16 Centre for Mathematics and Computer Science > Kruislaan 413 Email: Arjen.de.Vries(a)cwi.nl > 1098 SJ Amsterdam tel: +31-(0)20-5924306 > The Netherlands fax: +31-(0)20-5924312 > ===================== http://www.cwi.nl/~arjen/ ==================== -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] join performance issue!
by Stefan Manegold 25 Jul '06

25 Jul '06

Dear all, to prevent you from falling into the same trap as I did this afternoon when trying to analyze, measure & compare positional (fetch-)join performance, here's a warning and my detailed story: WARNING: When running 'join(BAT,BAT);' in MIL or 'algebra.join(BAT,BAT);' in MAL, be aware that this includes not only the actual join algorithm, but also a BATpropcheck() call to check and set properties of the join result (ALWAYS, i.e., also with the default --debug=0 !) NOTE: The latter can easily be up to 85% (*eighty-five* percent!) of the time you measure with a simple wall-clock around the 'join(BAT,BAT);' or 'algebra.join(BAT,BAT);' (see details below) ! Detailed Story: Preparing to invent and implement another cache-aware/-friendly positional (fetch-)join algorithm for some special cases, I was trying to assess the performance of the current positional (fetch-)join algorithms for the following three cases: 1) join(ld,r); 2) join(lo,r); 3) join(lu,r); In all three cases, r is a [V(o)ID, INT] BAT, i.e., has a dense non-materialized head and a randomly distributed integer tail. ld, lo, lu are all [INT, OID] BATs with a randomly distributed integer tail, and an OID head that is in ld: dense (and marked so in the BAT header) (but materialized) in lo: ordered (actually also dense, but only marked ordered, not marked dense in the BAT header) in lu: unordered (random shuffle of an originally dense sequense) Thus, the three case trigger the use following join implementations, respectively: 1) densefetchjoin 2) orderedfetchjoin 3) defaultfetchjoin I used the same size (count) four BATs, and the join is a perfect 1:1 hit; hence the result is as big as the inputs. Here are the wall-clock times (in micro seconds) for BATs of 10000000 tuples MonetDB 4.13.1 (CVS HEAD) default compilation (gcc -g -O2) 64-bits and 64-bit OIDs 2 GHz AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ 2 GB RAM 1) join(ld,r); 2065162 us 2) join(lo,r); 2090277 us 3) join(lu,r); 3058936 us A close look under the hood (i.e., in BATjoin in src/gdk/gdk_relop.mx) reveals the following split timings for the actual join (batjoin()) and the property checking (BATpropcheck()): batjoin() BATpropcheck() join() 1) join(ld,r); 275331 us 1774275 us 2065162 us (abs) 13.33 % 85.91 % 100.00 % (rel) 2) join(lo,r); 303311 us 1770722 us 2090277 us (abs) 14.51 % 84.71 % 100.00 % (rel) 3) join(lu,r); 1268057 us 1775788 us 3058936 us (abs) 41.45 % 58.05 % 100.00 % (rel) Well, there are some lessons to learn: - Be aware of the BATpropcheck when you measure join performance! - The "mandatory" BATpropcheck is only used for BATjoin and BATleftjoin in src/gdk/gdk_relop.mx --- at least as far as I know. - The "mandatory" BATpropcheck in BATjoin and BATleftjoin is there for a good reason: With joins, it's hard to guess the properties of the result (are head and/or tail sorted, dense, and or key/unique?) in general, but the performance of MonetDB highly depends on the availability of these properties; hence, while making the join (considerably) more expensive, the investment (might) pay-off with later operation on the join result. - We should consider making this BATpropcheck in BATjoin and BATleftjoin configurable from the outside, e.g., by adding an extra optional argument to join() to request to skipping the BATpropcheck. I hope you find this information useful... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1