- developers-list - monetdb.org

[Monetdb-developers] next release: mps or algebra?
by Peter Boncz 17 Oct '07

17 Oct '07

Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose. Maurice' original questions were: (a) whether the new release would be algebra-only (short answer: hopefully). (b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes) (c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no) On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote: > Hi Maurice, et al, > > The release of (hopefully) this week will still be mps based. I also > anticipate at least one bug-fix release (ie 0.20.2), or even more. However, > the next major release (0.22) should be algebra based, in my opinion. I am > certainly reluctant to keep investing any more effort in mps. I would aim > for an mps-free release, yes. > > The time-goal of that release is "XMAS 2007", as far as we are concerned. > For the work of built-in functions I have good hope that with the help of > Lefteris we can come a long way. > > I hope and think updates can also be ported by joerd in that time-frame. As > for pftijah, if Jan Flokstra has time, it should be possible to port that as > well in 2-3 months. > > There are some hairy issues, one of them being free-form recursion (needed > by you), another one being precompiled modules (needed by XRPC). The > resolution of these needs assistance from Munich, and maybe related. It needs > a method to compile certain functions into scripted procedures (here MIL) > and calling them. One algorithm is to do this for all functions in modules, > as well as for all recursive functions. The issue to resolve is the > representation of the parameters. > > Depending on how much time Jan Rittinger has available for this, we will > have to find creative solutions for this. Indeed, a backup solution would be > to keep handling certain queries for the time being with mps, but that > surely is ugly. > > Peter > > -----Original Message----- > From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] > Sent: dinsdag 16 oktober 2007 16:38 > To: Peter Boncz [CWI]; Stefan Manegold [CWI] > Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) > Subject: Nieuwe releases en mps-backend > > > Hoi, > > Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham > vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter > sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet > allemaal kon beantwoorden: > > a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal > niet meer bevat of alleen maar de algebra-backend als default heeft? > b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een > package van alle modules van MonetDB/XQuery? Maw zegt het niets over > eventuele `subreleases' van individuele modules? > > c.. Op de wiki hebben we een pagina met open issues voor de > algebra-release. Wat is de status van die open issues? Ik zie namelijk op de > pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. > Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige > discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet > het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. > Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de > zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden > wegvallen voor non-insiders. Of andere onwenselijke consequenties van het > niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend. > > Groetjes, > Maurice. > > -- > ---------------------------------------------------------------------- > Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology > Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands > Email: m.vankeulen(a)utwente.nl, Phone: +31 534893688, Fax: +31 534892927 > Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5 8

[Monetdb-developers] FW: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Peter Boncz 17 Oct '07

17 Oct '07

Thanks Stefan, > I'll leave the further interpretation of the above results to > the interested recipient / reader. I will give it a try, then. > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s These h/SU results are strange, because a 5 x faster time due to a hash function with more collisions is.. puzzling. In any case, with 64-bits oids on a 8GB machine, the shredding generates very intense swapping, and Linux performance under swapping is known to be highly variable. Difficult to draw conclusions. I am still investigating the behavior of MT_mmap under load and it may well be that shredding performance can be improved by better memory management (less swapping). From the result below we know that it can be done in 25m. The not-yet checked in improvements in index construction will strongly reduce the QU[1] time by taking less memory; in the order of the 100m seen below. > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s These numbers show that MonetDB prefers to operate on RAM resident data. I am glad you reproduced my 25 minutes h/SU result. Must be a coicidence that the mini hash self-join benchmark on 50M string numbers ran in 26s with the new hash and 43s with the old hash (we see the same numbers in minutes here repeated for shredding). In my current version, QU[2] with the ordered indices is back at 17s. Reason for 1m15s is that you tend to loose the hash-table on the index bats due to swapping. So that includes hash table creation. Whether QU[1] is acceptable is highly questionable. Starting the database takes 1.5 hours!! I am seriously considering to switch to persistent ordered indices for the updatable case. For that to happen, the worksing set needs to be extended with persistent delta-bats on those indices, changes to these deltas need to be logged (in order to make the indices recoverable). > -----Original Message----- > From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] > Sent: Wednesday, October 17, 2007 1:54 PM > To: monetdb-developers(a)lists.sourceforge.net > Cc: Peter Boncz; Peter Boncz > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, > 1.143, 1.143.2.1 > > > Just for the records: > > I finally managed to finsh my experiments regarding > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > and the related code changes. For those interested, here's > the detailed > story: > > > "S08-64" System (beo-24): > - 2x 64-bit Dual-Core Opteron270 @ 2 Ghz > - 8 GB memory > - MonetDB/XQuery 0.20, 64-bit, 64-bit OIDs, --enable-optimize > (gcc 4.1.2) > > "S16-32" System (core-1): > - 4x 64-bit Dual-Core Opteron870 @ 2 Ghz > - 16 GB memory > - MonetDB/XQuery 0.20, 64-bit, 32-bit OIDs, --enable-optimize > (gcc 4.1.2) > > Document: > http://mirror.openstreetmap.nl/planet/planet-071003.osm.bz2 > (extracted: 19 GB XML file) > > "SR" Shredding read-only: > pf:add-doc(".../planet-071003.osm","planet-071003.osm") > > "SU" Shredding updateable: > pf:add-doc(".../planet-071003.osm","planet-071003.osm","planet > -071003.osm",5) > > "QR"/"QU" Count query: > count(doc("planet-071003.osm")//*) > > Configurations: > m: without Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143 of gdk_posix.mx) > M: with Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143.2.1 of gdk_posix.mx) > h: without Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134 of gdk_atoms.mx) > H: with Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134.6.1 of gdk_atoms.mx) > > > Results (wall-clock times): > > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s > > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s > > (NB: "SR" includes building of indices, while "SU" does not; > consequently, "QR" can exploit the indices built during > "SR", while "QU[1]" > has to build the indices first, and only "QU[2]" can exploit them.) > > > Apparently, the mmap fix in gdk_posix.mx seems to be > sufficient to prevent > the remap-ERROR reported (for a system & configuration > similar to "S08-64") > in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > > I'll leave the further interpretation of the above results to > the interested > recipient / reader. > > > Stefan > >

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Stefan Manegold 17 Oct '07

17 Oct '07

Just for the records: I finally managed to finsh my experiments regarding [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… and the related code changes. For those interested, here's the detailed story: "S08-64" System (beo-24): - 2x 64-bit Dual-Core Opteron270 @ 2 Ghz - 8 GB memory - MonetDB/XQuery 0.20, 64-bit, 64-bit OIDs, --enable-optimize (gcc 4.1.2) "S16-32" System (core-1): - 4x 64-bit Dual-Core Opteron870 @ 2 Ghz - 16 GB memory - MonetDB/XQuery 0.20, 64-bit, 32-bit OIDs, --enable-optimize (gcc 4.1.2) Document: http://mirror.openstreetmap.nl/planet/planet-071003.osm.bz2 (extracted: 19 GB XML file) "SR" Shredding read-only: pf:add-doc(".../planet-071003.osm","planet-071003.osm") "SU" Shredding updateable: pf:add-doc(".../planet-071003.osm","planet-071003.osm","planet-071003.osm",5) "QR"/"QU" Count query: count(doc("planet-071003.osm")//*) Configurations: m: without Peter's mmap fix in gdk_posix.mx (i.e., using rev. 1.143 of gdk_posix.mx) M: with Peter's mmap fix in gdk_posix.mx (i.e., using rev. 1.143.2.1 of gdk_posix.mx) h: without Peter's new string hash function in gdk_atoms.mx (i.e., using rev. 1.134 of gdk_atoms.mx) H: with Peter's new string hash function in gdk_atoms.mx (i.e., using rev. 1.134.6.1 of gdk_atoms.mx) Results (wall-clock times): S08-64: SR QR SU QU[1] QU[2] mh 659m34s 1m09s 81m06s ERROR - mH - - 383m17s ERROR - Mh - - 77m03s 344m32s 1m34s MH 644m01s 0m49s 390m42s 342m47s 1m36s S16-32: SR QR SU QU[1] QU[2] mh 127m59s 0m17s 43m14s ERROR - mH 110m33s 0m16s 26m26s ERROR - Mh 128m11s 0m18s 44m00s 100m50s 1m15s MH 191m42s 0m17s 25m43s 101m37s 1m21s (NB: "SR" includes building of indices, while "SU" does not; consequently, "QR" can exploit the indices built during "SR", while "QU[1]" has to build the indices first, and only "QU[2]" can exploit them.) Apparently, the mmap fix in gdk_posix.mx seems to be sufficient to prevent the remap-ERROR reported (for a system & configuration similar to "S08-64") in [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… I'll leave the further interpretation of the above results to the interested recipient / reader. Stefan On Tue, Oct 16, 2007 at 01:46:47PM +0200, Peter Boncz wrote: > Hi, > > Hm, I cannot really understand the purpose of the question. And what is > wrong with performance fixes? > > both fixes are related to the same bug: > - the remap failing is addressed by the gdk_posix fix > - the shredding in the bug report taking excessively long is addressed by > the gdk_atoms fix > > indeed, any hash function can have collisions.. it all depends on the > distribution. > > Peter > > PS most probably, this mail (sent from my home account) will be rejected by > the sourceforge mailing list -- and I cannot sent through CWI from home as > the secure mail sending is not supported by CWI staff for microsoft > emailers. > > > -----Original Message----- > From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] > Sent: dinsdag 16 oktober 2007 12:01 > To: Peter Boncz > Cc: monetdb-developers(a)lists.sourceforge.net > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, > 1.143.2.1 > > > Peter, > > which part of your changes do fix the problem with updatedable shredding of > large XML documents as reporten in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > ? > > The new has string function in gdk_atoms.mx or the file descriptor fixes in > gdk_posix.mx? > > The former looks for like a performance fix to me --- too many collisions > should only slows the system down, but not copromize its > fucntionallity/correctness, right? > Also with the new string has functions ("too") many collisions can still > occur with certain datasets ... > > Stefan > > > On Sun, Oct 14, 2007 at 08:31:36PM +0000, Stefan Manegold wrote: > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15103 > > > > Modified Files: > > Tag: MonetDB_1-20 > > gdk_atoms.mx gdk_posix.mx > > Log Message: > > > > [checkin on behalf of Peter] > > > > fixing XQuery bug > > [ 1811229 ] [ADT] Adding large document, with update support > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > > > > gdk_atoms.mx: > > - hash collisions in strings that consists of digits only (a common case!) > > we now use a fast derivative of the Bob Jenkins function from now on > > > > Really bad collisions, in case of the 20GB document of the bug report, > > shredding took 8 hours before, 1 hour after this change. > > > > NOTE: this change affects the binary format (string heaps) and all > product > > families, as the hash function is a compiled-in macro! > > In particular, lookup operations and joins on SQL (Monet4/5) > columns > > consisting of digits only, but stored in a VARCHAR, should be > faster > > after this check-in. > > > > gdk_posix.mx > > - we lost track of the file descriptor for large heaps (the file desc is > given > > to the mmap-monitoring-thread to close later), such that the remap > function > > could fail (when it was given the illegal file descriptor 0) > > > > NOTE: this change only affects xquery it only uses remap() -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] Compiling MonetDB5 (Current) failed on GNU.32.32.d-Fedora6
by Fabian Groffen 17 Oct '07

17 Oct '07

Maybe it is better to use fork/exec here instead of system as well when fixing this? You could "hardcode" the mcc path in your call, using what you got from autoconf so you more probably get the tool you expect. On 17-10-2007 08:53:11 +0200, Testweb Compilation wrote: > 852 char buf[1024]; > 853 snprintf(buf,1024,"mcc %s",*fname); > 854 system(buf); > 855 (void) ret; > 856 return MAL_SUCCEED; > > /export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/src/compiler/mal_compiler.mx: In function 'MCmcc': > /export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/src/compiler/mal_compiler.mx:854: warning: ignoring return value of 'system', declared with attribute warn_unused_result > make[5]: *** [lib_mal_compiler_la-mal_compiler.lo] Error 1 > make[5]: Leaving directory `/export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/Fedora6/src/compiler' > make[4]: *** [all] Error 2 > make[4]: Leaving directory `/export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/Fedora6/src/compiler' > make[3]: *** [all-recursive] Error 1 > make[3]: Leaving directory `/export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/Fedora6/src' > make[2]: *** [all] Error 2 > make[2]: Leaving directory `/export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/Fedora6/src' > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory `/export/scratch1/monet/monet.GNU.32.32.d.8963/MonetDB5/Fedora6' > make: *** [all] Error 2 > grep: /export/scratch1/monet/monet.GNU.32.32.d.8963/TestTools/ALL.MonetDB5: No such file or directory

2 1

[Monetdb-developers] PhD Thesis Steenhagen Optimization of object query languages
by darabi＠web.de 16 Oct '07

16 Oct '07

Hello, sorry for this OT question. I was reading the literature relevant to what I would like to do and found the papers of H. Steenhagen. A thesis is referenced very often which I could order a dead-tree copy of. But that would probably take very long: Title: Optimization of object query languages Author: Hendrika Janna Steenhagen (1954-) Year: 1995 Note: Proefschrift Universiteit Twente Met lit. opg., index. - Met een samenvatting in het Nederlands ISBN: 90-9008745-1 Does anybody on the list have a digital copy of it or knows where one is available? Many thanks Kambiz

3 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel algebra.mx, , 1.184, 1.185 bat5.mx, , 1.26, 1.27 mmath.mx, , 1.35, 1.36
by Romulo Goncalves 16 Oct '07

16 Oct '07

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv28642 > > Modified Files: > algebra.mx bat5.mx mmath.mx > Log Message: > Move all signatures to the .h section to ease compilation of > programs using the interface directly. More files to do. > > > Index: mmath.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/mmath.mx,v > retrieving revision 1.35 > retrieving revision 1.36 > diff -u -d -r1.35 -r1.36 > --- mmath.mx 13 Sep 2007 13:45:13 -0000 1.35 > +++ mmath.mx 14 Oct 2007 08:41:47 -0000 1.36 > @@ -206,16 +206,9 @@ > @h > #ifndef __MMATH_H__ > #define __MMATH_H__ > - > -#include <gdk.h> > - > -#endif /* __MMATH_H__ */ > - > -@+ Implementation Code > -@c > #include "mal_config.h" > - > -#include "mmath.h" > +#include "mal.h" > +#include "mal_exception.h" > #include <math.h> > > extern double sqrt(double x); > @@ -234,29 +227,6 @@ > #define mmath_export extern > #endif > > -#define acos_unary(x, z) *z = acos(*x) > -#define asin_unary(x, z) *z = asin(*x) > -#define atan_unary(x, z) *z = atan(*x) > -#define atan2_binary(x, y, z) *z = atan2(*x,*y) > -#define cos_unary(x, z) *z = cos(*x) > -#define sin_unary(x, z) *z = sin(*x) > -#define tan_unary(x, z) *z = tan(*x) > - > -#define cosh_unary(x, z) *z = cosh(*x) > -#define sinh_unary(x, z) *z = sinh(*x) > -#define tanh_unary(x, z) *z = tanh(*x) > - > -#define exp_unary(x, z) *z = exp(*x) > -#define log_unary(x, z) *z = log(*x) > -#define log10_unary(x, z) *z = log10(*x) > - > -#define pow_binary(x, y, z) *z = pow(*x,*y) > -#define sqrt_unary(x, z) *z = sqrt(*x) > - > -#define ceil_unary(x, z) *z = ((-1.0<*x)&&(*x<0.0))?0.0:ceil(*x) > -#define fabs_unary(x, z) *z = fabs(*x) > -#define floor_unary(x, z) *z = floor(*x) > -#define fmod_binary(x, y, z) *z = fmod(*x,*y) > > #ifdef _MSC_VER > #include <float.h> > @@ -306,11 +276,78 @@ > #endif /* HAVE_ISINF */ > #endif /* HAVE_FPCLASSIFY */ > > -#include "mal.h" > -#include "mal_exception.h" > -mmath_export str math_unary_FINITE(bit *res, dbl *a); > -mmath_export str math_unary_ISNAN(bit *res, dbl *a); > -mmath_export str math_unary_ISINF(int *res, dbl *a); > +#define acos_unary(x, z) *z = acos(*x) > +#define asin_unary(x, z) *z = asin(*x) > +#define atan_unary(x, z) *z = atan(*x) > +#define atan2_binary(x, y, z) *z = atan2(*x,*y) > +#define cos_unary(x, z) *z = cos(*x) > +#define sin_unary(x, z) *z = sin(*x) > +#define tan_unary(x, z) *z = tan(*x) > + > +#define cosh_unary(x, z) *z = cosh(*x) > +#define sinh_unary(x, z) *z = sinh(*x) > +#define tanh_unary(x, z) *z = tanh(*x) > + > +#define exp_unary(x, z) *z = exp(*x) > +#define log_unary(x, z) *z = log(*x) > +#define log10_unary(x, z) *z = log10(*x) > + > +#define pow_binary(x, y, z) *z = pow(*x,*y) > +#define sqrt_unary(x, z) *z = sqrt(*x) > + > +#define ceil_unary(x, z) *z = ((-1.0<*x)&&(*x<0.0))?0.0:ceil(*x) > +#define fabs_unary(x, z) *z = fabs(*x) > +#define floor_unary(x, z) *z = floor(*x) > +#define fmod_binary(x, y, z) *z = fmod(*x,*y) > + > +@= unopbaseM5_export > +mmath_export str MATHunary@1@3(@3 *res , @3 *a ); > +@= unopM5_export > + @:unopbaseM5_export(@1,@2,dbl)@ > + @:unopbaseM5_export(@1,@2,flt)@ > +@= binopbaseM5_export > +mmath_export str MATHbinary@1@3(@3 *res, @3 *a, @3 *b ); > +@= binopM5_export > + @:binopbaseM5_export(@1,@2,dbl)@ > + @:binopbaseM5_export(@1,@2,flt)@ > + > +@h > +@:unopM5_export(_ACOS,acos)@ > +@:unopM5_export(_ASIN,asin)@ > +@:unopM5_export(_ATAN,atan)@ > +@:binopM5_export(_ATAN2,atan2)@ > +@:unopM5_export(_COS,cos)@ > +@:unopM5_export(_SIN,sin)@ > +@:unopM5_export(_TAN,tan)@ > + > +@:unopM5_export(_COSH,cosh)@ > +@:unopM5_export(_SINH,sinh)@ > +@:unopM5_export(_TANH,tanh)@ > + > +@:unopM5_export(_EXP,exp)@ > +@:unopM5_export(_LOG,log)@ > +@:unopM5_export(_LOG10,log10)@ > + > +@:binopM5_export(_POW,pow)@ > +@:unopM5_export(_SQRT,sqrt)@ > + > +@:unopM5_export(_CEIL,ceil)@ > +@:unopbaseM5_export(_FABS,fabs,dbl)@ > +@:unopM5_export(_FLOOR,floor)@ > +@:binopM5_export(_FMOD,fmod)@ > + > + > +mmath_export str MATHunary_ISNAN(bit *res, dbl *a); > +mmath_export str MATHunary_ISINF(int *res, dbl *a); > +mmath_export str MATHunary_FINITE(bit *res, dbl *a); > +mmath_export str MATHrandint(int *res); > +mmath_export str MATHsrandint(int *seed); > +mmath_export str MATHpi(dbl *pi); > +#endif /* __MMATH_H__ */ > + > +@+ Implementation Code > +@c > +#include "mmath.h" > > @:unop(_ACOS,acos)@ > @:unop(_ASIN,asin)@ > @@ -375,7 +412,6 @@ > } > return MAL_SUCCEED; > } > -@ > > @c > str > @@ -453,7 +489,6 @@ > Wrapping around the M4 code base > @- > @= unopbaseM5 > -mmath_export str MATHunary@1@3(@3 *res , @3 *a ); > str MATHunary@1@3(@3 *res , @3 *a ) { > #ifdef DEBUG > printf( "MATHunary@1@3\n"); > @@ -476,7 +511,6 @@ > @:unopbaseM5(@1,@2,flt)@ > > @= binopbaseM5 > -mmath_export str MATHbinary@1@3(@3 *res, @3 *a, @3 *b ); > str MATHbinary@1@3(@3 *res, @3 *a, @3 *b ) { > #ifdef DEBUG > printf( "MATHbinary@1\n"); > @@ -495,7 +529,6 @@ > @:binopbaseM5(@1,@2,flt)@ > @ > @= roundM5 > -mmath_export str MATHbinary_ROUND@1(@1 *res, @1 *x, int *y); > str MATHbinary_ROUND@1(@1 *res, @1 *x, int *y) { If you remove the export, you will break the compilation on windows. Maybe the following Mserver5 start can tell you which functions should be exported, but unfortunately they aren't. ;) E:\MonetDB\Current\sql\NT>RunMserver5.bat > # MonetDB server v5.3.0, based on kernel v1.21.0 > # Serving database 'demo' > # Compiled for i686-pc-win32/32bit with 32bit OIDs dynamically linked > # Copyright (c) 1993-2007 CWI, all rights reserved > # Visit http://monetdb.cwi.nl/ for further information > #warning: please don't forget to set your vault key! > #(see E:\MonetDB\Current\MonetDB5\NT\etc\monetdb5.conf) > !MALException:MAL.getAddress:address of 'mmath.MATHbinary_ROUNDflt' not > found > !MALException:MAL.getAddress:address of 'mmath.MATHbinary_ROUNDdbl' not > found > !MALException:MAL.getAddress:address of 'mmath.math_unary_ISNAN' not found > !MALException:MAL.getAddress:address of 'mmath.math_unary_ISINF' not found > !MALException:MAL.getAddress:address of 'mmath.math_unary_FINITE' not found > !MALException:MAL.getAddress:address of 'algebra.ALGselectNotNil' not found > !MALException:MAL.getAddress:address of 'algebra.ALGuhashsplit' not found > !MALException:MAL.getAddress:address of 'group.GRPmin' not found > !MALException:MAL.getAddress:address of 'group.GRPmax' not found > # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ > > >\q Regards, Romulo

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Stefan Manegold 16 Oct '07

16 Oct '07

Peter, the purpose and/or reason of/for my question is the fact that we cannot release the binary packages that Sjoerd built yesterday, because the binary database format has been modified without increasing the GDK version number. We simply forgot the latter. Hence, a new Mserver cannot notice in case it is started on an pld database, and instead of complaining properly, it will just behave strangely, crash and/or corrup the database. This is not good. Consequently, Sjoerd needs to re-do the building once again, and we have to think of how to solve the above problem. Obviously, one solution is th "simply" increase the GDK version number. But then, users have to abandon or dump/retore their databases now and with the next release (GDK-2). An other alternative would be to undo the string hash function changes in the release branch, keeping the same binary format for now, and include both binary format changes in one release. In particular if the string hash function changes are "only" for performance reasons, both Sjoerd, Niels, and I would favor the second solution. After all, we and our users have been able to live with the "old" string hash function for years. Those that really require the new one can switch to the development version. I'm busy with testing --- on a 8GB dual-dual-core 2GHz Opteron running 64-bit FedoraCore 6 and and optimized 64-bit Mserver with 64-bit OIDs, shredding the 19 GB file read-only took 11 hours, both with the old and the new hash functions (Mserver grew to 10 GB res, >50 GB virt). Shredding updateable without your two fixes took only 2 hours, but the resulting DB is currupt as reported in [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 . Shredding updateable with your two fixes is still busy, running for more than 6 hours, now... Stefan On Tue, Oct 16, 2007 at 01:46:47PM +0200, Peter Boncz wrote: > Hi, > > Hm, I cannot really understand the purpose of the question. And what is > wrong with performance fixes? > > both fixes are related to the same bug: > - the remap failing is addressed by the gdk_posix fix > - the shredding in the bug report taking excessively long is addressed by > the gdk_atoms fix > > indeed, any hash function can have collisions.. it all depends on the > distribution. > > Peter > > PS most probably, this mail (sent from my home account) will be rejected by > the sourceforge mailing list -- and I cannot sent through CWI from home as > the secure mail sending is not supported by CWI staff for microsoft > emailers. > > > -----Original Message----- > From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] > Sent: dinsdag 16 oktober 2007 12:01 > To: Peter Boncz > Cc: monetdb-developers(a)lists.sourceforge.net > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, > 1.143.2.1 > > > Peter, > > which part of your changes do fix the problem with updatedable shredding of > large XML documents as reporten in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > ? > > The new has string function in gdk_atoms.mx or the file descriptor fixes in > gdk_posix.mx? > > The former looks for like a performance fix to me --- too many collisions > should only slows the system down, but not copromize its > fucntionallity/correctness, right? > Also with the new string has functions ("too") many collisions can still > occur with certain datasets ... > > Stefan > > > On Sun, Oct 14, 2007 at 08:31:36PM +0000, Stefan Manegold wrote: > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15103 > > > > Modified Files: > > Tag: MonetDB_1-20 > > gdk_atoms.mx gdk_posix.mx > > Log Message: > > > > [checkin on behalf of Peter] > > > > fixing XQuery bug > > [ 1811229 ] [ADT] Adding large document, with update support > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > > > > gdk_atoms.mx: > > - hash collisions in strings that consists of digits only (a common case!) > > we now use a fast derivative of the Bob Jenkins function from now on > > > > Really bad collisions, in case of the 20GB document of the bug report, > > shredding took 8 hours before, 1 hour after this change. > > > > NOTE: this change affects the binary format (string heaps) and all > product > > families, as the hash function is a compiled-in macro! > > In particular, lookup operations and joins on SQL (Monet4/5) > columns > > consisting of digits only, but stored in a VARCHAR, should be > faster > > after this check-in. > > > > gdk_posix.mx > > - we lost track of the file descriptor for large heaps (the file desc is > given > > to the mmap-monitoring-thread to close later), such that the remap > function > > could fail (when it was given the illegal file descriptor 0) > > > > NOTE: this change only affects xquery it only uses remap() > > > > > > Index: gdk_posix.mx > > =================================================================== > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > > retrieving revision 1.143 > > retrieving revision 1.143.2.1 > > diff -u -d -r1.143 -r1.143.2.1 > > --- gdk_posix.mx 4 Sep 2007 17:55:20 -0000 1.143 > > +++ gdk_posix.mx 14 Oct 2007 20:31:33 -0000 1.143.2.1 > > @@ -615,7 +615,7 @@ > > MT_mmap_tab[i].writable = writable; > > MT_mmap_tab[i].fd = fd; > > MT_mmap_tab[i].pincnt = 0; > > - fd = -1; > > + fd = -fd; > > } > > (void) pthread_mutex_unlock(&MT_mmap_lock); > > return fd; > > @@ -1051,9 +1051,7 @@ > > } > > if (ret != (void *) -1L) { > > hdl->fixed = ret; > > - fd = MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > > - if (fd <= 0) > > - hdl->hdl = (void *) 0; /* MT_mmap_new keeps the fd */ > > + hdl->hdl = (void*) (ssize_t) MT_mmap_new(path, ret, len, fd, (mode & > MMAP_WRITABLE)); > > } > > return ret; > > } > > @@ -1061,13 +1059,12 @@ > > void * > > MT_mmap_remap(MT_mmap_hdl *hdl, off_t off, size_t len) > > { > > - void *ret; > > - > > - ret = mmap(hdl->fixed, > > + int fd = (int) (ssize_t) hdl->hdl; > > + void *ret = mmap(hdl->fixed, > > len, > > ((hdl->mode & MMAP_WRITABLE) ? PROT_WRITE : 0) | PROT_READ, > > ((hdl->mode & MMAP_COPY) ? (MAP_PRIVATE | MAP_NORESERVE) : > MAP_SHARED) | (hdl->fixed ? MAP_FIXED : 0), > > - (int) (ssize_t) hdl->hdl, > > + (fd < 0)?-fd:fd, > > off); > > > > if (ret != (void *) -1L) { > > @@ -1083,9 +1080,7 @@ > > MT_mmap_close(MT_mmap_hdl *hdl) > > { > > int fd = (int) (ssize_t) hdl->hdl; > > - > > - if (fd) > > - close(fd); > > + if (fd > 0) close(fd); > > hdl->hdl = NULL; > > } > > > > > > Index: gdk_atoms.mx > > =================================================================== > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_atoms.mx,v > > retrieving revision 1.134 > > retrieving revision 1.134.6.1 > > diff -u -d -r1.134 -r1.134.6.1 > > --- gdk_atoms.mx 2 May 2007 16:16:58 -0000 1.134 > > +++ gdk_atoms.mx 14 Oct 2007 20:31:32 -0000 1.134.6.1 > > @@ -1878,13 +1878,19 @@ > > rotates all characters together. It is optimized to process 2 characters > > at a time (adding 16-bits to the hash value each iteration). > > @h > > -#define GDK_STRHASH(x,y) { \ > > - str _c = (str) (x); \ > > - for((y)=0; _c[0] && _c[1]; _c+=2) { \ > > - (y) = ((y) << 3) ^ ((y) >> 11) ^ ((y) >> 17) ^ (_c[1] << > 8) ^ _c[0];\ > > - } \ > > - (y) ^= _c[0]; \ > > +#define GDK_STRHASH(x,y) {\ > > + str _key = (str) (x);\ > > + int _i;\ > > + for (_i = y = 0; _key[_i]; _i++) {\ > > + y += _key[_i];\ > > + y += (y << 10);\ > > + y ^= (y >> 6);\ > > + }\ > > + y += (y << 3);\ > > + y ^= (y >> 11);\ > > + y += (y << 15);\ > > } > > + > > @c > > hash_t > > strHash(str s) > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. > > Still grepping through log files to find problems? Stop. > > Now Search log events and configuration files using AJAX and a browser. > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > _______________________________________________ > > Monetdb-checkins mailing list > > Monetdb-checkins(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Stefan Manegold 16 Oct '07

16 Oct '07

Peter, which part of your changes do fix the problem with updatedable shredding of large XML documents as reporten in [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… ? The new has string function in gdk_atoms.mx or the file descriptor fixes in gdk_posix.mx? The former looks for like a performance fix to me --- too many collisions should only slows the system down, but not copromize its fucntionallity/correctness, right? Also with the new string has functions ("too") many collisions can still occur with certain datasets ... Stefan On Sun, Oct 14, 2007 at 08:31:36PM +0000, Stefan Manegold wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15103 > > Modified Files: > Tag: MonetDB_1-20 > gdk_atoms.mx gdk_posix.mx > Log Message: > > [checkin on behalf of Peter] > > fixing XQuery bug > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… > > gdk_atoms.mx: > - hash collisions in strings that consists of digits only (a common case!) > we now use a fast derivative of the Bob Jenkins function from now on > > Really bad collisions, in case of the 20GB document of the bug report, > shredding took 8 hours before, 1 hour after this change. > > NOTE: this change affects the binary format (string heaps) and all product > families, as the hash function is a compiled-in macro! > In particular, lookup operations and joins on SQL (Monet4/5) columns > consisting of digits only, but stored in a VARCHAR, should be faster > after this check-in. > > gdk_posix.mx > - we lost track of the file descriptor for large heaps (the file desc is given > to the mmap-monitoring-thread to close later), such that the remap function > could fail (when it was given the illegal file descriptor 0) > > NOTE: this change only affects xquery it only uses remap() > > > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.143 > retrieving revision 1.143.2.1 > diff -u -d -r1.143 -r1.143.2.1 > --- gdk_posix.mx 4 Sep 2007 17:55:20 -0000 1.143 > +++ gdk_posix.mx 14 Oct 2007 20:31:33 -0000 1.143.2.1 > @@ -615,7 +615,7 @@ > MT_mmap_tab[i].writable = writable; > MT_mmap_tab[i].fd = fd; > MT_mmap_tab[i].pincnt = 0; > - fd = -1; > + fd = -fd; > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > return fd; > @@ -1051,9 +1051,7 @@ > } > if (ret != (void *) -1L) { > hdl->fixed = ret; > - fd = MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > - if (fd <= 0) > - hdl->hdl = (void *) 0; /* MT_mmap_new keeps the fd */ > + hdl->hdl = (void*) (ssize_t) MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > } > return ret; > } > @@ -1061,13 +1059,12 @@ > void * > MT_mmap_remap(MT_mmap_hdl *hdl, off_t off, size_t len) > { > - void *ret; > - > - ret = mmap(hdl->fixed, > + int fd = (int) (ssize_t) hdl->hdl; > + void *ret = mmap(hdl->fixed, > len, > ((hdl->mode & MMAP_WRITABLE) ? PROT_WRITE : 0) | PROT_READ, > ((hdl->mode & MMAP_COPY) ? (MAP_PRIVATE | MAP_NORESERVE) : MAP_SHARED) | (hdl->fixed ? MAP_FIXED : 0), > - (int) (ssize_t) hdl->hdl, > + (fd < 0)?-fd:fd, > off); > > if (ret != (void *) -1L) { > @@ -1083,9 +1080,7 @@ > MT_mmap_close(MT_mmap_hdl *hdl) > { > int fd = (int) (ssize_t) hdl->hdl; > - > - if (fd) > - close(fd); > + if (fd > 0) close(fd); > hdl->hdl = NULL; > } > > > Index: gdk_atoms.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_atoms.mx,v > retrieving revision 1.134 > retrieving revision 1.134.6.1 > diff -u -d -r1.134 -r1.134.6.1 > --- gdk_atoms.mx 2 May 2007 16:16:58 -0000 1.134 > +++ gdk_atoms.mx 14 Oct 2007 20:31:32 -0000 1.134.6.1 > @@ -1878,13 +1878,19 @@ > rotates all characters together. It is optimized to process 2 characters > at a time (adding 16-bits to the hash value each iteration). > @h > -#define GDK_STRHASH(x,y) { \ > - str _c = (str) (x); \ > - for((y)=0; _c[0] && _c[1]; _c+=2) { \ > - (y) = ((y) << 3) ^ ((y) >> 11) ^ ((y) >> 17) ^ (_c[1] << 8) ^ _c[0];\ > - } \ > - (y) ^= _c[0]; \ > +#define GDK_STRHASH(x,y) {\ > + str _key = (str) (x);\ > + int _i;\ > + for (_i = y = 0; _key[_i]; _i++) {\ > + y += _key[_i];\ > + y += (y << 10);\ > + y ^= (y >> 6);\ > + }\ > + y += (y << 3);\ > + y ^= (y >> 11);\ > + y += (y << 15);\ > } > + > @c > hash_t > strHash(str s) > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] XRPCwrapper fails to compile with java 1.4
by Stefan Manegold 16 Oct '07

16 Oct '07

Hoi, apparently, the new XPRCwrapper code fails to compile with java 1.4 (our RedHat 4 WS machine, titan, is the only testing machine that still has only java 1.4, all others have java 1.5); for details see below and/or http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.ntv.64.64… Question: Assuming that there is no easy way to make the new XRPC warpper code java 1.4 compliant: should we modify (tighten) our requirements accordingly, i.e., requiring (only) java 1.5 instead of (at least) java 1.4 to build MonetDB* (java 1.6 is not (yet?) supported)? Stefan ======== "/usr/local/bin/ant" -f "`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml" -Dbuilddir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java" -Djardir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java" -Dbasedir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java" distall Buildfile: /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml distall: prepare: [echo] Debug is true, optimise is true compile_util: [echo] Compiling Utilities [javac] Compiling 9 source files to /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/NamespaceContextImpl.java:22: package javax.xml.namespace does not exist [javac] import javax.xml.namespace.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/NamespaceContextImpl.java:29: cannot resolve symbol [javac] symbol : class NamespaceContext [javac] location: class nl.cwi.monetdb.xquery.util.NamespaceContextImpl [javac] public class NamespaceContextImpl implements NamespaceContext{ [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:23: package javax.xml.namespace does not exist [javac] import javax.xml.namespace.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:24: package javax.xml.xpath does not exist [javac] import javax.xml.xpath.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:28: package com.sun.org.apache.xml.internal.dtm.ref does not exist [javac] import com.sun.org.apache.xml.internal.dtm.ref.DTMNodeList; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:372: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:409: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] public static NamedNodeMap getNodeAttributes(XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:432: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] public static String[] getNodeListAttribute(XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:415: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:415: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:416: cannot resolve symbol [javac] symbol : variable XPathConstants [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] inputSource, XPathConstants.NODESET); [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:439: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:439: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:440: cannot resolve symbol [javac] symbol : variable XPathConstants [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] inputSource, XPathConstants.NODESET); [javac] ^ [javac] 14 errors BUILD FAILED /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml:55: The following error occurred while executing this line: /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml:107: Compile failed; see the compiler error output for details. Total time: 6 seconds make[7]: *** [distall_ant_target] Error 1 ======== -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5 12

Re: [Monetdb-developers] BAT sizes
by Stefan Manegold 15 Oct '07

15 Oct '07

[I once again felt free to share this with the community ...] Henning, in case BAT capacities are significantly larger than their actual content (count), this might indeed have a negative influence on performance. (1) in case the BAT is memory-mapped when loaded, it "only" blocks some more address space than strictly necessary (no problem on 64-bit systems, potentially a problem on 32-bit systems); (2) in case the BAT is malloced when loaded, it also occupied some more memory than strictly necessary (potential problem on but 64- & 32-bit systems). However, unless there is some accurate estimation, it is often hard (or virually impossible) to "guess" a BATs size before filling it; hence, a "generous" initial size allocation is good to avoid expensive BAT extents. In your case, I'm lost concerning which BATs your taling about. The shredder-generated pre_* (actually rid_*) BATs need to be allocated before reading the document; hence, there is know knowledge about the number of nodes in the document, and as far as I can tell no trivial way to estimate this accurately. Hence, the shredder needs to guess something --- JanF can tell more, I guess... In case of the TIJAH indices, I have no clue at all, how/where/when they are built and whether there might be better information available to not overallocate but allocate only just enough space. You or some of your colleagues in Twente should know all the details. Finally, is there any concrete case where you actually experiences any problems due to "over-allocation", or are you just wondering? Stefan ps: in the case given below, the batsize just fits the BAT's capacity; only the count is smaller than the capacity (obviously, it cannot be larger) --- if you want/need to know why, you better ask him/her who allocated/created/filled the "tj_DFLT_FT_INDEX_size1" BAT ... On Mon, Oct 15, 2007 at 02:36:46PM +0200, Henning Rode wrote: > hej stefan, > > sorry, that i did not answer earlier. i justed wanted to report the > actual sizes of pf/tijah indices in a paper. so that is done now. > > still, i was asking myself, whether it might have any kind of > performance influences, that BAT capacities are so much higher than the > actual BAT counts. This is of course handy, when we still want to add > new entries, but once we indexed a collection, we usually only query it. > > in case of our "pre_size" BAT this difference between BATsize and > BATdsksize can easily be 250MB or more. > > best -henning > > mil>var t := bat("tj_DFLT_FT_INDEX_size1"); > mil>t.count().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |203091470 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.capacity().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |260898816 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.batsize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |1043599360 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.batdsksize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |812366848 > > > > mil>var x := t.copy(); > mil>x.count().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |203091470 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.capacity().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |260898816 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.batsize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |1043599360 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.batdsksize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |812366848 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.access(BAT_READ); > > > Stefan Manegold wrote: > > Henning, > > > > you should also check & report b.capacity(), i.e., > > > > b.count(); > > b.capacity(); > > b.info().reverse().like("batBuns").like("size").print(); > > b.batsize(); > > b.batdsksize(); > > > > var c := b.copy(); > > > > c.count(); > > c.capacity(); > > c.info().reverse().like("batBuns").like("size").print(); > > c.batsize(); > > c.batdsksize(); > > > > Stefan > > > > > > On Sat, Oct 06, 2007 at 07:43:28PM +0200, Stefan Manegold wrote: > >> [felt free to cc the monetdb-developers list as more people might be > >> interested or want to contribute] > >> > >> Henning, > >> > >> are you just "concerned" or are you having concrete problems with the bat > >> sizes? > >> > >> In cany case, to give any reasonable answer we'd need to know more about the > >> details. In particular how large is the BAT your talking about. > >> > >> I.e., with "b" being your BAT and "c := b.copy()", please check & report > >> > >> b.count(); > >> b.info().reverse().like("batBuns").like("size").print(); > >> b.batsize(); > >> b.batdsksize(); > >> > >> c.count(); > >> c.info().reverse().like("batBuns").like("size").print(); > >> c.batsize(); > >> c.batdsksize(); > >> > >> Stefan > >> > >> > >> On Fri, Oct 05, 2007 at 01:47:01PM +0200, Henning Rode wrote: > >>> hej stefan, > >>> > >>> thanks for the answer. so in conclusion, the over-allocation of memory > >>> is quite normal, and nothing to worry about. > >>> > >>> i was more surprised that the copied BAT still has this considerable > >>> over-allocation of memory, though it exactly knows how many entries it > >>> needs to hold. > >>> > >>> groeten -henning > >> -- > >> | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > >> | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > >> | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > >> | The Netherlands | Fax : +31 (20) 592-4312 | > > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1