- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_octopus.mx, , 1.57, 1.58 opt_macro.mx, , 1.47, 1.48
by Martin Kersten 30 Jul '09

30 Jul '09

Fabian Groffen wrote: > On 29-07-2009 20:19:48 +0000, Martin Kersten wrote: >> octopus - check if the head is managed by merovingian > >> - if (list == NULL || needed == NULL){ >> + /* check if the head is properly administered for Octopus */ >> + SABAOTHgetLocalConnection(&name); > > This check will (almost) always succeed, setting name to non-NULL and > by no means checks for merovingian. Is this what you intended to > happen? well, OPCinitcode uses this call and got a null pointer in the situation that 1) 2 legs were created through monetdb and 2) the head was an mserver5 started separately and not createdb with monetdb. So, the test was for this null string and abandon octopus code. Otherwise, the final derived code would contain a remoter.connect(nil,"monetdb",.... > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

2 1

[Monetdb-developers] Announcement: May2009-SP2 bug fix release
by Sjoerd Mullender 29 Jul '09

29 Jul '09

The MonetDB team at CWI/MonetDB BV is pleased to announce the May 2009-SP2 bug fix release of the MonetDB suite of programs. This is the second bug fix release for the May2009 release. More information on this release can be found at <http://monetdb.cwi.nl/projects/monetdb/Development/Releases/May2009/#May200…>. The sources and binaries are available at <http://monetdb.cwi.nl/projects/monetdb/Download/>. -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal algebraExtensions.mx, , 1.60, 1.61
by Martin Kersten 28 Jul '09

28 Jul '09

Should i restart the 400Mx400M experiment? I see a lot of BBPtrims passing by Stefan Manegold wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv1551/MonetDB5/src/modules/mal > > Modified Files: > algebraExtensions.mx > Log Message: > > revive 1Gx1G by creating less partitions in the first step > > > U algebraExtensions.mx > Index: algebraExtensions.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/algebraExtensions.mx,v > retrieving revision 1.60 > retrieving revision 1.61 > diff -u -d -r1.60 -r1.61 > --- algebraExtensions.mx 28 Jul 2009 17:01:05 -0000 1.60 > +++ algebraExtensions.mx 28 Jul 2009 18:45:50 -0000 1.61 > @@ -780,7 +780,6 @@ > } > else > { > - int n = 0; > /* smaller exceeds L2 > * => partition for L2 in n steps (depth-first) such that no step > * generates more than 1024 new partitions per partition > @@ -803,8 +802,10 @@ > parts <<= 1; > } > @) > - n = ((p - 1) / 10) + 1; > - p /= n; > + p = p % 10; > + if (p == 0) { > + p = 10; > + } > parts = 1 << p; > } > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/kernel unix.mx, , 1.24, 1.25
by Sjoerd Mullender 28 Jul '09

28 Jul '09

Why do you want to be able to flush all changed files to disk? MonetDB already does flushes when they are needed for transaction support. I don't think we need to tell the kernel to synchronize any other files with the disk. This function doesn't do anything with memory management within the process. Furthermore, this function is not available on Windows. Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/kernel > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv20459 > > Modified Files: > unix.mx > Log Message: > We want to experiment with complete flushing memory in out of memory > queries. > > > U unix.mx > Index: unix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/kernel/unix.mx,v > retrieving revision 1.24 > retrieving revision 1.25 > diff -u -d -r1.24 -r1.25 > --- unix.mx 7 Jan 2009 14:15:12 -0000 1.24 > +++ unix.mx 27 Jul 2009 06:29:44 -0000 1.25 > @@ -35,6 +35,10 @@ > address UNIXsetenv > comment "Set the environment variable string."; > > +command sync() :void > +address UNIXsync > +comment "Flush memory, expensive but sometimes needed"; > + > @- Implementation > The remainder is a straight forward mapping to the underlying > facilities > @@ -113,4 +117,10 @@ > return MAL_SUCCEED; > } > > +unix_export str UNIXsync(int *res); > +str UNIXsync(int *res){ > + (void) res; > + sync(); > + return MAL_SUCCEED; > +} > @} > > > ------------------------------------------------------------------------------ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

2 1

[Monetdb-developers] Connection fails
by Riham Abdel Kader 27 Jul '09

27 Jul '09

Hi all, I have lately re-installed the Jun2008 release of MonetDB/XQuery. I can start Mserver successfully and connect to it with mclient to execute XQueries. When I try to connect to it using a MapiSocket from my java program, I get the following error: "java.net.ConnectException: Connection refused". I also tried: > telnet localhost 50000 and I get: Trying ::1... telnet: connect to address ::1: Connection refused Any idea what I could be doing wrong? Thanks for your time, Riham

3 3

[Monetdb-developers] Suggestion to ditch SHA-1, and embrace SHA-2
by Stefan de Konink 24 Jul '09

24 Jul '09

As already found in the news SHA-1 is advised to migrated off by 2010. I would suggest SHA-2 in protocol 9. Stefan

6 19

Re: [Monetdb-developers] [ monetdb-Bugs-2787825 ] mclient: stdin + statement
by Wouter Alink 23 Jul '09

23 Jul '09

The bug (#2787825) seems to be closed for comments, but I think this bug should not be closed until the documentation gets updated. typing "man mclient" on the stable branch tells me: --statement=stmt (-s stmt) Execute the specified query. The query is run before any queries from files specified on the command line are run, and before the interactive session is started (if the --interactive option is given). This is not in line with Martin's latest comment. Martin, could you re-open the bug (I don't have the permissions to do so)? Greetings, Wouter 2009/7/20 SourceForge.net <noreply(a)sourceforge.net>: > Bugs item #2787825, was opened at 2009-05-06 14:21 > Message generated for change (Comment added) made by mlkersten > You can respond by visiting: > https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2787825&group_… > > Please note that this message will contain a full copy of the comment thread, > including the initial issue submission, for this request, > not just the latest update. > Category: Mapi > Group: Clients CVS Head >>Status: Closed >>Resolution: Wont Fix > Priority: 5 > Private: No > Submitted By: Wouter Alink (vzzzbx) > Assigned to: Nobody/Anonymous (nobody) > Summary: mclient: stdin + statement > > Initial Comment: > It seems that there is a problem with both providing data via stdin and via the -s feature in mclient. (see example below). > > A possible solution could perhaps be to forbid this use. Another solution would be to define a behaviour: either read the '-s' first or the stdin first. (perhaps this already is the case, but I couldn't find any documentation about it) > > $ cat data.dat > 1 > 2 > 3 > 4 > 5 > $ N=4; head -n $N data.dat | mclient -lsql -p50151 -dtest -s "copy $N records into aap from STDIN;" > MAPI = monetdb@localhost:50151 > QUERY = copy 4 records into aap from STDIN; > ERROR = !SQLException:sql:value ';' while parsing ';' from line 0 field 0 not inserted, expecting type int > !SQLException:importTable:failed to import table > > > ---------------------------------------------------------------------- > >>Comment By: Martin Kersten (mlkersten) > Date: 2009-07-20 21:48 > > Message: > Standard input is ignored in combination with -s. > Closing it as at best it can be considered a niche feature request. > > ---------------------------------------------------------------------- > > Comment By: Wouter Alink (vzzzbx) > Date: 2009-05-07 21:55 > > Message: > as discussed on the monetdb-users list, using either the -s _or_ the stdin > works fine (except for other reported/unreported bugs), but the combination > fails. (stefan's example works fine). > > I can very well imagine that using a combination should not be allowed > (and should not even become a feature request), but I feel that the current > message is not very helpful. > > And, actually (I hadn't thought of this option before), if I would have > specified "-i" then the documentation (mclient --help) says it reads from > stdin _after_ reading the command line args, but it generates the same > error. > > After some more tests I discovered that: > - when using the command line args + stdin + mentioning '-i', the > semi-colon after "copy $N records into aap from STDIN;" should be left out, > so the following does work: > > $ echo "1 > 2 > 3 > 4 > 5" | mclient -lsql -dtest -hskadi -p50151 -i -s "COPY 5 RECORDS INTO aap > FROM STDIN" > > (notice the omission of ';' after the COPY statement) > > If I do exactly the same, but leave out the '-i', no error is displayed, > but nothing gets inserted either. > > If I use only stdin only: > > $ echo "COPY 5 RECORDS INTO aap FROM STDIN; > 1 > 2 > 3 > 4 > 5" | mclient -lsql -dtest -hskadi -p50151 > > then this works (only if the ';' after the COPY statement is present). > > I don't know whether there are two different bugs mentioned in this > explanation, but I think there definitely is something wrong. > > by the way: the create statement for aap is: "CREATE TABLE aap (x int);" > > ---------------------------------------------------------------------- > > Comment By: Stefan Manegold (stmane) > Date: 2009-05-07 19:30 > > Message: > What about: > > { N=4 ; echo "copy $N records into aap from STDIN;" ; head -n $N data.dat > ; } | mclient -lsql -p50151 -dtest > > ? > > > ---------------------------------------------------------------------- > > Comment By: Sjoerd Mullender (sjoerd) > Date: 2009-05-07 14:26 > > Message: > Where does it say that this is supported? > If it doesn't say anywhere, this is a feature request, not a bug. > > ---------------------------------------------------------------------- > > You can respond by visiting: > https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2787825&group_… >

2 3

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_inline.mx, , 1.22, 1.23
by Sjoerd Mullender 20 Jul '09

20 Jul '09

Should this be applied to the Stable branch? Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv11631 > > Modified Files: > opt_inline.mx > Log Message: > Protect lower levels against compound inline functions as well. > > > U opt_inline.mx > Index: opt_inline.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/optimizer/opt_inline.mx,v > retrieving revision 1.22 > retrieving revision 1.23 > diff -u -d -r1.22 -r1.23 > --- opt_inline.mx 16 May 2009 08:54:40 -0000 1.22 > +++ opt_inline.mx 20 Jul 2009 14:37:32 -0000 1.23 > @@ -50,6 +50,21 @@ > > extern int OPTinlineMultiplex(Client cntxt, MalBlkPtr mb, InstrPtr p); > > +static int > +isCorrectInline(MalBlkPtr mb){ > + /* make sure we have a simple inline function with a singe return */ > + InstrPtr p; > + int i, retseen=0; > + > + for( i= 1; i < mb->stop; i++){ > + p= getInstrPtr(mb,i); > + if ( p->token == RETURNsymbol || p->token == YIELDsymbol || > + p->barrier == RETURNsymbol || p->barrier == YIELDsymbol) > + retseen++; > + } > + return retseen == 1; > +} > + > int > OPTinlineImplementation(Client cntxt, MalBlkPtr mb, MalStkPtr stk, InstrPtr p) > { > @@ -85,7 +100,8 @@ > Check if the function definition is tagged as being inlined. > @c > if (sig->token == FUNCTIONsymbol && > - varGetProp(q->blk, getArg(sig, 0), inlineProp) != NULL) { > + varGetProp(q->blk, getArg(sig, 0), inlineProp) != NULL && > + isCorrectInline(q->blk) ) { > (void) inlineMALblock(mb,i,q->blk); > i--; > actions++; > > > ------------------------------------------------------------------------------ > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal algebraExtensions.mx, , 1.32, 1.33
by Stefan Manegold 18 Jul '09

18 Jul '09

On Sat, Jul 18, 2009 at 10:21:00AM +0000, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv30235 > > Modified Files: > algebraExtensions.mx > Log Message: > load and unload the bats in each round of bloom prejoin, assuming that both dont fit in memory at the same time > ... results in (when comfigured with optimization enabled): ======== gcc -DHAVE_CONFIG_H -I. -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal -I../../.. -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal -I../atoms -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../atoms -I../kernel -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../kernel -I../../mal -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../../mal -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/include/MonetDB -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/include/MonetDB/mapilib -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/include/MonetDB -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/include/MonetDB/common -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/include/MonetDB/gdk -I/usr/include/libxml2 -std=c99 -DLIBALGEBRAEXTENSIONS -O2 -Wall -Wextra -O6 -fomit-frame-pointer -finline-functions -falign-loops=4 -falign-jumps=4 -falign-functions=4 -fexpensive-optimizations -funroll-loops -frerun-cse-after-loop -frerun-loop-opt -ftree-vectorize -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 -D_REENTRANT -c algebraExtensions.c -fPIC -DPIC -o .libs/lib_algebraExtensions_la-algebraExtensions.o cc1: warnings being treated as errors /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx: In function ‘GDKbloomfilter_int_l’: /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:406: error: ‘blim’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:660: note: ‘blim’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:410: error: ‘blog’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:661: note: ‘blog’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:416: error: ‘bv’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:658: note: ‘bv’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:423: error: ‘clim’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:660: note: ‘clim’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:425: error: ‘cv’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:658: note: ‘cv’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:443: error: ‘clog’ may be used uninitialized in this function /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:661: note: ‘clog’ was declared here /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/algebraExtensions.mx:659: error: ‘b’ may be used uninitialized in this function make[8]: *** [lib_algebraExtensions_la-algebraExtensions.lo] Error 1 make[8]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src/modules/mal' make[7]: *** [all-recursive] Error 1 make[7]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src/modules/mal' make[6]: *** [all] Error 2 make[6]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src/modules/mal' make[5]: *** [all-recursive] Error 1 make[5]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src/modules' make[4]: *** [all] Error 2 make[4]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src/modules' make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src' make[2]: *** [all] Error 2 make[2]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/net/rig.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Current/build.--enable-strict_--disable-debug_--enable-optimize_--disable-assert/MonetDB5' make: *** [all] Error 2 ======== Stefan > > U algebraExtensions.mx > Index: algebraExtensions.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/algebraExtensions.mx,v > retrieving revision 1.32 > retrieving revision 1.33 > diff -u -d -r1.32 -r1.33 > --- algebraExtensions.mx 18 Jul 2009 08:43:03 -0000 1.32 > +++ algebraExtensions.mx 18 Jul 2009 10:20:58 -0000 1.33 > @@ -110,7 +110,7 @@ > is calculated for the heap(s) only. > @c > > -/* #define DEBUG_BLOOMFILTER*/ > +#define DEBUG_BLOOMFILTER > > static > size_t GDKbatfootprint(BAT *b){ > @@ -274,7 +274,7 @@ > cp = BATslice(cr, cl, cl+blk); > j = BATjoin(b,cp, BUN_NONE); > #ifdef DEBUG_BLOOMFILTER > - stream_printf(GDKout,"result " SZFMT " " SZFMT " " SZFMT "\n", bl, cl, BATcount(j)); > + stream_printf(GDKout,"result " SZFMT " " SZFMT " " SZFMT "\n", blk, cl, BATcount(j)); > #endif > if (bn == 0) > bn = BATnew( j->htype, j->ttype, BATcount(j)*blim/blk); > @@ -375,6 +375,30 @@ > } > > #define bloomhash(x,y,z) ((unsigned int) ((x)*(y))%(z)) > +#define bloomhash1(x,y,z) ((int) ((((x)*((double)y))-((int)((x)*(y))))*(z))) > +/* #define bloomhash1(x,y,z) ((unsigned int) ((x)*(y))%(z)) */ > + > +/*static double primes[]= {19,29,31,41,43,53,59,61};*/ > + > +@= print_bbloom > +#ifdef DEBUG_BLOOMFILTER > +counter = 0; > +for (i = 0; i < mb; i++) { > + counter += bbloom[i]; > +} > +stream_printf(GDKout,"bbloom has %d 1's and %d 0's\n", counter, mb-counter); > +counter = 0; > +#endif > + > +@= print_cbloom > +#ifdef DEBUG_BLOOMFILTER > +counter = 0; > +for (i = 0; i < mc; i++) { > + counter += cbloom[i]; > +} > +stream_printf(GDKout,"cbloom has %d 1's and %d 0's\n", counter, mc-counter); > +counter = 0; > +#endif > > @= body_smallsubset > char *bbloom; > @@ -474,42 +498,91 @@ > > @= body_outofmemory > char * cbloom, *bbloom; > - int bp[4]; > - int cp[4]; > + char bit; > + /*int bp[4]; > + int cp[4];*/ > + double bp[4]; > + double cp[4]; > int counter = 0; > + > + b = BATdescriptor(*bid); > + bv = (int*) Tloc(b, BUNfirst(b)); > + blim = BATcount(b); > + blog = (int) log(blim); > + > mb = blim*2; > - mc = clim*2; > + //mc = clim*2; > bbloom = (char *) GDKmalloc(mb); > - cbloom = (char *) GDKmalloc(mc); > + //cbloom = (char *) GDKmalloc(mc); > memset(bbloom, 0, mb); > - memset(cbloom, 0, mc); > + //memset(cbloom, 0, mc); > for (i = 0; i < 4; i++) { > bp[i] = (1 << (blog + i))+1; > - cp[i] = (1 << (clog + i))+1; > + //cp[i] = (1 << (clog + i))+1; > + /* > + bp[i] = primes[i]/((double) (sizeof(wrd)*8)); > + cp[i] = primes[i+1]/((double) (sizeof(wrd)*8)); > + */ > } > > - /* for (b) { built bloom } */ > + /* for (b) { built bloom } */ > for (i = 0; i < blim; i++) { > setBit(bbloom, bloomhash(bp[0],bv[i],mb), 1); > } > + /*@:print_bbloom@*/ > + BBPunfix(*bid); > + > + c = BATdescriptor(*cid); > + cv = (int*) Tloc(c, BUNfirst(c)); > + clim = BATcount(c); > + clog = (int) log(clim); > + mc = clim*2; > + cbloom = (char *) GDKmalloc(mc); > + memset(cbloom, 0, mc); > + > + for (i = 0; i < 4; i++) { > + cp[i] = (1 << (clog + i))+1; > + } > > /* for (c) { test bbloom; built cbloom } */ > for (i = 0; i < clim; i++) { > - if (tstBit(bbloom, bloomhash(bp[0],cv[i],mb))) { > + bit = tstBit(bbloom, bloomhash(bp[0],cv[i],mb)); > + setBit(cbloom, bloomhash(cp[0],cv[i],mc), > + bit | tstBit(cbloom, bloomhash(cp[0],cv[i],mc))); > + setBit(cbloom, bloomhash(cp[1],cv[i],mc), > + bit | tstBit(cbloom, bloomhash(cp[1],cv[i],mc))); > + /*if (tstBit(bbloom, bloomhash(bp[0],cv[i],mb))) { > setBit(cbloom, bloomhash(cp[0],cv[i],mc), 1); > setBit(cbloom, bloomhash(cp[1],cv[i],mc), 1); > counter++; > - } > + }*/ > } > + /*@:print_cbloom@*/ > + BBPunfix(*cid); > > #ifdef DEBUG_BLOOMFILTER > stream_printf(GDKout, "1st round c %d\n", counter); > counter = 0; > #endif > > + b = BATdescriptor(*bid); > + bv = (int*) Tloc(b, BUNfirst(b)); > memset(bbloom, 0, mb); > /* for (b) { test cbloom; rebuilt bbloom } */ > for (i = 0; i < blim; i++) { > + /* > + bit = tstBit(cbloom, bloomhash(cp[0],bv[i],mc)) && > + tstBit(cbloom, bloomhash(cp[1],bv[i],mc)); > + > + setBit(bbloom, bloomhash(bp[0],bv[i],mb), > + bit | tstBit(bbloom, bloomhash(bp[0],bv[i],mb))); > + setBit(bbloom, bloomhash(bp[1],bv[i],mb), > + bit | tstBit(bbloom, bloomhash(bp[1],bv[i],mb))); > + setBit(bbloom, bloomhash(bp[2],bv[i],mb), > + bit | tstBit(bbloom, bloomhash(bp[2],bv[i],mb))); > + setBit(bbloom, bloomhash(bp[3],bv[i],mb), > + bit | tstBit(bbloom, bloomhash(bp[3],bv[i],mb))); > + */ > if ((tstBit(cbloom, bloomhash(cp[0],bv[i],mc)) && > tstBit(cbloom, bloomhash(cp[1],bv[i],mc)))) { > setBit(bbloom, bloomhash(bp[0],bv[i],mb), 1); > @@ -519,13 +592,18 @@ > counter++; > } > } > + /*@:print_bbloom@*/ > + BBPunfix(*bid); > > #ifdef DEBUG_BLOOMFILTER > stream_printf(GDKout, "1st round b %d\n", counter); > counter = 0; > #endif > > + > /* for (c) { check bbloom } */ > + c = BATdescriptor(*cid); > + cv = (int*) Tloc(c, BUNfirst(c)); > *cn = BATnew(TYPE_oid,TYPE_int, clim/4); > memset(cbloom, 0, mc); > for (i = 0; i < clim; i++) { > @@ -542,12 +620,17 @@ > counter++; > } > } > + /*@:print_cbloom@*/ > + BBPunfix(*cid); > + GDKfree(bbloom); > > #ifdef DEBUG_BLOOMFILTER > stream_printf(GDKout, "2nd round c %d\n", counter); > counter = 0; > #endif > > + b = BATdescriptor(*bid); > + bv = (int*) Tloc(b, BUNfirst(b)); > *bn = BATnew(TYPE_oid,TYPE_int, blim/4); > /* for (b) {check cbloom} */ > for (i = 0; i < blim; i++) { > @@ -560,28 +643,30 @@ > counter++; > } > } > + BBPunfix(*bid); > + GDKfree(cbloom); > > #ifdef DEBUG_BLOOMFILTER > stream_printf(GDKout, "2nd round b %d\n", counter); > counter = 0; > #endif > > - > @c > > void > -GDKbloomfilter_int_l(BAT *b, BAT *c, BAT **bn, BAT **cn, int algo) { > +GDKbloomfilter_int_l(bat *bid, bat *cid, BAT **bn, BAT **cn, int algo) { > int *bv, *cv; > + BAT *b, *c; > BUN i, blim, clim; > int clog, blog; > BUN mb, mc; > mb = mc = 0; > > - /* b is the smaller one */ > + /* b is the smaller one > if ( BATcount(b) > BATcount(c) ) { > GDKbloomfilter_int_l(c, b, cn, bn, algo); > return; > - } > + } */ > > /* consider it an out-of-memory situation > if ( b->ttype != TYPE_int || GDKbatfootprint(b) < 25 ){ > @@ -592,12 +677,13 @@ > return; > }*/ > > + /* > bv = (int*) Tloc(b, BUNfirst(b)); > cv = (int*) Tloc(c, BUNfirst(c)); > blim= BATcount(b); > clim= BATcount(c); > clog = (int) log(clim); > - blog = (int) log(blim); > + blog = (int) log(blim); */ > > if (algo == 2) { > /* only build one bloom filter on the smaller BAT, > @@ -653,6 +739,7 @@ > (void) cntxt; > (void) mb; > > + > if ((b = BATdescriptor(*bid)) == NULL) > throw(MAL, "bbp.prejoin", INTERNAL_BAT_ACCESS); > if ( (c = BATdescriptor(*cid)) == NULL) { > @@ -667,7 +754,7 @@ > case 2: /* lefteris 1st */ > case 3: /* 2nd */ > case 4: /* 3rd */ > - GDKbloomfilter_int_l(b, c, &bn, &cn, algo); > + GDKbloomfilter_int_l(bid, cid, &bn, &cn, algo); > break; > case 5: /* blocked join */ > bn = GDKblockjoin(b,c); > > > ------------------------------------------------------------------------------ > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] testing/src Mtest.py.in, , 1.26, 1.27
by Stefan Manegold 17 Jul '09

17 Jul '09

On Thu, Jul 16, 2009 at 12:54:20PM +0000, Fabian wrote: > Update of /cvsroot/monetdb/testing/src > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv5974/src > > Modified Files: > Mtest.py.in > Log Message: > Have you ever wanted your machine to nearly die under a load of 20+ by > running Mtest? Now you can, using --jobs=<huge-number>. > > Implemented Parallel Mtest (r), parallelising where no Mtest has tested > before, using X worker jobs, where X defaults to 1. > > Parallelisation currently is done over directories only (inter-directory > dependencies are hard to track). No more chdirs are performed, so all > arguments are made absolute paths, resulting in some long output paths > here and there, compared to the old behaviour of Mtest. This might require quite a bunch of tests to be adapted accordingly, because so far basically all tests assumed that they were executed in the TSTTRGDIR (though there are tests that "don't care")... ... you already fixed some; but I guess there are (many?) more to do ... Wasn't there a chance to do a per-thread chdir? Or does Python not allow that? Stefan > Please test, enjoy and fix (preferably in parallel ;) ) > > > U Mtest.py.in > Index: Mtest.py.in > =================================================================== > RCS file: /cvsroot/monetdb/testing/src/Mtest.py.in,v > retrieving revision 1.26 > retrieving revision 1.27 > diff -u -d -r1.26 -r1.27 > --- Mtest.py.in 3 Jun 2009 11:42:33 -0000 1.26 > +++ Mtest.py.in 16 Jul 2009 12:54:18 -0000 1.27 > @@ -46,6 +46,9 @@ > import socket > import struct > import select > +from threading import Thread > +import copy > +import string > #import signal > #import getpass > import Mfilter > @@ -107,18 +110,22 @@ > [...1251 lines suppressed...] > + w.join() > + t = w.getTime() > + elem = w.getElem() > t_ = t_ + t > if elem is not None: > body.append(elem) > @@ -3756,7 +3832,7 @@ > if THISFILE == "Mtest.py": > env['TSTDIR'] = "" > env['TSTTRGDIR'] = os.path.join(env['TSTTRGBASE'],TSTPREF) > - CreateHtmlIndex(env, *body) > + CreateHtmlIndex(htmlenv, env, *body) > > Failed = 0 > for f in Failure[1:]: > @@ -3843,3 +3919,4 @@ > > # END > ############################################################################# > +# vim: set ts=4 sw=4 expandtab : > > > ------------------------------------------------------------------------------ > Enter the BlackBerry Developer Challenge > This is your chance to win up to $100,000 in prizes! For a limited time, > vendors submitting new applications to BlackBerry App World(TM) will have > the opportunity to enter the BlackBerry Developer Challenge. See full prize > details at: http://p.sf.net/sfu/Challenge > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 3