- developers-list - monetdb.org

[Monetdb-developers] need help in monetdb xquery interface
by anshul gangwar 05 Mar '10

05 Mar '10

hello , I want to know whether there is any expression or command or anything equivalent to explain sql construct. Also What is the way to get histogram statistics i.e way to find that histogram exists on attribute or not through Xquery interface. not through sql . Please reply as soon as possible. thank you. The INTERNET now has a personality. YOURS! See your Yahoo! Homepage. http://in.yahoo.com/

1 0

[Monetdb-developers] Announcement: New Feb2010 feature release of MonetDB suite
by Sjoerd Mullender 02 Mar '10

02 Mar '10

The MonetDB team at CWI/MonetDB BV is pleased to announce the Feb2010 feature release of the MonetDB suite of programs. More information (including release notes) on this release is available at <http://monetdb.cwi.nl/Development/Releases/Feb2010/>. -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/mal mal_interpreter.mx, Feb2010, 1.337.2.33, 1.337.2.34
by Stefan Manegold 01 Mar '10

01 Mar '10

On Mon, Mar 01, 2010 at 10:20:01AM +0100, Sjoerd Mullender wrote: > Note that this particular initialization doesn't do anything. Static > variables are automatically initialized to 0. Thanks! --- I overlooked that one --- then the rather high value of hottop (600037897) I encountered was indeed only cause by incrementing it "unboundedly" rather than an "unexpected" initialization ... > Also note that in the HEAD branch, hottop is initialized to MAXHOT > (causing a propagation conflict). I indeed did not check the HEAD branch --- I cannot tell which is supposed to be the "correct" initial value, 0 or MAXHOT --- maybe Martin can help ... Stefan > On 2010-03-01 09:27, Stefan Manegold wrote: > > Update of /cvsroot/monetdb/MonetDB5/src/mal > > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv17757/MonetDB5/src/mal > > > > Modified Files: > > Tag: Feb2010 > > mal_interpreter.mx > > Log Message: > > > > make sure we initialize the static global variable hottop at least once; > > I hope, 0 is the correct initial value. > > > > > > Index: mal_interpreter.mx > > =================================================================== > > RCS file: /cvsroot/monetdb/MonetDB5/src/mal/mal_interpreter.mx,v > > retrieving revision 1.337.2.33 > > retrieving revision 1.337.2.34 > > diff -u -d -r1.337.2.33 -r1.337.2.34 > > --- mal_interpreter.mx 28 Feb 2010 18:49:24 -0000 1.337.2.33 > > +++ mal_interpreter.mx 1 Mar 2010 08:27:04 -0000 1.337.2.34 > > @@ -89,7 +89,7 @@ > > lng claim; /* actual claim on memory*/ > > int bid; > > } hotpotatoes[MAXHOT]; > > -static int hottop; > > +static int hottop = 0; > > @- > > The struct alignment leads to 40% gain in simple instructions when set. > > @c > > > > > > ------------------------------------------------------------------------------ > > Download Intel® Parallel Studio Eval > > Try the new software tools for yourself. Speed compiling, find bugs > > proactively, and fine-tune applications for parallel performance. > > See why Intel Parallel Studio got high marks during beta. > > http://p.sf.net/sfu/intel-sw-dev > > _______________________________________________ > > Monetdb-checkins mailing list > > Monetdb-checkins(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > > -- > Sjoerd Mullender > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |

1 0

[Monetdb-developers] Mx segfaults during MonetDB build
by Mark Bucciarelli 01 Mar '10

01 Mar '10

Hi, I'm getting a segfault when running Mx, in buildtools/Mx/Code.c, CodeSubBlk(). The symptom is around line 305---s[-1] is not accessible memory. Any suggestions for how to debug/fix this? I got as far as looking at def in the parent frame (CodeSub()): (gdb) print *def $9 = { d_dir = Mxmacro, d_cmd = 0x8592e110 "DEFseqscan", d_blk = 0x89ca2000 "@:DEFseqscan_@[10](@1,@2,@3,@4,@5,@6,@7,@8,@9)@\n", d_mod = 0, d_sec = 0, d_cms = 0xd0d0d0d0 <Address 0xd0d0d0d0 out of bounds>, d_file = 0x86c3a080 "gdk_scanselect_defs.mx", d_line = 132 } (gdb) This is the Feb2010 branch, on OpenBSD 4.6 with the following malloc.conf options: A ``Abort''. malloc() will coredump the process, rather than tol- erate internal inconsistencies or incorrect usage. This is the default and a very handy debugging aid, since the core file rep- resents the time of failure, rather than when the bogus pointer was used. G ``Guard''. Enable guard pages. Each page size or larger alloca- tion is followed by a guard page that will cause a segmentation fault upon any access. J ``Junk''. Fill some junk into the area allocated. Currently junk is bytes of 0xd0 when allocating; this is pronounced ``Duh''. :-) Freed chunks are filled with 0xdf. P ``Move allocations within a page.'' Allocations larger than half a page but smaller than a page are aligned to the end of a page to catch buffer overruns in more cases. This is the default. I was getting these when building Nov2009, but I was able to get past them by restarting the build multiple times. I'd like to fix it this time around. Fortunately, after a gmake clean the crash is reproduceable. Thanks, m

2 1

[Monetdb-developers] pf:new-doc
by Tinco Andringa 26 Feb '10

26 Feb '10

Hello MonetDB developers, In my quest to develop an object oriented interface to monetdb/xquery on ruby using datamapper, I've stumbled upon the following problem. It is at this moment impossible to add a document to monetdb that does not have a source outside monetdb. The only function to add documents to monetdb seems to be pf:add-doc, which takes an url parameter with a location of an xml document it should import. To fully abstract the database from the ruby application, it would be nice if the developers could add a new (empty) document to the database. Would it be a nice idea to have the following function: pf:new-doc("document_name.xml"[, "collection_name.xml"[,"<some><initial>content</initial></some>"]]) It would make users of my api a little happier, and perhaps MonetDB/XQuery as a whole, a bit more normal ;) Kind regards, Tinco Andringa

1 1

[Monetdb-developers] SQL concurrency/serialization
by Isidor Zeuner 25 Feb '10

25 Feb '10

Dear MonetDB developers, on concurrently running writing transactions, I am getting a rather narrowly descriptive error message: "!SQLException:sql.trans:COMMIT: failed". My first attempts to deal with this issue suggest that it happens exactly in those cases where the concurrent writes create some sort of conflict. I am aware that MonetDB does not focus on concurrency, and of the suggestion to serialize multiple writers. However, I'd like to know what the current status of concurrency is. To what extent are SQL primitives implemented which allow to let the server take care of serializing writes, for example through explicit table locking or transaction isolation? Best regards, Isidor Zeuner

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/optimizer opt_mapreduce.mx, , 1.13, 1.14
by Martin Kersten 24 Feb '10

24 Feb '10

Fabian wrote: > Update of /cvsroot/monetdb/MonetDB5/src/optimizer > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv10474 > > Modified Files: > opt_mapreduce.mx > Log Message: > One of the optimisers 'inline,remap,evaluate,costModel,coercions,emptySet,aliases' replaces a nil statement in such a way that it ends up as argument 'nil' witout type, causing execution to fail. Push barrier up, such that we don't get undefined var errors because the reduce phase was performed in the dataflow block too. You can freeze the type of variables using setVarUDFtype. As a line of defense, at the end of an optimizer all non-UDFtypes are thrown away and a fresh type resolve step is taken.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13, 1.13.2.1
by Sjoerd Mullender 23 Feb '10

23 Feb '10

Are you sure *all* these changes are for disabling tokenizer? Including the property changes? None of this will now be propagated, so if any of this should land in the HEAD, you need to do it manually (but then please identically). On 2010-02-22 16:43, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv32552/src/modules/mal/rdf > > Modified Files: > Tag: Feb2010 > rdf_shredder.mx > Log Message: > disable the tokenizer.mx from m5 > > > Index: rdf_shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v > retrieving revision 1.13 > retrieving revision 1.13.2.1 > diff -u -d -r1.13 -r1.13.2.1 > --- rdf_shredder.mx 8 Jan 2010 12:08:58 -0000 1.13 > +++ rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 > @@ -27,7 +27,9 @@ > #include "url.h" > #include "rdf.h" > #include "raptor.h" > +#if 0 > #include "../tokenizer.h" > +#endif > > typedef struct graphBATdef { > graphBATType batType; /* BAT type */ > @@ -36,6 +38,8 @@ > int tailType; /* type of right column */ > } graphBATdef; > > +static BUN batsz = 10000000; > + > /* this list should be kept alligned with the graphBATType enum */ > #if STORE == TRIPLE_STORE > static graphBATdef graphdef[N_GRAPH_BAT] = { > @@ -132,7 +136,8 @@ > @:rdf_BUNappend_unq(@1, @2)@ > #endif > > -@= rdf_BUNappend_unq > + > +@= rdf_BUNappend_unq_1 > bun = BUNfnd(BATmirror(@1),(ptr)@2); > if (bun == BUN_NONE) { > if (BATcount(@1) > 4 * @1->T->hash->mask) { > @@ -148,6 +153,20 @@ > bun = (@1)->hseqbase + bun; > } > > +@= rdf_BUNappend_unq > +bun = BUNfnd(BATmirror(@1),(ptr)@2); > +if (bun == BUN_NONE) { > + if (BATcount(@1) > 4 * @1->T->hash->mask) { > + HASHdestroy(@1); > + BAThash(BATmirror(@1), 2*BATcount(@1)); > + } > + bun = (BUN) @1->batCount; > + @1 = BUNappend(@1, (ptr)@2, TRUE); > + if (@1 == NULL) { > + @:raptor_exception(pdata, "could not append in@1")@ > + } > +} > + > @= rdf_tknzr_insert > { > str t = @1; > @@ -226,12 +245,9 @@ > > /* disable all properties */ > b->tsorted = FALSE; > - b->T->nosorted = 0; > b->tdense = FALSE; > - b->T->nodense = 0; > b->tkey = FALSE; > - b->T->nokey[0] = 0; > - b->T->nokey[1] = 1; > + b->hdense = TRUE; > > return b; > } > @@ -264,7 +280,7 @@ > pdata->graph[i] = create_BAT ( > graphdef[i].headType, > graphdef[i].tailType, > - BATTINY); /* DOTO: estimate size */ > + batsz); /* DOTO: estimate size */ > if (pdata->graph[i] == NULL) { > return NULL; > } > @@ -274,7 +290,7 @@ > pdata->graph[MAP_LEX] = create_BAT ( > graphdef[MAP_LEX].headType, > graphdef[MAP_LEX].tailType, > - BATTINY); /* DOTO: estimate size */ > + batsz); /* DOTO: estimate size */ > if (pdata->graph[MAP_LEX] == NULL) { > return NULL; > } > @@ -346,10 +362,10 @@ > static str > post_processing (parserData *pdata) > { > -#if STORE == TRIPLE_STORE > BUN cnt; > BAT *map_oid = NULL, *S = NULL, *P = NULL, *O = NULL, *ctref= NULL; > BAT **graph = pdata->graph; > +#ifdef _TKNZR_H > BATiter bi, mi; > BUN p, d, r; > oid *bt; > @@ -379,7 +395,31 @@ > P = graph[P_sort]; > O = graph[O_sort]; > cnt = BATcount(S); > +#else > + /* order MAP_LEX */ > + BATorder(BATmirror(graph[MAP_LEX])); > + map_oid = BATmark(graph[MAP_LEX], 0); /* BATmark will create a copy */ > + BATorder(map_oid); > + BATsetaccess(map_oid, BAT_READ); /* force BAtmark not to copy bat */ > + map_oid = BATmirror(BATmark(BATmirror(map_oid), 0)); > + BATsetaccess(graph[MAP_LEX], BAT_READ); /* force BATmark not to copy bat */ > + graph[MAP_LEX] = BATmirror(BATmark(BATmirror(graph[MAP_LEX]), 0)); > + > + /* convert old oids of S_sort, P_sort, O_sort to new ones */ > + cnt = BATcount(graph[S_sort]); > + S = BATleftfetchjoin(graph[S_sort], map_oid, cnt); > + if (S == NULL) goto bailout; > + BBPreclaim(graph[S_sort]); > + P = BATleftfetchjoin(graph[P_sort], map_oid, cnt); > + if (P == NULL) goto bailout; > + BBPreclaim(graph[P_sort]); > + O = BATleftfetchjoin(graph[O_sort], map_oid, cnt); > + if (O == NULL) goto bailout; > + BBPreclaim(graph[O_sort]); > + BBPreclaim(map_oid); > +#endif > > +#if STORE == TRIPLE_STORE > /* order SPO/SOP */ > graph[S_sort] = BATmirror(BATsort(BATmirror(S))); /* sort on S */ > @:order(graph[S_sort],P,O,PO)@ > @@ -408,6 +448,14 @@ > > return MAL_SUCCEED; > > +#elif STORE == MLA_STORE > + graph[S_sort] = S; > + graph[P_sort] = P; > + graph[O_sort] = O; > + > + return MAL_SUCCEED; > +#endif > + > bailout: > if (map_oid != NULL) BBPreclaim(map_oid); > if (ctref != NULL) BBPreclaim(ctref); > @@ -415,11 +463,6 @@ > if (P != NULL) BBPreclaim(P); > if (O != NULL) BBPreclaim(O); > return NULL; > - > -#elif STORE == MLA_STORE > - (void) pdata; > - return MAL_SUCCEED; > -#endif > } > > @= clean_raptor > @@ -455,15 +498,21 @@ > (void) graphname; > > /* init tokenizer */ > +#ifdef _TKNZR_H > if (TKNZRopen (NULL, schema) != MAL_SUCCEED) { > throw(RDF, "rdf.rdfShred", > "could not open the tokenizer\n"); > } > +#else > + (void) schema; > +#endif > > /* Init pdata */ > pdata = parserData_create(*location); > if (pdata == NULL) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > @:clean@ > throw(RDF, "rdf.rdfShred", > "could not allocate enough memory for pdata\n"); > @@ -473,7 +522,9 @@ > raptor_init(); > pdata->rparser = rparser = raptor_new_parser("guess"); > if (rparser == NULL) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > raptor_finish(); > @:clean@ > throw(RDF, "rdf.rdfShred", "could not create raptor parser object\n"); > @@ -484,7 +535,9 @@ > /* Parse URI or local file. */ > ret = URLisaURL(&isURI, location); > if (ret != MAL_SUCCEED) { > +#ifdef _TKNZR_H > TKNZRclose(&iret); > +#endif > @:clean@ > return ret; > } else if (isURI) { > @@ -496,6 +549,9 @@ > iret = raptor_parse_file(rparser, uri, NULL); > } > @:clean_raptor@ > +#ifdef _TKNZR_H > + TKNZRclose(&iret); > +#endif > > graph = pdata->graph; > assert (pdata->tcount == BATcount(graph[S_sort]) && > @@ -504,23 +560,18 @@ > > /* error check */ > if (iret) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", "parsing failed\n"); > } > if (pdata->exception) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "%s\n", pdata->exceptionMsg); > } else if (pdata->fatal) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last fatal error was:\n%s\n", > pdata->fatalMsg); > } else if (pdata->error) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last error was:\n%s\n", > pdata->errorMsg); > } else if (pdata->warning) { > - TKNZRclose(&iret); > throw(RDF, "rdf.rdfShred", "last warning was:\n%s\n", > pdata->warningMsg); > } > @@ -528,7 +579,6 @@ > /* post processing step */ > ret = post_processing(pdata); > if (ret != MAL_SUCCEED) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", "could not post-proccess data"); > } > @@ -536,7 +586,6 @@ > /* prepare return bat of bats */ > retbat = BATnew(TYPE_void, TYPE_bat, N_GRAPH_BAT); > if (retbat == NULL) { > - TKNZRclose(&iret); > @:clean@ > throw(RDF, "rdf.rdfShred", > "could not allocate enough memory for return bat"); > @@ -548,7 +597,6 @@ > } > > GDKfree(pdata); > - TKNZRclose(&iret); > BBPkeepref(*retval = retbat->batCacheid); > > return MAL_SUCCEED; > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13.2.1, 1.13.2.2
by Sjoerd Mullender 22 Feb '10

22 Feb '10

Propagation prevented. On 2010-02-22 17:24, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf > In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv8115/rdf > > Modified Files: > Tag: Feb2010 > rdf_shredder.mx > Log Message: > remove any files and code related to the tokenizer from the stable branch. These changes should not be propagated to the head > > > Index: rdf_shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v > retrieving revision 1.13.2.1 > retrieving revision 1.13.2.2 > diff -u -d -r1.13.2.1 -r1.13.2.2 > --- rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 > +++ rdf_shredder.mx 22 Feb 2010 16:24:23 -0000 1.13.2.2 > @@ -27,9 +27,6 @@ > #include "url.h" > #include "rdf.h" > #include "raptor.h" > -#if 0 > -#include "../tokenizer.h" > -#endif > > typedef struct graphBATdef { > graphBATType batType; /* BAT type */ > > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal/rdf rdf_shredder.mx, Feb2010, 1.13, 1.13.2.1
by Lefteris 22 Feb '10

22 Feb '10

the tokenizer has been removed from the Makefile.ag of the stable branch, and this #include can be deleted all the way. I have said many times that none of this code is ready for stable branches and yes they should have never been propagate to the stable. We haven't announced any MonetDB/RDF module official so it should not be used by any user. For me the #if 0 is not a problem On Mon, Feb 22, 2010 at 4:46 PM, Fabian Groffen <Fabian.Groffen(a)cwi.nl> wrote: > On 22-02-2010 15:43:44 +0000, Lefteris Sidirourgos wrote: >> Update of /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf >> In directory sfp-cvsdas-1.v30.ch3.sourceforge.com:/tmp/cvs-serv32552/src/modules/mal/rdf >> >> Modified Files: >> Tag: Feb2010 >> rdf_shredder.mx >> Log Message: >> disable the tokenizer.mx from m5 > > Dunno if this was/is discussed, but isn't it a much cleaner solution to > just remove it from the code base in this branch, and keep it as is in > the HEAD (e.g. request to prevent propagation?). The 'if 0's and > commented out code look a bit messy. > > >> Index: rdf_shredder.mx >> =================================================================== >> RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/rdf/rdf_shredder.mx,v >> retrieving revision 1.13 >> retrieving revision 1.13.2.1 >> diff -u -d -r1.13 -r1.13.2.1 >> --- rdf_shredder.mx 8 Jan 2010 12:08:58 -0000 1.13 >> +++ rdf_shredder.mx 22 Feb 2010 15:43:41 -0000 1.13.2.1 >> @@ -27,7 +27,9 @@ >> #include "url.h" >> #include "rdf.h" >> #include "raptor.h" >> +#if 0 >> #include "../tokenizer.h" >> +#endif > > ------------------------------------------------------------------------------ > Download Intel® Parallel Studio Eval > Try the new software tools for yourself. Speed compiling, find bugs > proactively, and fine-tune applications for parallel performance. > See why Intel Parallel Studio got high marks during beta. > http://p.sf.net/sfu/intel-sw-dev > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins >

1 0