October 2006 - developers-list

[Monetdb-developers] Signed/unsigned and NULL
by Fabian Groffen 24 Oct '06

24 Oct '06

I just figured that the following yields in a signed/unsigned warning: 1 void myfunc(char *s) { 2 if (s != NULL) { 3 ... 4 } 5 } The error occurs on line 2. I cannot find the definition of NULL, but I suspect it is defined like: #define NULL 0 Obviously 0 is signed while s (pointer) is not. Maybe we should use #define NULL (char*)0 instead?

3 4

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/common sql_types.mx, 1.72, 1.73
by Martin Kersten 22 Oct '06

22 Oct '06

first measurement Q13 drops to 3.98 sec (and 4.84 in second run) It was 2.17

1 0

[Monetdb-developers] sql_logdir -> logdir -> {sql,xquery}_logdir
by Fabian Groffen 21 Oct '06

21 Oct '06

Hi all, I probably missed something, but after the recent commits some questions popped up: - why sql_logdir -> logdir? - why was this solution not a good idea wich led to a revert and addition of xquery_logdir? Not that I don't agree, but I think it's good to know at least why/what is the problem.

3 2

Re: [Monetdb-developers] strings
by Martin Kersten 20 Oct '06

20 Oct '06

Martin, (et al) Thanks for your comments (Martin's repeated below). For the moment, I settled for a function that ensures no reading beyond the zero byte. The previous proposal did that. It still process two bytes per iteration, though: @- Hash Function The string hash function is a very simple hash function that xors and rotates all characters together. It is optimized to process 2 characters at a time (adding 16-bits to the hash value each iteration). @h #define GDK_STRHASH(x,y) { \ str _c = (str) (x); \ for((y)=0; _c[0] && _c[1]; _s+=2) { \ (y) = ((y) << 5) ^ ((y) >> 7) ^ ((_c[0] << 8) | _c[1]);\ } \ (y) ^= _c[0]; \ } Shared BAT heaps using views could be done. We could modify the view mechanism to allow a VIEW to have its own BUN heap, but still share a heap from a parent. This would allow still only a single parent, though. Touching the view code, however, is quite delicate matter. The suggestion for elimination of all doubles *always* I find not desirable. This policy is what caused crashes I had to go fix in the data center of ABN AMRO. Think of unique string keys -- even if we could dynamically tune the hash table size (we don't) as the BAT grows to avoid long collision lists, we would waste a lot of memory and gain nothing. The current adaptive approach is extremely robust against any distribution thrown at it: - on small bats, the overhead of the (mostly) useless hash table is limited (4KB) - on bats with a 'small' (L2 cachable) heap, the current "working" (=double-eliminating) hash table has just about the right size (256KB) such that things work nice when they can. Out-of-cache hash tables are never nice. - on bats with too many unique values we still profit from local correlations thanks to the repeated flushing. Such distributions would if fully double eliminated, need a (robust!) dynamic allocation scheme, plus a *lot* of memory to keep all the unique values. Not worth it. Thus we have a data structure that requires *no* a-priori distribution knowledge, introduces a limited maximal amount of storage overhead, and reduces storage maximally (given the low resource budget), in the best case achieving full double elimination that can be exploited with ELIMDOUBLES(h). I agree it could be cleaner to remove the hash table fully, and rely on explicit application-driven double elimination. But, as described, this mechanism never hurts us (much), and I guess that for many many use cases, of which we are not even aware now, it actually is delivering us a lot of gain, just because it is pervasively applied. I do not fully grasp your suggestion for hash functions that are distribution-dependent; especially how they would work incrementally, when values are added to a BAT over time. In that case, the distribution would change continually, and thus the hash function.. Peter -----Original Message----- From: Martin Kersten [mailto:Martin.Kersten@cwi.nl] Sent: Friday, October 20, 2006 5:15 PM To: Peter Boncz Cc: Niels Nes; Martin Kersten; Stefan Manegold; Sjoerd Mullender Subject: strings Peter, Niels and I were discussing the snippet shown in the attachment two days ago. The snippet is interesting for several reasons. The most important one is that the cost of this string join comes from inserting 800K strings, *not* the str hash join itself. (observed with callgrind) Amongst others, it triggered (again) the wish to have more control at the BAT level to deal with strings values. For example, it would be nice to know that *all* doubles are eliminated (not only per chunk). In combination with a fast 'uniqueCount' it could make some operations a lot faster. Alternatively, it would be nice if you could refer into the heap of another BAT, e.g. by marking a column as 'shared heap' and re-using the BATview scheme in the target to protect it. (In the 19.sql case, we actually should have dropped the string in the semijoin operation. Calling for either a new gdk operation or a better optimization step.) Wat betreft je voorstel. Ik zou zoeken in de volgende richting. De hash wordt in belangrijke mate bepaald door de statistische kans dat je hetzelfde patroon tegenkomt. Tevens weten we dat in de praktijk het gebruik van characters (en character combinaties) zipfian gedistribueerd is. Dit kan je gebruiken door in de header of de string heap een encoding tabel op te nemen die ervoor zorgt dat nieuwe encoding meer/minder effect heeft op de hash functie. Voorbeeld laat alle veel voorkomende characters weg en neem altijd N minder voorkomende characters. Of gebruik alternatieve bit patronen die de hash Deze codering zou je zelfs als apparte optimalizatie stap op de BAT kunnen loslaten. De default is gebaseerd op een eenvoudige tekst analyse. Verder kan de hash table (nu 1K best wat groter)

1 0

[Monetdb-developers] Compiling buildtools..
by Romulo Goncalves 20 Oct '06

20 Oct '06

Dear developers, I tried three times more and now everything works. It is strange such error. I continue to interest in reason for such error. Best Regards, Romulo Goncalves

2 2

[Monetdb-developers] Compiling buildtools.
by Romulo Goncalves 20 Oct '06

20 Oct '06

I update my buildtools,bootstraap, configure, make and I got this error: /ufs/goncalve/scratch/MonetDB/buildtools/build/_MX/Mx -x C /ufs/goncalve/MonetDB/buildtools/mel/object.mx /ufs/goncalve/MonetDB/buildtools/mel/object.mx: ./object.cc - created if g++ -DHAVE_CONFIG_H -I. -I/ufs/goncalve/MonetDB/buildtools/mel -I. -g -MT object.o -MD -MP -MF ".deps/object.Tpo" -c -o object.o object.cc; \ then mv -f ".deps/object.Tpo" ".deps/object.Po"; else rm -f ".deps/object.Tpo"; exit 1; fi g++: Internal error: Segmentation fault (program cc1plus) Please submit a full bug report. See <URL:http://bugzilla.redhat.com/bugzilla> for instructions. make[2]: *** [object.o] Error 1 make[2]: Leaving directory `/net/amelia/export/scratch1/goncalve/MonetDB/buildtools/build/_MEL' make[1]: *** [all] Error 2 make[1]: Leaving directory `/net/amelia/export/scratch1/goncalve/MonetDB/buildtools/build/_MEL' make: *** [all] Error 2 [goncalve@amelia scripts]$ Any idea? Regards, Romulo

2 1

[Monetdb-developers] Planning the next MonetDB release
by Stefan Manegold 15 Oct '06

15 Oct '06

Dear all, with the recent deadlines passed and the next(?) one (SIGMOD) five weeks ahead, it might be a good time to assess the current status of developments around MonetDB to make more concrete plans for the next release. I guess, we all agree that the next release should be *the* MonetDB/XQuery Update release, i.e., providing an implementation of the W3C XQuery Update Facility (http://www.w3.org/TR/xqupdate/) for MonetDB/XQuery. Hence, the major focus is obviously on finishing this implementation. See details below under "XQuery Updates". Concerning MQX/PF, there are three more subjects/areas where we need to see to which extend they should be officially announced with the release (and hence, need to be "finished" before the release): - algebra path (TUM: JanR, Jens) - recursions (TUM: JanR, Jens) - XRpc (CWI: Jennie, Peter) Further, the 64-bit Windows port of MQX (i.e., pathfinder) is still open. Concerning MonetDB in general, we need to see to which extend we want/need to improve the compilation process before the release: - clean-up configure scripts - clean-up usage of *-config.h include files - "meta scripts" to simplify the cvs checkout/update and build process Additionally, Niels & Fabian probably also have a todo list for MonetDB/SQL. Finally, all parts require some thorough testing --- especially all newly added features, in particular the XQuery update & transaction features and the new multi-document support --- simply using it (like with the recently added (multi-doc) benchmark tests) is a good start... Also, there are 50(!) open bug reports, and for many (most?) bugs that have been filed or closed since 01.01.2006, no test script has been provided, yet, to monitor their status. Lots of work to do... As said, this is to trigger an assessment of the current state and a planning for the release, hence, please feel free to comment and add your points. I suggest to discuss this in (or after) "MADAM" tomorrow and in the PF skype conference on Tuesday. Stefan XQuery Updates ============== As far as I'm aware, the major missing features are transaction support, mainly logging & recovery. Undoubtedly, this is a CWI task. Apart from that, we need to finish/extend the implementation of document management within MonetDB/XQuery --- both in MIL & XQuery. My vision is that we provide comparable (if not equal) functionality in both MIL & XQuery, providing functions beyond the XQuery standard where necessary. The envisaged functionally should cover the following: - access a single document by name (or URI) XQ!: fn:doc($uri as xs:string?) as document-node()? Done. - access all documents in a collection XQ!: fn:collection($arg as xs:string?) as node()* ToDo! - shred a single document into a new "singleton" collection: MIL: shred_doc(str URI [,str doc-name [,str col-name]] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?doc-name:URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. XQ!: fn:doc($uri as xs:string?) as document-node()? (implies doc-name = URI, col-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. - shred a set of documents into a single new collection: MIL: shred_doc(BAT[void,str] URI [,BAT[void,str] doc-name [,str col-name]] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?first-doc-name:first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ!: loop-lifted fn:doc($uri as xs:string?) as document-node()? (implies doc-name = URI, col-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. XQ?: fn:doc($uri as xs:string*) as document-node()? (implies doc-name = URI, col-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo? - shred a set of documents into a set of new collections: MIL: shred_doc(BAT[void,str] URI [,BAT[void,str] doc-name [,BAT[void,str] col-name]] [,BAT[void,lng] freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?doc-name:URI, freeSlotsPerPage = 0 (i.e., read-only)) (constraint: freeSlotsPerPage must be identical for all documents within a collection) Done. XQ?: loop-lifted fn:doc($uri as xs:string*) as document-node()? (implies doc-name = URI, col-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo? - shred a single document and add it to an existing collection: MIL: add_doc(str URI, str col-name [,str doc-name] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: fn:add_doc($uri as xs:string, $coll as xs:string) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - shred a set of documents and add it to a single existing collection: MIL: add_doc(BAT[void,str] URI, str col-name [,BAT[void,str] doc-name] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: loop-lifted fn:add_doc($uri as xs:string, $coll as xs:string) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: fn:add_doc($uri as xs:string*, $coll as xs:string) as document-node()? (implies doc-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - shred a set of documents and add it to a set of existing collections: MIL: add_doc(BAT[void,str] URI, BAT[void,str] col-name [,BAT[void,str] doc-name] [,BAT[void,lng] freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) (constraint: freeSlotsPerPage must be identical for all documents within a collection) ToDo! XQ?: fn:add_doc($uri as xs:string*, $coll as xs:string*) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - delete a single document by name (and the whole collection, if this was the last document in that collection) MIL: delete_doc(str doc-name) : void Done. XQ?: fn:del_doc($uri as xs:string) as ??? ToDo! - dito for a set of documents? - delete a collection MIL: delete_col(str col-name) : void ToDo! XQ?: fn:del_col($col as xs:string) as ??? ToDo! - dito for a set of collections? - delete all cached/persistent documents MIL: delete_all_docs(bit cachedOnly) : void Done. XQ?: fn:del_all_docs($cachedOnly as xs:boolean) as ??? ToDo! - view all explicitly shredded and/or cached documents and collections (at least their names, maybe more info) in the database/cache, a la "xmlcache_print()" & "xmldb_print()" - view/list all documents (names) in a collection - manage cache rules Anything more? -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] shredding loop-lifted fn:doc() into one collection
by Stefan Manegold 15 Oct '06

15 Oct '06

Peter, our fix of Friday night to shred all documents of a loop-lifted fn:doc() call into one single collection (as opposed to creating one collection per document) --- not checked in, yet; see attached "pathfinder.mx.LLdoc.PATCH." for details --- seems to work fine (at least correct; speed/performance needs to be analyzed in detail) for file-URLs, i.e., documents that get indeed cached persistently). However, for non-file URLs (i.e., document that do get cached and hence stay trnasient), it did/does not work, basically as the docuemnts are shredded into the "transient" collection, and are (hence?) not assigned unique document IDs --- the document do not end up in the doc_* BATs. A HACK(?) to use the (unified) temporary document-/collection-ID for all documents (see attached "pathfinder.mx.LLdoc.FIX.") seems to work --- however, I have no idea, whether this is what was indended... More "live" on Monday. Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Problems with SQL module...
by Romulo Goncalves 12 Oct '06

12 Oct '06

Dear all, I founded some problems in SQL module. Maybe they are not problems just a misunderstood. To see the problems in your computer just start a mserver5, load sql module, and sql.start(); Then start a mclient -lsql. First Problem: Execute this two queries in the mclient: CREATE FUNCTION zero(id int) RETURNS int BEGIN IF id > 0 THEN RETURN 1; ELSE RETURN 0; END IF; END; CREATE FUNCTION TEST() RETURNS BOOLEAN BEGIN DECLARE x int; SET x = zero(1); IF x > 0 THEN RETURN TRUE; ELSE RETURN FALSE; END IF; END; UNTIL now you should not find any problems. Lets see the first problem: In the mclient execute: SELECT test(); I got this error: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1098918240 (LWP 14589)] 0x00002aaaaaaba2c1 in idcmp (n=0x2aaab485227b "destroy", m=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/mal/mal_type.mx:251 251 if (* n == * m) (gdb) up #1 0x00002aaab483ab94 in isUpdated (mb=0xf59548, pc=9) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_support.mx:885 885 if (idcmp("destroy", getFunctionId(q)) = = 0) (gdb) up #2 0x00002aaab483f287 in OPTcommonTermsImplementation (mb=0xf59548, stk=0x0, pci=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_commonTerms.mx:17 6 176 if (hasSameSignature(p, q) && (gdb) up #3 0x00002aaab483f718 in OPTcommonTerms (mb=0xf59548, stk=0x0, p=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_commonTerms.mx:24 9 Line number 249 out of range; /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimize r/opt_commonTerms.mx has 220 lines. (gdb) up #4 0x00002aaab4efb73a in addQueryToCache (c=0x503a10) at /ufs/goncalve/MonetDB/SQL/sql/src/backends/monet5/sql_optimizer.mx:434 434 OPTcommonTerms(mb,0,0); /* common terms */ (gdb) up #5 0x00002aaab4efa84e in backend_create_func (be=0x10e1218, f=0x10e80b8) at /ufs/goncalve/MonetDB/SQL/sql/src/backends/monet5/sql_gencode.mx:1938 1938 addQueryToCache(c); If you want all the details, run your mserver with gdb (I know that you know the procedures for debugging). ;) Second problem: If I restart the mclient and restart the mserver: The mclient can see the functions in the table sys.functions, but the client started in the server by sql.start() does see the functions created. Third problem: Just execute the following query: select * from functions where name like 'hash'; Why they are so many rows with the same information for the function hash? Fourth problem: Just create the following function: CREATE FUNCTION niels(id_val int) RETURNS TABLE (id int) BEGIN DECLARE TABLE roro(id int); RETURN TABLE (SELECT id from functions where id = id_val order by id); END; I will try to correct this bugs, but until there I will appreciate some help. This problems are delaying the translation of Skyserver functions and also the my implementation of Dependency Control table. Best Regards, Romulo Goncalves

3 2

[Monetdb-developers] re-organizing PF/MXQ tests
by Stefan Manegold 12 Oct '06

12 Oct '06

Dear all, before adding another ~80 tests (all 4 flavors of the XBench benchmark suite) to the current pathfinder test set of 1384 tests, I'd like to propose a re-organization/re-structuring of the pathfinder test set. With all tests enabled, testing takes almost 30 minutes --- with an optimized 64-bit binary with 32-bit OIDs on my Athlon64 X2 3800+ (2GHz) with 2GB RAM. A quick analysis clusters the tests in 5 categories: 1) compiler ("pf" only) tests 2) tests for features provided by the MXQ runtime 3) tests for additional/optional modules (PFtijah) 4) tests derived from bug reports, use cases, etc. 5) benchmark tests Currently, the respective tests reside in the follwoing directory trees: 1) compiler// 2) runtime// 3) modules// 4) tests// 5) tests// This allows to conveniently run each category individually (e.g., `Mtest -r compiler/`), except from the "tests/" tree, which contains two categories: 4 & 5. I propose to replace 5) tests// by 5) benchmarks// i.e., add "benchmarks/" and move all benchmark tests from tests// to benchmarks//, basically: mv tests/{MBench,X007,XBench,XMach-1,XMark,XPath,XPathMark} benchmarks/ Additionally, I'd suggest to also move the tests/XRpc/xmark_queries to benchmarks/XMark/XRpc Open question: should we then also move tests/Music to benchmarks/Music ? Votes? Well, as you might imagine, I'v everything ready here for checkin ;-) Hence, unless there are complains or different suggestions by Saturday (Oct 14 2006) morning, I'll apply above changes during the weekend. Kind regards, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0