- developers-list - monetdb.org

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/semantics xquery_fo.c, 1.83, 1.84
by Jan Rittinger 26 Oct '06

26 Oct '06

Do you really mean string as return type of fn:nid? On 10/26/2006 02:12 PM, Peter Boncz wrote with possible deletions: > Update of /cvsroot/monetdb/pathfinder/compiler/semantics > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv8300/compiler/semantics > > Modified Files: > xquery_fo.c > Log Message: > fast NID node retriveal by NID > - moved support for id/idref node retrieval into pathfinder.mx > having this as an API function will allow us to provide this functionality > using indices later (i.e. be able to lock and isolte a hash index). > it also helps a port of this functionality to algebra. > > for the moment, there are no shared hash tables, and ID/IDREF queries > on updatable XML documents will be slowish (sequential scans needed) > the new NID access is a good alternative to look up nodes. > > - implemented in ID node retrieval support for string sthat are numbers, > these are interpreted as NIDs, not ID attributes. > > - implemented a new fn:nid(element()*) as xs:string* builtin function > that returns the NIDs of element nodes. > > - fixed some constant support things (mposjoin) > > > > Index: xquery_fo.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/semantics/xquery_fo.c,v > retrieving revision 1.83 > retrieving revision 1.84 > diff -u -d -r1.83 -r1.84 > --- xquery_fo.c 19 Oct 2006 19:32:27 -0000 1.83 > +++ xquery_fo.c 26 Oct 2006 12:12:40 -0000 1.84 > @@ -184,6 +184,11 @@ > .par_ty = (PFty_t[]) { PFty_opt (PFty_xs_string ()) }, > .ret_ty = PFty_opt (PFty_doc (PFty_xs_anyNode ())) } }, > .alg = PFbui_fn_doc } > + , /* fn:idref (xs:element) as integer */ > + { .ns = PFns_fn, .loc = "nid", > + .arity = 1, .sig_count = 1, .sigs = { { > + .par_ty = (PFty_t[]) { PFty_xs_anyElement () }, > + .ret_ty = PFty_xs_string () } } } > , /* fn:id (string*) as element* */ > { .ns = PFns_fn, .loc = "id", > .arity = 1, .sig_count = 1, .sigs = { { > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > -- Jan Rittinger Database Systems Technische Universität München (Germany) http://www-db.in.tum.de/~rittinge/

1 0

[Monetdb-developers] Upcoming changes in the type system affect existing code!
by Jens Teubner 26 Oct '06

26 Oct '06

Pathfinder folks, I am currently busy rewriting Pathfinder's type system. In the short-term view, this rewrite will bring us a much cleaner and more consistent representation of types, and I'm quite confident that there will be less bugs in the new code; I have actually found quite a few in the existing code while doing the rewrite. Most importantly, however, on a longer term view, my rewrites pave the ground to implement also *dynamic* typing in Pathfinder in an elegant and standards-compliant manner. The rewrite will also change the internal representation of types in Pathfinder. Most of the current built-in primitive types will become *named* types, in line with the XQuery specs. Most importantly, this makes Pathfinder highly flexible and easily extendible with respect to types. Unfortunately, there's a bunch of code flying around in the project that inspects the (current) internal type representation in a deprecated manner. My changes *will* break such code. This mainly affects code that looks into the enum field PFty_t.type, a field that is not (and has never been) supposed to be inspected by code outside types.c and subtyping.c. milprint_summer.c in particular is full of such references. ONCE I'M DONE WITH MY REWRITES, SUCH CODE *WILL* BREAK. MY NEXT CHECKIN WILL TAG AFFECTED CODE IN MILPRINT_SUMMER.C WITH #ifdef'S. WHOEVER FEELS RESPONSIBLE FOR THIS ABUSE OF THE CURRENT TYPE REPRESENTATION OR WANTS TO RESCUE THE CODE IN milprint_summer.c FROM DYING CAN USE THESE #ifdef'S TO FIND THE APPROPRIATE PLACES AND FIX THE RESPECTIVE CODE. I WILL *NOT* FIX THE AFFECTED CODE MYSELF. The "official" way to inspect Pathfinder's types is to use the functions provided by types.c and subtyping.c. The subtype check PFty_subtype() is the function that allows most of the checks against types that one ever needs. Additional helpers can be PFty_prime() (which normalizes a structured type to have the form `t1 | t2 | t3 | ...') or PFty_quantifier() (determining the quantifier, `0', `1', `?', `+', or `*', of a given type). The test whether a type t is the type `xs:string' may, e.g., implemented by testing PFty_subtype (t, PFty_xs_string ()). (Some care has to be taken, though, that this is a *subtype* relationship. Often, this is what you actually need. If you want to test for type *equivalence*, use PFty_equality(), which applies the subtype test in both directions.) Quantifiers are usually tested with a subtype test against a regular expression over the `item' type. PFty_subtype (t, PFty_item ()), e.g., tests whether the quantifier of t is exactly one. You may want to look into other pieces of our code, e.g., core2alg.brg, to see how PFty_subtype() and others can be used to test types. Just watch out for invocations of PFty_subtype(). If you need any further assistance regarding types, just contact me. Jens -- Jens Teubner Technische Universitaet Muenchen, Department of Informatics D-85748 Garching, Germany Tel: +49 89 289-17259 Fax: +49 89 289-17263 Unix is user friendly - it's just picky about it's friends.

1 0

[Monetdb-developers] mclient does not close its session.
by Romulo Goncalves 25 Oct '06

25 Oct '06

Dear developers, I start a mserver5, I load the sql module, and I start a sql client with sql.start() in the mserver prompt. In a new shell I start an mclient and I load a sql file to create a user and a schema within one transaction: mclient < create_user.sql After to run this command line the client never closes its session. I check the mserver system tables and the user and the schema were created. This is the file loaded in mclient: START TRANSACTION; CREATE USER "skyserver" WITH PASSWORD 'skyserver' NAME 'SkyServer' SCHEMA "sys";CREATE SCHEMA "skyserver" AUTHORIZATION "skyserver"; ALTER USER "skyserver" SET SCHEMA "skyserver"; COMMIT; Any idea? Regards, Romulo

2 1

[Monetdb-developers] [SQL] Header line change, support for warnings/info
by Fabian Groffen 24 Oct '06

24 Oct '06

I have a prepared a change for SQL's meta-data headers to use a different symbol than the widely accepted hash (#) used for comments in MonetDB. Instead, the percentage sign (%) is used to indicate a structured meta-data header. This change allows JDBC to be strict in its parsing, while it also supports comments/info/warnings over the protocol. Mapi has also been patched to support this. An additional advantage of this change is that meta-data no longer is considered a comment by Mtest; as such changes in this meta-data (which usually upset or change the behaviour of JDBC) are immediately visible, even though there is no JDBC test available. In the changes implemented one or multiple comment lines can follow after a regular Response, just before the prompt or next start of header. I changed the signatures of the functions that create the Responses such that they can take an extra string argument "w" (to comply with Niels' coding style) which represents a warning message. This string may contain newlines, in which case the string is properly send as multiple comment lines over the stream. Before I do this massive checkin (lots of mapproves in SQL) I prefer to get a go/no-go first. In case of a no-go, I will check in my changes on a branch.

2 2

Re: [Monetdb-developers] [Monetdb-checkins] buildtools/mel mel.mx, 1.11, 1.12
by Stefan Manegold 24 Oct '06

24 Oct '06

Niels, all our testing machines have flex 2.5.4, except the Gentoo machine (pegatoo) and the Debian machine (core-1); both use flex 2.5.33 (default on Gentoo, from /soft on Debian where 2.5.31 is default) --- and now fail to compile buildtools (mel) with " if g++ -DHAVE_CONFIG_H -I. -I/var/tmp/monet.GNU.32.32.d.14085/buildtools/mel -I. -g -MT mel.yy.o -MD -MP -MF ".deps/mel.yy.Tpo" -c -o mel.yy.o mel.yy.cc; \ then mv -f ".deps/mel.yy.Tpo" ".deps/mel.yy.Po"; else rm -f ".deps/mel.yy.Tpo"; exit 1; fi mel.ll:21: error: redefinition of ‘int yylineno’ lex.yy.c:346: error: ‘int yylineno’ previously defined here make[1]: *** [mel.yy.o] Error 1 " (cf. http://monetdb.cwi.nl/testing/projects/monetdb/Current/buildtools/.GNU.64.6… http://monetdb.cwi.nl/testing/projects/monetdb/Current/buildtools/.GNU.32.3… ) Did you check with flex 2.5.33 on some other machine? Which one? And in combination with which bison? I recall that there was a problem with an undefined yylineno when using flex 2.5.33 & bison 2.3 on openSUSE 10.2 *Alpha* (as reported by Riham), but was not able to reproduce it on our machines with flex 2.5.33 and bison 2.2 (gentoo) or bison 1.875d (debian). Stefan On Mon, Oct 23, 2006 at 07:16:26PM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/buildtools/mel > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv24999/mel > > Modified Files: > mel.mx > Log Message: > only flex 2.5.5 to 2.5.31 define yylineno (well we are I think only sure > about 2.5.31). <= 2.5.4 and >= 2.5.33 surely don't define yylineno. > > > > Index: mel.mx > =================================================================== > RCS file: /cvsroot/monetdb/buildtools/mel/mel.mx,v > retrieving revision 1.11 > retrieving revision 1.12 > diff -u -d -r1.11 -r1.12 > --- mel.mx 29 Aug 2006 10:38:43 -0000 1.11 > +++ mel.mx 23 Oct 2006 19:16:24 -0000 1.12 > @@ -506,7 +506,7 @@ > extern yystype yylval; > > #ifdef FLEX_SCANNER > -#if !defined(YY_FLEX_SUBMINOR_VERSION) || YY_FLEX_SUBMINOR_VERSION < 5 > +#if !defined(YY_FLEX_SUBMINOR_VERSION) || YY_FLEX_SUBMINOR_VERSION < 5 || YY_FLEX_SUBMINOR_VERSION > 31 > /* flex 2.5.31 declares yylineno. I don't know which other versions > do as well. */ > int yylineno; > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 4

[Monetdb-developers] Signed/unsigned and NULL
by Fabian Groffen 24 Oct '06

24 Oct '06

I just figured that the following yields in a signed/unsigned warning: 1 void myfunc(char *s) { 2 if (s != NULL) { 3 ... 4 } 5 } The error occurs on line 2. I cannot find the definition of NULL, but I suspect it is defined like: #define NULL 0 Obviously 0 is signed while s (pointer) is not. Maybe we should use #define NULL (char*)0 instead?

3 4

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/common sql_types.mx, 1.72, 1.73
by Martin Kersten 22 Oct '06

22 Oct '06

first measurement Q13 drops to 3.98 sec (and 4.84 in second run) It was 2.17

1 0

[Monetdb-developers] sql_logdir -> logdir -> {sql,xquery}_logdir
by Fabian Groffen 21 Oct '06

21 Oct '06

Hi all, I probably missed something, but after the recent commits some questions popped up: - why sql_logdir -> logdir? - why was this solution not a good idea wich led to a revert and addition of xquery_logdir? Not that I don't agree, but I think it's good to know at least why/what is the problem.

3 2

Re: [Monetdb-developers] strings
by Martin Kersten 20 Oct '06

20 Oct '06

Martin, (et al) Thanks for your comments (Martin's repeated below). For the moment, I settled for a function that ensures no reading beyond the zero byte. The previous proposal did that. It still process two bytes per iteration, though: @- Hash Function The string hash function is a very simple hash function that xors and rotates all characters together. It is optimized to process 2 characters at a time (adding 16-bits to the hash value each iteration). @h #define GDK_STRHASH(x,y) { \ str _c = (str) (x); \ for((y)=0; _c[0] && _c[1]; _s+=2) { \ (y) = ((y) << 5) ^ ((y) >> 7) ^ ((_c[0] << 8) | _c[1]);\ } \ (y) ^= _c[0]; \ } Shared BAT heaps using views could be done. We could modify the view mechanism to allow a VIEW to have its own BUN heap, but still share a heap from a parent. This would allow still only a single parent, though. Touching the view code, however, is quite delicate matter. The suggestion for elimination of all doubles *always* I find not desirable. This policy is what caused crashes I had to go fix in the data center of ABN AMRO. Think of unique string keys -- even if we could dynamically tune the hash table size (we don't) as the BAT grows to avoid long collision lists, we would waste a lot of memory and gain nothing. The current adaptive approach is extremely robust against any distribution thrown at it: - on small bats, the overhead of the (mostly) useless hash table is limited (4KB) - on bats with a 'small' (L2 cachable) heap, the current "working" (=double-eliminating) hash table has just about the right size (256KB) such that things work nice when they can. Out-of-cache hash tables are never nice. - on bats with too many unique values we still profit from local correlations thanks to the repeated flushing. Such distributions would if fully double eliminated, need a (robust!) dynamic allocation scheme, plus a *lot* of memory to keep all the unique values. Not worth it. Thus we have a data structure that requires *no* a-priori distribution knowledge, introduces a limited maximal amount of storage overhead, and reduces storage maximally (given the low resource budget), in the best case achieving full double elimination that can be exploited with ELIMDOUBLES(h). I agree it could be cleaner to remove the hash table fully, and rely on explicit application-driven double elimination. But, as described, this mechanism never hurts us (much), and I guess that for many many use cases, of which we are not even aware now, it actually is delivering us a lot of gain, just because it is pervasively applied. I do not fully grasp your suggestion for hash functions that are distribution-dependent; especially how they would work incrementally, when values are added to a BAT over time. In that case, the distribution would change continually, and thus the hash function.. Peter -----Original Message----- From: Martin Kersten [mailto:Martin.Kersten@cwi.nl] Sent: Friday, October 20, 2006 5:15 PM To: Peter Boncz Cc: Niels Nes; Martin Kersten; Stefan Manegold; Sjoerd Mullender Subject: strings Peter, Niels and I were discussing the snippet shown in the attachment two days ago. The snippet is interesting for several reasons. The most important one is that the cost of this string join comes from inserting 800K strings, *not* the str hash join itself. (observed with callgrind) Amongst others, it triggered (again) the wish to have more control at the BAT level to deal with strings values. For example, it would be nice to know that *all* doubles are eliminated (not only per chunk). In combination with a fast 'uniqueCount' it could make some operations a lot faster. Alternatively, it would be nice if you could refer into the heap of another BAT, e.g. by marking a column as 'shared heap' and re-using the BATview scheme in the target to protect it. (In the 19.sql case, we actually should have dropped the string in the semijoin operation. Calling for either a new gdk operation or a better optimization step.) Wat betreft je voorstel. Ik zou zoeken in de volgende richting. De hash wordt in belangrijke mate bepaald door de statistische kans dat je hetzelfde patroon tegenkomt. Tevens weten we dat in de praktijk het gebruik van characters (en character combinaties) zipfian gedistribueerd is. Dit kan je gebruiken door in de header of de string heap een encoding tabel op te nemen die ervoor zorgt dat nieuwe encoding meer/minder effect heeft op de hash functie. Voorbeeld laat alle veel voorkomende characters weg en neem altijd N minder voorkomende characters. Of gebruik alternatieve bit patronen die de hash Deze codering zou je zelfs als apparte optimalizatie stap op de BAT kunnen loslaten. De default is gebaseerd op een eenvoudige tekst analyse. Verder kan de hash table (nu 1K best wat groter)

1 0

[Monetdb-developers] Compiling buildtools..
by Romulo Goncalves 20 Oct '06

20 Oct '06

Dear developers, I tried three times more and now everything works. It is strange such error. I continue to interest in reason for such error. Best Regards, Romulo Goncalves

2 2