October 2007 - developers-list

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Stefan Manegold 16 Oct '07

16 Oct '07

Peter, the purpose and/or reason of/for my question is the fact that we cannot release the binary packages that Sjoerd built yesterday, because the binary database format has been modified without increasing the GDK version number. We simply forgot the latter. Hence, a new Mserver cannot notice in case it is started on an pld database, and instead of complaining properly, it will just behave strangely, crash and/or corrup the database. This is not good. Consequently, Sjoerd needs to re-do the building once again, and we have to think of how to solve the above problem. Obviously, one solution is th "simply" increase the GDK version number. But then, users have to abandon or dump/retore their databases now and with the next release (GDK-2). An other alternative would be to undo the string hash function changes in the release branch, keeping the same binary format for now, and include both binary format changes in one release. In particular if the string hash function changes are "only" for performance reasons, both Sjoerd, Niels, and I would favor the second solution. After all, we and our users have been able to live with the "old" string hash function for years. Those that really require the new one can switch to the development version. I'm busy with testing --- on a 8GB dual-dual-core 2GHz Opteron running 64-bit FedoraCore 6 and and optimized 64-bit Mserver with 64-bit OIDs, shredding the 19 GB file read-only took 11 hours, both with the old and the new hash functions (Mserver grew to 10 GB res, >50 GB virt). Shredding updateable without your two fixes took only 2 hours, but the resulting DB is currupt as reported in [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 . Shredding updateable with your two fixes is still busy, running for more than 6 hours, now... Stefan On Tue, Oct 16, 2007 at 01:46:47PM +0200, Peter Boncz wrote: > Hi, > > Hm, I cannot really understand the purpose of the question. And what is > wrong with performance fixes? > > both fixes are related to the same bug: > - the remap failing is addressed by the gdk_posix fix > - the shredding in the bug report taking excessively long is addressed by > the gdk_atoms fix > > indeed, any hash function can have collisions.. it all depends on the > distribution. > > Peter > > PS most probably, this mail (sent from my home account) will be rejected by > the sourceforge mailing list -- and I cannot sent through CWI from home as > the secure mail sending is not supported by CWI staff for microsoft > emailers. > > > -----Original Message----- > From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] > Sent: dinsdag 16 oktober 2007 12:01 > To: Peter Boncz > Cc: monetdb-developers(a)lists.sourceforge.net > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, > 1.143.2.1 > > > Peter, > > which part of your changes do fix the problem with updatedable shredding of > large XML documents as reporten in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > ? > > The new has string function in gdk_atoms.mx or the file descriptor fixes in > gdk_posix.mx? > > The former looks for like a performance fix to me --- too many collisions > should only slows the system down, but not copromize its > fucntionallity/correctness, right? > Also with the new string has functions ("too") many collisions can still > occur with certain datasets ... > > Stefan > > > On Sun, Oct 14, 2007 at 08:31:36PM +0000, Stefan Manegold wrote: > > Update of /cvsroot/monetdb/MonetDB/src/gdk > > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15103 > > > > Modified Files: > > Tag: MonetDB_1-20 > > gdk_atoms.mx gdk_posix.mx > > Log Message: > > > > [checkin on behalf of Peter] > > > > fixing XQuery bug > > [ 1811229 ] [ADT] Adding large document, with update support > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=56 > 967&atid=482468 > > > > gdk_atoms.mx: > > - hash collisions in strings that consists of digits only (a common case!) > > we now use a fast derivative of the Bob Jenkins function from now on > > > > Really bad collisions, in case of the 20GB document of the bug report, > > shredding took 8 hours before, 1 hour after this change. > > > > NOTE: this change affects the binary format (string heaps) and all > product > > families, as the hash function is a compiled-in macro! > > In particular, lookup operations and joins on SQL (Monet4/5) > columns > > consisting of digits only, but stored in a VARCHAR, should be > faster > > after this check-in. > > > > gdk_posix.mx > > - we lost track of the file descriptor for large heaps (the file desc is > given > > to the mmap-monitoring-thread to close later), such that the remap > function > > could fail (when it was given the illegal file descriptor 0) > > > > NOTE: this change only affects xquery it only uses remap() > > > > > > Index: gdk_posix.mx > > =================================================================== > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > > retrieving revision 1.143 > > retrieving revision 1.143.2.1 > > diff -u -d -r1.143 -r1.143.2.1 > > --- gdk_posix.mx 4 Sep 2007 17:55:20 -0000 1.143 > > +++ gdk_posix.mx 14 Oct 2007 20:31:33 -0000 1.143.2.1 > > @@ -615,7 +615,7 @@ > > MT_mmap_tab[i].writable = writable; > > MT_mmap_tab[i].fd = fd; > > MT_mmap_tab[i].pincnt = 0; > > - fd = -1; > > + fd = -fd; > > } > > (void) pthread_mutex_unlock(&MT_mmap_lock); > > return fd; > > @@ -1051,9 +1051,7 @@ > > } > > if (ret != (void *) -1L) { > > hdl->fixed = ret; > > - fd = MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > > - if (fd <= 0) > > - hdl->hdl = (void *) 0; /* MT_mmap_new keeps the fd */ > > + hdl->hdl = (void*) (ssize_t) MT_mmap_new(path, ret, len, fd, (mode & > MMAP_WRITABLE)); > > } > > return ret; > > } > > @@ -1061,13 +1059,12 @@ > > void * > > MT_mmap_remap(MT_mmap_hdl *hdl, off_t off, size_t len) > > { > > - void *ret; > > - > > - ret = mmap(hdl->fixed, > > + int fd = (int) (ssize_t) hdl->hdl; > > + void *ret = mmap(hdl->fixed, > > len, > > ((hdl->mode & MMAP_WRITABLE) ? PROT_WRITE : 0) | PROT_READ, > > ((hdl->mode & MMAP_COPY) ? (MAP_PRIVATE | MAP_NORESERVE) : > MAP_SHARED) | (hdl->fixed ? MAP_FIXED : 0), > > - (int) (ssize_t) hdl->hdl, > > + (fd < 0)?-fd:fd, > > off); > > > > if (ret != (void *) -1L) { > > @@ -1083,9 +1080,7 @@ > > MT_mmap_close(MT_mmap_hdl *hdl) > > { > > int fd = (int) (ssize_t) hdl->hdl; > > - > > - if (fd) > > - close(fd); > > + if (fd > 0) close(fd); > > hdl->hdl = NULL; > > } > > > > > > Index: gdk_atoms.mx > > =================================================================== > > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_atoms.mx,v > > retrieving revision 1.134 > > retrieving revision 1.134.6.1 > > diff -u -d -r1.134 -r1.134.6.1 > > --- gdk_atoms.mx 2 May 2007 16:16:58 -0000 1.134 > > +++ gdk_atoms.mx 14 Oct 2007 20:31:32 -0000 1.134.6.1 > > @@ -1878,13 +1878,19 @@ > > rotates all characters together. It is optimized to process 2 characters > > at a time (adding 16-bits to the hash value each iteration). > > @h > > -#define GDK_STRHASH(x,y) { \ > > - str _c = (str) (x); \ > > - for((y)=0; _c[0] && _c[1]; _c+=2) { \ > > - (y) = ((y) << 3) ^ ((y) >> 11) ^ ((y) >> 17) ^ (_c[1] << > 8) ^ _c[0];\ > > - } \ > > - (y) ^= _c[0]; \ > > +#define GDK_STRHASH(x,y) {\ > > + str _key = (str) (x);\ > > + int _i;\ > > + for (_i = y = 0; _key[_i]; _i++) {\ > > + y += _key[_i];\ > > + y += (y << 10);\ > > + y ^= (y >> 6);\ > > + }\ > > + y += (y << 3);\ > > + y ^= (y >> 11);\ > > + y += (y << 15);\ > > } > > + > > @c > > hash_t > > strHash(str s) > > > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Splunk Inc. > > Still grepping through log files to find problems? Stop. > > Now Search log events and configuration files using AJAX and a browser. > > Download your FREE copy of Splunk now >> http://get.splunk.com/ > > _______________________________________________ > > Monetdb-checkins mailing list > > Monetdb-checkins(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Stefan Manegold 16 Oct '07

16 Oct '07

Peter, which part of your changes do fix the problem with updatedable shredding of large XML documents as reporten in [ 1811229 ] [ADT] Adding large document, with update support http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… ? The new has string function in gdk_atoms.mx or the file descriptor fixes in gdk_posix.mx? The former looks for like a performance fix to me --- too many collisions should only slows the system down, but not copromize its fucntionallity/correctness, right? Also with the new string has functions ("too") many collisions can still occur with certain datasets ... Stefan On Sun, Oct 14, 2007 at 08:31:36PM +0000, Stefan Manegold wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv15103 > > Modified Files: > Tag: MonetDB_1-20 > gdk_atoms.mx gdk_posix.mx > Log Message: > > [checkin on behalf of Peter] > > fixing XQuery bug > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=1811229&group_id=5… > > gdk_atoms.mx: > - hash collisions in strings that consists of digits only (a common case!) > we now use a fast derivative of the Bob Jenkins function from now on > > Really bad collisions, in case of the 20GB document of the bug report, > shredding took 8 hours before, 1 hour after this change. > > NOTE: this change affects the binary format (string heaps) and all product > families, as the hash function is a compiled-in macro! > In particular, lookup operations and joins on SQL (Monet4/5) columns > consisting of digits only, but stored in a VARCHAR, should be faster > after this check-in. > > gdk_posix.mx > - we lost track of the file descriptor for large heaps (the file desc is given > to the mmap-monitoring-thread to close later), such that the remap function > could fail (when it was given the illegal file descriptor 0) > > NOTE: this change only affects xquery it only uses remap() > > > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.143 > retrieving revision 1.143.2.1 > diff -u -d -r1.143 -r1.143.2.1 > --- gdk_posix.mx 4 Sep 2007 17:55:20 -0000 1.143 > +++ gdk_posix.mx 14 Oct 2007 20:31:33 -0000 1.143.2.1 > @@ -615,7 +615,7 @@ > MT_mmap_tab[i].writable = writable; > MT_mmap_tab[i].fd = fd; > MT_mmap_tab[i].pincnt = 0; > - fd = -1; > + fd = -fd; > } > (void) pthread_mutex_unlock(&MT_mmap_lock); > return fd; > @@ -1051,9 +1051,7 @@ > } > if (ret != (void *) -1L) { > hdl->fixed = ret; > - fd = MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > - if (fd <= 0) > - hdl->hdl = (void *) 0; /* MT_mmap_new keeps the fd */ > + hdl->hdl = (void*) (ssize_t) MT_mmap_new(path, ret, len, fd, (mode & MMAP_WRITABLE)); > } > return ret; > } > @@ -1061,13 +1059,12 @@ > void * > MT_mmap_remap(MT_mmap_hdl *hdl, off_t off, size_t len) > { > - void *ret; > - > - ret = mmap(hdl->fixed, > + int fd = (int) (ssize_t) hdl->hdl; > + void *ret = mmap(hdl->fixed, > len, > ((hdl->mode & MMAP_WRITABLE) ? PROT_WRITE : 0) | PROT_READ, > ((hdl->mode & MMAP_COPY) ? (MAP_PRIVATE | MAP_NORESERVE) : MAP_SHARED) | (hdl->fixed ? MAP_FIXED : 0), > - (int) (ssize_t) hdl->hdl, > + (fd < 0)?-fd:fd, > off); > > if (ret != (void *) -1L) { > @@ -1083,9 +1080,7 @@ > MT_mmap_close(MT_mmap_hdl *hdl) > { > int fd = (int) (ssize_t) hdl->hdl; > - > - if (fd) > - close(fd); > + if (fd > 0) close(fd); > hdl->hdl = NULL; > } > > > Index: gdk_atoms.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_atoms.mx,v > retrieving revision 1.134 > retrieving revision 1.134.6.1 > diff -u -d -r1.134 -r1.134.6.1 > --- gdk_atoms.mx 2 May 2007 16:16:58 -0000 1.134 > +++ gdk_atoms.mx 14 Oct 2007 20:31:32 -0000 1.134.6.1 > @@ -1878,13 +1878,19 @@ > rotates all characters together. It is optimized to process 2 characters > at a time (adding 16-bits to the hash value each iteration). > @h > -#define GDK_STRHASH(x,y) { \ > - str _c = (str) (x); \ > - for((y)=0; _c[0] && _c[1]; _c+=2) { \ > - (y) = ((y) << 3) ^ ((y) >> 11) ^ ((y) >> 17) ^ (_c[1] << 8) ^ _c[0];\ > - } \ > - (y) ^= _c[0]; \ > +#define GDK_STRHASH(x,y) {\ > + str _key = (str) (x);\ > + int _i;\ > + for (_i = y = 0; _key[_i]; _i++) {\ > + y += _key[_i];\ > + y += (y << 10);\ > + y ^= (y >> 6);\ > + }\ > + y += (y << 3);\ > + y ^= (y >> 11);\ > + y += (y << 15);\ > } > + > @c > hash_t > strHash(str s) > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] XRPCwrapper fails to compile with java 1.4
by Stefan Manegold 16 Oct '07

16 Oct '07

Hoi, apparently, the new XPRCwrapper code fails to compile with java 1.4 (our RedHat 4 WS machine, titan, is the only testing machine that still has only java 1.4, all others have java 1.5); for details see below and/or http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.ntv.64.64… Question: Assuming that there is no easy way to make the new XRPC warpper code java 1.4 compliant: should we modify (tighten) our requirements accordingly, i.e., requiring (only) java 1.5 instead of (at least) java 1.4 to build MonetDB* (java 1.6 is not (yet?) supported)? Stefan ======== "/usr/local/bin/ant" -f "`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml" -Dbuilddir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java" -Djardir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java" -Dbasedir="`readlink -f /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java" distall Buildfile: /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml distall: prepare: [echo] Debug is true, optimise is true compile_util: [echo] Compiling Utilities [javac] Compiling 9 source files to /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/RedHat4WS/src/tools/java [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/NamespaceContextImpl.java:22: package javax.xml.namespace does not exist [javac] import javax.xml.namespace.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/NamespaceContextImpl.java:29: cannot resolve symbol [javac] symbol : class NamespaceContext [javac] location: class nl.cwi.monetdb.xquery.util.NamespaceContextImpl [javac] public class NamespaceContextImpl implements NamespaceContext{ [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:23: package javax.xml.namespace does not exist [javac] import javax.xml.namespace.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:24: package javax.xml.xpath does not exist [javac] import javax.xml.xpath.*; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:28: package com.sun.org.apache.xml.internal.dtm.ref does not exist [javac] import com.sun.org.apache.xml.internal.dtm.ref.DTMNodeList; [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:372: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:409: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] public static NamedNodeMap getNodeAttributes(XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:432: cannot resolve symbol [javac] symbol : class XPath [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] public static String[] getNodeListAttribute(XPath xPath, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:415: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:415: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:416: cannot resolve symbol [javac] symbol : variable XPathConstants [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] inputSource, XPathConstants.NODESET); [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:439: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:439: cannot resolve symbol [javac] symbol : class DTMNodeList [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] DTMNodeList nodeList = (DTMNodeList) xPath.evaluate(xPathExpr, [javac] ^ [javac] /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/src/nl/cwi/monetdb/xquery/util/XRPCMessage.java:440: cannot resolve symbol [javac] symbol : variable XPathConstants [javac] location: class nl.cwi.monetdb.xquery.util.XRPCMessage [javac] inputSource, XPathConstants.NODESET); [javac] ^ [javac] 14 errors BUILD FAILED /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml:55: The following error occurred while executing this line: /export/scratch1/monet/monet.ntv.64.64.d.8915/pathfinder/src/tools/java/build.xml:107: Compile failed; see the compiler error output for details. Total time: 6 seconds make[7]: *** [distall_ant_target] Error 1 ======== -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5 12

Re: [Monetdb-developers] BAT sizes
by Stefan Manegold 15 Oct '07

15 Oct '07

[I once again felt free to share this with the community ...] Henning, in case BAT capacities are significantly larger than their actual content (count), this might indeed have a negative influence on performance. (1) in case the BAT is memory-mapped when loaded, it "only" blocks some more address space than strictly necessary (no problem on 64-bit systems, potentially a problem on 32-bit systems); (2) in case the BAT is malloced when loaded, it also occupied some more memory than strictly necessary (potential problem on but 64- & 32-bit systems). However, unless there is some accurate estimation, it is often hard (or virually impossible) to "guess" a BATs size before filling it; hence, a "generous" initial size allocation is good to avoid expensive BAT extents. In your case, I'm lost concerning which BATs your taling about. The shredder-generated pre_* (actually rid_*) BATs need to be allocated before reading the document; hence, there is know knowledge about the number of nodes in the document, and as far as I can tell no trivial way to estimate this accurately. Hence, the shredder needs to guess something --- JanF can tell more, I guess... In case of the TIJAH indices, I have no clue at all, how/where/when they are built and whether there might be better information available to not overallocate but allocate only just enough space. You or some of your colleagues in Twente should know all the details. Finally, is there any concrete case where you actually experiences any problems due to "over-allocation", or are you just wondering? Stefan ps: in the case given below, the batsize just fits the BAT's capacity; only the count is smaller than the capacity (obviously, it cannot be larger) --- if you want/need to know why, you better ask him/her who allocated/created/filled the "tj_DFLT_FT_INDEX_size1" BAT ... On Mon, Oct 15, 2007 at 02:36:46PM +0200, Henning Rode wrote: > hej stefan, > > sorry, that i did not answer earlier. i justed wanted to report the > actual sizes of pf/tijah indices in a paper. so that is done now. > > still, i was asking myself, whether it might have any kind of > performance influences, that BAT capacities are so much higher than the > actual BAT counts. This is of course handy, when we still want to add > new entries, but once we indexed a collection, we usually only query it. > > in case of our "pre_size" BAT this difference between BATsize and > BATdsksize can easily be 250MB or more. > > best -henning > > mil>var t := bat("tj_DFLT_FT_INDEX_size1"); > mil>t.count().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |203091470 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.capacity().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |260898816 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.batsize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |1043599360 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.batdsksize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |812366848 > > > > mil>var x := t.copy(); > mil>x.count().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |203091470 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.capacity().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |260898816 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.batsize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |1043599360 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>x.batdsksize().print(); > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > |812366848 > > | > +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ > mil>t.access(BAT_READ); > > > Stefan Manegold wrote: > > Henning, > > > > you should also check & report b.capacity(), i.e., > > > > b.count(); > > b.capacity(); > > b.info().reverse().like("batBuns").like("size").print(); > > b.batsize(); > > b.batdsksize(); > > > > var c := b.copy(); > > > > c.count(); > > c.capacity(); > > c.info().reverse().like("batBuns").like("size").print(); > > c.batsize(); > > c.batdsksize(); > > > > Stefan > > > > > > On Sat, Oct 06, 2007 at 07:43:28PM +0200, Stefan Manegold wrote: > >> [felt free to cc the monetdb-developers list as more people might be > >> interested or want to contribute] > >> > >> Henning, > >> > >> are you just "concerned" or are you having concrete problems with the bat > >> sizes? > >> > >> In cany case, to give any reasonable answer we'd need to know more about the > >> details. In particular how large is the BAT your talking about. > >> > >> I.e., with "b" being your BAT and "c := b.copy()", please check & report > >> > >> b.count(); > >> b.info().reverse().like("batBuns").like("size").print(); > >> b.batsize(); > >> b.batdsksize(); > >> > >> c.count(); > >> c.info().reverse().like("batBuns").like("size").print(); > >> c.batsize(); > >> c.batdsksize(); > >> > >> Stefan > >> > >> > >> On Fri, Oct 05, 2007 at 01:47:01PM +0200, Henning Rode wrote: > >>> hej stefan, > >>> > >>> thanks for the answer. so in conclusion, the over-allocation of memory > >>> is quite normal, and nothing to worry about. > >>> > >>> i was more surprised that the copied BAT still has this considerable > >>> over-allocation of memory, though it exactly knows how many entries it > >>> needs to hold. > >>> > >>> groeten -henning > >> -- > >> | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > >> | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > >> | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > >> | The Netherlands | Fax : +31 (20) 592-4312 | > > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools monet_version.c.in, MonetDB_5-2, NONE, 1.1.2.1 monet_version.h, MonetDB_5-2, NONE, 1.1.2.1 mserver5.mx, MonetDB_5-2, NONE, 1.29.2.2 Makefile.ag, MonetDB_5-2, 1.54.2.1, 1.54.2.2 mserver5.mx.in, MonetDB_5-2, 1.33, NONE
by Stefan Manegold 14 Oct '07

14 Oct '07

Niels, thanks for the fix --- unfortunately a compile from scratch fails as follows: ======== make[5]: Entering directory /net/corona.ins.cwi.nl/export/scratch0/manegold/Monet/Testing/Stable/build.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/MonetDB5/src/tools' /bin/sh ../../libtool --tag=CC --mode=link gcc -Wall -Wextra -std=c99 -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 -D_REENTRANT -o mserver5 -export-dynamic mserver5.o ../mal/libmal.la -L/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64 -lbat -lmutils -lstream -lz -lbz2 -lpthread -ldl -lpcre -lssl -lcrypto -lcrypt gcc -Wall -Wextra -std=c99 -g -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 -D_REENTRANT -o .libs/mserver5 mserver5.o -Wl,--export-dynamic ../mal/.libs/libmal.so -L/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64 -lreadline -ltermcap /ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64/libbat.so /ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64/libstream.so /ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64/libmutils.so -lz -lbz2 -lpthread -ldl -lpcre -lssl -lcrypto -lcrypt -Wl,--rpath -Wl,/ufs/manegold/_/scratch0/Monet/Testing/Stable/prefix.--enable-strict_--disable-optimize_--enable-debug_--enable-assert/lib64 mserver5.o: In function `main': /ufs/manegold/_/scratch0/Monet/Testing/Stable/source/MonetDB5/src/tools/mserver5.mx:452: undefined reference to `monet_version' collect2: ld returned 1 exit status make[5]: *** [mserver5] Error 1 ======== It seems that autogen does not handle .c.in files properly, at least not in "SOURCE =" lists ... Stefan On Sun, Oct 14, 2007 at 08:20:25AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv25912/src/tools > > Modified Files: > Tag: MonetDB_5-2 > Makefile.ag > Added Files: > Tag: MonetDB_5-2 > monet_version.c.in monet_version.h mserver5.mx > Removed Files: > Tag: MonetDB_5-2 > mserver5.mx.in > Log Message: > tablet fixed null pointer dereference > split out the monet_version code as it needs configure to set variables. > (mx.in's are now not allowed anymore) > > > Index: Makefile.ag > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/tools/Makefile.ag,v > retrieving revision 1.54.2.1 > retrieving revision 1.54.2.2 > diff -u -d -r1.54.2.1 -r1.54.2.2 > --- Makefile.ag 3 Oct 2007 09:45:59 -0000 1.54.2.1 > +++ Makefile.ag 14 Oct 2007 08:20:22 -0000 1.54.2.2 > @@ -19,7 +19,7 @@ > INCLUDES = -I../mal $(CLIENTS_INCS) $(MONETDB_INCS) $(OPENSSL_INCS) $(PCRE_CFLAGS) > > bin_mserver5 = { > - SOURCES = mserver5.mx.in > + SOURCES = mserver5.mx monet_version.h monet_version.c.in > LIBS = ../mal/libmal @SHARED_LIBS@ $(MONETDB_LIBS) \ > -lbat -lmutils -lstream \ > $(SOCKET_LIBS) $(Z_LIBS) $(BZ_LIBS) \ > > --- NEW FILE: monet_version.h --- > #ifndef MONETDB_VERSION_H > #define MONETDB_VERSION_H > > extern void monet_version(void); > > #endif /* MONETDB_VERSION_H */ > > --- NEW FILE: mserver5.mx --- > @' The contents of this file are subject to the MonetDB Public License > @' Version 1.1 (the "License"); you may not use this file except in > @' compliance with the License. You may obtain a copy of the License at > @' http://monetdb.cwi.nl/Legal/MonetDBLicense-1.1.html > @' > @' Software distributed under the License is distributed on an "AS IS" > @' basis, WITHOUT WARRANTY OF ANY KIND, either express or implied. See the > @' License for the specific language governing rights and limitations > @' under the License. > @' > @' The Original Code is the MonetDB Database System. > @' > @' The Initial Developer of the Original Code is CWI. > @' Portions created by CWI are Copyright (C) 1997-2007 CWI. > @' All Rights Reserved. > > @f mserver5 > @a M.L. Kersten, P. Boncz, Niels Nes, Stefan Manegold, Sjoerd Mullender > @v 5.0 > @* The Monet Server > The program @code{ mserver5} is the MonetDB server. It is started by the database > administrator. The console is primarilly meant to further initialize > the server, such as starting internet listeners. > > @+ Manual Page > The server is a multithreaded program. There is one system thread, and for > each service, e.g. a client session or transaction service, > there is one worker thread. > As a default, the server also starts the internet listener thread, > on the port number specified in the parameter file. > > @- Usage > @verbatim > usage: mserver5 [options] [scripts] > --dbname=<database_name> > --dbfarm=<directory> > --dbinit=<stmt> Server prepare statement > --config=<config_file> > --debug=<number> trace server actions[0] > --daemon=yes|no run in background [no] > --set <option>=<value> set environment value > --help this list of options > --version version and compile time info > @end verbatim > > The server options have the following meaning: > @table @code > @item --dbname <db-name> > open the database <db-name>. > @item --config <config-file> > where to find the environment settings > @item --dbinit <stmt> > execute the statement first. > @end itemize > > @{ > @+ Implementation > @h > #ifndef _MONET_GLOBAL_H_ > #define _MONET_GLOBAL_H_ > #include "mal_config.h" > #include "monet_options.h" > #include "mal.h" > #include "mal_session.h" > #include "mal_import.h" > #include "mal_client.h" > #include "mal_function.h" > > /* #define MONET_GLOBAL_DEBUG */ > #endif /* _MONET_GLOBAL_H_ */ > @c > #include "mal_config.h" > #include "mserver5.h" > #include "mal_authorize.h" > #include "mal_sabaoth.h" > #include <stdio.h> > #include <errno.h> > #include <string.h> /* strerror */ > #include <locale.h> > #include "monet_version.h" > > #ifdef _CRTDBG_MAP_ALLOC > /* Windows only: > our definition of new and delete clashes with the one if > _CRTDBG_MAP_ALLOC is defined. > */ > #undef _CRTDBG_MAP_ALLOC > #endif > > #if defined(_MSC_VER) && _MSC_VER >= 1400 > #define getcwd _getcwd > #endif > @- > The architecture is setup to handle multiple streams of requests. > The first thread started represents the server. It reads from standard input > and writes to standard input. This is also a way to recognize the server > actions. > To start the server in the background one should use the argument -background. > This closes standard input. Direct execution in the background may cause > the server to hang in stdio for input from the terminal. > @ > The server thread started remains in existence until all other threads die. > The server is stopped by cntrl-D or receiving the quit command. > @ > > @c > static int malloc_init = 1; > /* NEEDED? */ > #if defined(_MSC_VER) && defined(__cplusplus) > #include <eh.h> > void > mserver_abort() > { > fprintf(stderr, "\n! mserver_abort() was called by terminate(). !\n"); > fflush(stderr); > MT_global_exit(0); > } > #endif > > void > usage(char *prog) > { > fprintf(stderr, "Usage: %s [options] [scripts]\n", prog); > fprintf(stderr, " --dbname=<database_name> \n"); > fprintf(stderr, " --dbfarm=<database_directory> \n"); > fprintf(stderr," --version \n"); > fprintf(stderr," --help for more details\n"); > exit(0); > } > void > usage2(char *prog) > { > fprintf(stderr, "Usage: %s [options] [scripts]\n", prog); > fprintf(stderr, " --dbname=<database_name> \n"); > fprintf(stderr, " --dbfarm=<directory> \n"); > fprintf(stderr, " --dbinit=<stmt> Server prepare statement\n"); > fprintf(stderr, " --config=<config_file> \n"); > fprintf(stderr, " --daemon=yes|no run in background [no]\n"); > fprintf(stderr, " --set <option>=<value> set environment value\n"); > fprintf(stderr, " --help this list of options \n"); > fprintf(stderr, " --version version and compile time info\n"); > > fprintf(stderr,"The debug options:\n"); > fprintf(stderr," --threads\n"); > fprintf(stderr," --memory\n"); > fprintf(stderr," --properties\n"); > fprintf(stderr," --io\n"); > fprintf(stderr," --transactions\n"); > fprintf(stderr," --modules\n"); > fprintf(stderr," --algorithms\n"); > fprintf(stderr," --xproperties\n"); > fprintf(stderr," --performance\n"); > fprintf(stderr," --debug=<bitmask>\n"); > > exit(0); > } > > @- > A welcoming message is displayed to inform the user about recent > changes. > @c > void > monet_hello(opt *set, int setlen) > { > #ifdef STATIC > char *linkinfo = "statically"; > #else > char *linkinfo = "dynamically"; > #endif > char *msg = mo_find_option(set, setlen, "monet_welcome"); > > if (msg && strcmp(msg, "yes") == 0) { > printf("# MonetDB server v" VERSION ", based on kernel v%s\n", > GDKversion()); > printf("# Serving database '%s'\n", GDKgetenv("gdk_dbname")); > printf("# Compiled for %s/" SZFMT "bit with " SZFMT "bit OIDs %s linked\n", HOST, (size_t) (sizeof(ptr) * 8), (size_t) (sizeof(oid) * 8), linkinfo); > #ifdef MONET_GLOBAL_DEBUG > printf("# Configuration:%s\n", GDKgetenv("config")); > printf("# Database farm:%s\n", GDKgetenv("gdk_dbfarm")); > #endif > printf("# Copyright (c) 1993-2007 CWI, all rights reserved\n"); > printf("# Visit http://monetdb.cwi.nl/ for further information\n"); > } > } > > @- > Version information, and compile time options. > @c > #ifdef HAVE_LIBPCRE > #include <pcre.h> > #endif > #ifdef HAVE_OPENSSL > #include <openssl/opensslv.h> > #endif > > str > absolute_path(str s) > { > if (!MT_path_absolute(s)) { > str ret = (str) GDKmalloc(strlen(s) + strlen(monet_cwd) + 2); > > sprintf(ret, "%s%c%s", monet_cwd, DIR_SEP, s); > return ret; > } > return GDKstrdup(s); > } > > @- > The options obtained during initialization should be maintained as > a global structure for other components to extract information. > @c > #define BSIZE 8192 > > int > monet_init(opt *set, int setlen) > { > char *p; > opt *n = (opt *) malloc(setlen * sizeof(opt)); > int i, j, nlen = 0; > > char *dbname = mo_find_option(set, setlen, "gdk_dbname"); > char *dbfarm = mo_find_option(set, setlen, "gdk_dbfarm"); > char *alloc_map = mo_find_option(set, setlen, "gdk_alloc_map"); > > if (n == NULL || dbname == NULL || dbfarm == NULL || alloc_map == NULL) { > fprintf(stderr, "Error, no database name or directory\n"); > if (n) > free(n); > return 0; > } > > dbfarm = mo_substitute(set, setlen, dbfarm); > > if ((p = mo_find_option(set, setlen, "gdk_debug"))) > GDKdebug = strtol(p, NULL, 10); > > if ((p = mo_find_option(set, setlen, "gdk_mem_pagebits"))) > GDK_mem_pagebits = strtol(p, NULL, 10); > > if ((p = mo_find_option(set, setlen, "gdk_vmtrim"))) > GDK_vm_trim = strcasecmp(p, "yes") == 0; > > /* determine Monet's kernel settings */ > if (!GDKinit(dbname, dbfarm, strcasecmp(alloc_map, "yes") == 0) ) { > free(dbfarm); > free(n); > return 0; > } > free(dbfarm); > > @- > Find duplicate entries in the property list and move > them to the front. Actually, this should be done in > monet_options.mx.in > @c > for (i = 0; i < setlen; i++) { > int done = 0; > > for (j = 0; j < nlen; j++) { > if (strcmp(n[j].name, set[i].name) == 0) { > if (n[j].kind < set[i].kind) { > n[j] = set[i]; > } > done = 1; > break; > } > } > if (!done) { > n[nlen] = set[i]; > nlen++; > } > } > for (i = 0; i < nlen; i++) { > char *value; > > value = mo_substitute(n, nlen, n[i].value); > GDKsetenv(n[i].name, value); > free(value); > } > free(n); > > if ((p = GDKgetenv("gdk_mem_bigsize"))) { > /* when allocating >6% of all RAM; do so using vmalloc() iso malloc() */ > lng max_mem_bigsize = GDK_mem_maxsize/16; > > /* sanity check to avoid memory fragmentation */ > GDK_mem_bigsize = (size_t) MIN(max_mem_bigsize, strtol(p, NULL, 10)); > } > if (GDKgetenv_isyes("gdk_embedded") || GDKgetenv_isyes("embedded")) { > GDKembedded = 1; > } > > if (GDKgetenv_isyes("monet_daemon") || GDKgetenv_isyes("daemon")) { > monet_daemon = 1; > #ifdef HAVE_SETSID > setsid(); > #endif > } > > monet_hello(set, setlen); > /* you don't need the commandline arguments anymore */ > mo_free_options(set, setlen); > return 1; > } > > void emergencyBreakpoint(){ > /* just a handle to break after system initialization for GDB */ > } > > static void > handler(int sig) > { > (void)sig; > mal_exit(); > } > > int > main(int argc, char **av) > { > char *prog = *av; > opt *set = NULL; > int idx = 0, debug = 0, setlen = 0, listing = 0, i = 0; > str dbinit = NULL; > str err = MAL_SUCCEED; > > static struct option long_options[] = { > {"config", 1, 0, 'c'}, > {"dbname", 1, 0, 0}, > {"dbfarm", 1, 0, 0}, > {"dbinit", 1, 0, 0}, > {"daemon", 1, 0, 0}, > {"debug", 2, 0, 'd'}, > {"help", 0, 0, 'h'}, > {"version", 0, 0, 0}, > {"set", 1, 0, 's'}, > {"trace", 0, 0, 't'}, > {"threads",0,0,0}, > {"memory",0,0,0}, > {"properties",0,0,0}, > {"io",0,0,0}, > {"transaction",0,0,0}, > {"modules",0,0,0}, > {"algorithms",0,0,0}, > {"performance",0,0,0}, > {"xproperties",0,0,0}, > {0, 0, 0, 0} > }; > > @- > We give malloc advice here. Main goal: prevent fragmentation. > We do this by declaring everything below 2K as 'small'. These > values will be drawn from a fixed pools of 400K. > A grain size of 128 bytes is used to keep overhead low. > > We do this by declaring everything below 2K as 'small'. These > values will be drawn from a fixed pools of 400K. > A grain size of 128 bytes is used to keep overhead low. > Trivial remark: for dynamically linked executables the mallopt > capabilities depend on the malloc implementation used at run time. > > Unlike V4 we ignore the alloc_map advice, which leads to a much > faster system start. > @= mallopt > if (malloc_init) { > /* for (Red Hat) Linux (6.2) unused and ignored at least as of glibc-2.1.3-15 */ > /* for (Red Hat) Linux (8) used at least as of glibc-2.2.93-5 */ > if (mallopt(M_MXFAST, 192)) { > fprintf(stderr, "!monet: mallopt(M_MXFAST,192) fails.\n"); > } > #ifdef M_BLKSZ > if (mallopt(M_BLKSZ, 8*1024)) { > fprintf(stderr, "!monet: mallopt(M_BLKSZ,8*1024) fails.\n"); > } > #endif > } > malloc_init=0; > @c > > #if defined(_MSC_VER) && defined(__cplusplus) > set_terminate(mserver_abort); > #endif > if (setlocale(LC_CTYPE, "") == NULL) { > GDKfatal("cannot set locale\n"); > } > > #ifdef HAVE_MALLOPT > @:mallopt@ > #else > (void) malloc_init; /* still unused */ > #endif > > if (getcwd(monet_cwd, PATHLENGTH - 1) == NULL) { > perror("pwd"); > GDKfatal("monet_init: could not determine current directory\n"); > } > > if (!(setlen = mo_builtin_settings(&set))) > usage(prog); > > setlen = mo_add_option(&set, setlen, opt_config, "prefix", MONETDBPREFIX); > setlen = mo_add_option(&set, setlen, opt_config, "config", MONETDBCONFIG); > > for (;;) { > int option_index = 0; > > int c = getopt_long(argc, av, "c:d::t:r:h?s:m:i:a:e:x", > long_options, &option_index); > > if (c == -1) > break; > > switch (c) { > case 0: > if (strcmp(long_options[option_index].name, "dbname") == 0) { > setlen = mo_add_option(&set, setlen, opt_cmdline, "gdk_dbname", optarg); > break; > } > if (strcmp(long_options[option_index].name, "dbfarm") == 0) { > setlen = mo_add_option(&set, setlen, opt_cmdline, "gdk_dbfarm", optarg); > break; > } > if (strcmp(long_options[option_index].name, "dbinit") == 0) { > if (dbinit) > fprintf(stderr, "#warning: ignoring multiple --dbinit argument\n"); > else > dbinit = optarg; > break; > } > if (strcmp(long_options[option_index].name, "daemon") == 0) { > setlen = mo_add_option(&set, setlen, opt_cmdline, "gdk_daemon", optarg); > break; > } > if (strcmp(long_options[option_index].name, "version") == 0) { > monet_version(); > break; > } > /* debugging options */ > if (strcmp(long_options[option_index].name, "properties") == 0) { > debug |=GRPproperties; > break; > } > if (strcmp(long_options[option_index].name, "algorithms") == 0) { > debug|= GRPalgorithms; > break; > } > if (strcmp(long_options[option_index].name, "xproperties") == 0) { > debug|= GRPxproperties; > break; > } > if (strcmp(long_options[option_index].name, "performance") == 0) { > debug|= GRPperformance; > break; > } > if (strcmp(long_options[option_index].name, "io") == 0) { > debug|= GRPio; > break; > } > if (strcmp(long_options[option_index].name, "memory") == 0) { > debug|= GRPmemory; > break; > } > if (strcmp(long_options[option_index].name, "modules") == 0) { > debug|= GRPmodules; > break; > } > if (strcmp(long_options[option_index].name, "transactions") == 0) { > debug|= GRPtransactions; > break; > } > if (strcmp(long_options[option_index].name, "threads") == 0) { > debug|= GRPthreads; > break; > } > usage(prog); > break; > case 'c': > setlen = mo_add_option(&set, setlen, opt_cmdline, "config", optarg); > break; > case 'd': > if (optarg) { > setlen = mo_add_option(&set, setlen, opt_cmdline, "gdk_debug", optarg); > } else { > debug = 1; > } > break; > case 's':{ > /* should add option to a list */ > char *tmp = strchr(optarg, '='); > > if (tmp) { > *tmp = '\0'; > setlen = mo_add_option(&set, setlen, opt_cmdline, optarg, tmp + 1); > } else { > fprintf(stderr, "ERROR: wrong format %s\n", optarg); > } > } > break; > case 't': /* trace option, ignored to reduce testweb complaints > fprintf(stderr, "#warning: trace option not yet supported\n"); > */ > break; > case 'h': > if(strcmp(long_options[option_index].name, "help") == 0) { > usage2(prog); > break; > } > case '?': > usage(prog); > default: > fprintf(stderr, "ERROR: getopt returned character " > "code '%c' 0%o\n",c, c); > usage(prog); > } > } > > if (!(setlen = mo_system_config(&set, setlen))) > usage(prog); > > monet_script = (str *) GDKmalloc(sizeof(str) * (argc + 1)); > monet_script[idx] = NULL; > while (optind < argc) { > monet_script[idx] = absolute_path(av[optind]); > monet_script[idx + 1] = NULL; > optind++; > idx++; > } > > if (debug) > mo_print_options(set, setlen); > if (monet_init(set, setlen) == 0 ) > return 0; > /* propagate the debug flag to the environment table */ > if( debug){ > char buf[BUFSIZ]; > snprintf(buf,BUFSIZ,"%d",debug); > GDKsetenv("gdk_debug", buf); > GDKdebug=debug; /* overruled by command line */ > } > > GDKsetenv("monet_version", VERSION); > > /* configure sabaoth to use the right dbfarm and active database */ > SABAOTHinit(GDKgetenv("gdk_dbfarm"), GDKgetenv("gdk_dbname")); > /* wipe out all cruft, if left over */ > if ((err = SABAOTHwildRetreat(&i)) != MAL_SUCCEED) { > /* just swallow the error */ > GDKfree(err); > } > /* From this point, the server should exit cleanly. Discussion: > * even earlier? Sabaoth here registers the server has started up. */ > if ((err = SABAOTHregisterStart(&i)) != MAL_SUCCEED) { > /* throw the error at the user, but don't die */ > fprintf(stderr, "!%s\n", err); > GDKfree(err); > } > > #ifdef HAVE_SIGACTION > { > struct sigaction sa; > > sigemptyset(&sa.sa_mask); > sa.sa_flags = 0; > sa.sa_handler = handler; > if ( > sigaction(SIGINT, &sa, NULL) == -1 || > sigaction(SIGQUIT, &sa, NULL) == -1 || > sigaction(SIGTERM, &sa, NULL) == -1) > { > fprintf(stderr, "!unable to create signal handlers\n"); > } > } > #else > signal(SIGINT, handler); > #ifdef SIGQUIT > signal(SIGQUIT, handler); > #endif > signal(SIGTERM, handler); > #endif > > { > str lang = "mal"; > /* we inited mal before, so publish its existence */ > if ((err = SABAOTHmarchScenario(&i, &lang)) != MAL_SUCCEED) { > /* throw the error at the user, but don't die */ > fprintf(stderr, "!%s\n", err); > GDKfree(err); > } > } > > { > /* unlock the vault, first see if we can find the file which > * holds the secret */ > char* secret = alloca(sizeof(char) * 1024); > FILE* secretf; > size_t len; > > if (GDKgetenv("monet_vault_key") == NULL) { > /* use a default (hard coded, non safe) key */ > secret = "Xas632jsi2whjds8"; > fprintf(stderr, "#warning: please don't forget to set your " > "vault key!\n#(see %s)\n", GDKgetenv("config")); > } else { > if ((secretf = fopen(GDKgetenv("monet_vault_key"), "r")) == NULL) { > snprintf(secret, 1023, "unable to open vault_key_file %s: %s", > GDKgetenv("monet_vault_key"), strerror(errno)); > secret[1023] = '\0'; > GDKfatal(secret); > } > len = fread(secret, 1, 1023, secretf); > secret[len] = '\0'; > len = strlen(secret); /* secret can contain null-bytes */ > if (len < 5) { > fprintf(stderr, "#warning: your vault key is too short " > "(" SZFMT "), enlarge your vault key!\n", > len); /* penis */ > } > fclose(secretf); > } > if ((err = AUTHunlockVault(&secret)) != MAL_SUCCEED) > GDKfatal(err); > } > /* make sure the authorisation BATs are loaded */ > if ((err = AUTHinitTables()) != MAL_SUCCEED) > GDKfatal(err); > if (mal_init()) > return 0; > > @- > Time to execute the script files. > Start with the init command[todo], which may be defined in > the configuration file. > @c > MSinitClientPrg(mal_clients, "user","main"); > if (dbinit == NULL) > dbinit = GDKgetenv("dbinit"); > if (dbinit) { > str input = GDKstrdup(dbinit); > > callString(mal_clients, input, listing); > GDKfree(input); > } > > if (GDKgetenv("mal_listing")) > sscanf(GDKgetenv("mal_listing"), "%d", &listing); > > emergencyBreakpoint(); > for (i = 0; monet_script[i]; i++) { > str msg=evalFile(mal_clients, monet_script[i], listing); > /* check for internal exception message to terminate */ > if(msg && strcmp(msg,"MALException:client.quit:Server stopped.")==0) > mal_exit(); > if( msg) GDKfree(msg); > GDKfree(monet_script[i]); > monet_script[i] = 0; > } > GDKfree(monet_script); > if (monet_daemon) { > while(1) > MT_sleep_ms(5000); > } else > MSserveClient(mal_clients); > > /* mal_exit calls MT_global_exit, so statements after this call will > * never get reached */ > mal_exit(); > > return 0; > } > @} > > --- NEW FILE: monet_version.c.in --- > > #include "monet_version.h" > > void > monet_version() > { > printf("MonetDB server v" VERSION " (" SZFMT "-bit), " > "based on kernel v%s (" SZFMT "-bit oids)\n", > (size_t) (sizeof(ptr) * 8), GDKversion(), > (size_t) (sizeof(oid) * 8)); > printf("Copyright (c) 1993-2007 CWI, all rights reserved\n"); > printf("Visit http://monetdb.cwi.nl/ for further information\n"); > printf("Configured for prefix: " MONETDBPREFIX "\n"); > printf("Libraries:\n"); > #ifdef HAVE_LIBPCRE > printf(" libpcre: %s (%s)\n", pcre_version(), "@pcreversion@"); > #endif > #ifdef HAVE_OPENSSL > printf(" openssl: %s (%s)\n", OPENSSL_VERSION_TEXT, "@opensslversion@"); > #endif > printf("Compiled by: %s\n", "@builtby@"); > printf("Compilation: %s\n", "@compilercall@"); > #ifdef STATIC > printf("Linking : %s (static)\n", "@linkercall@"); > #else > printf("Linking : %s\n", "@linkercall@"); > #endif > exit(0); > } > > --- mserver5.mx.in DELETED --- -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] MAL access to tables created with SQL
by darabi＠web.de 13 Oct '07

13 Oct '07

Niels.Nes(a)cwi.nl wrote: > Well embedded_mal could still load the sql module and then extra > threads maybe created. Its more a warning, sql should be the one > making changes to the catalog. If you think of you application/extention > as do some sql to create tables, then exit sql, do querying with your > own frontend, it should work out just fine. Just to make sure, I understand correctly what you mean: - I can open an sql session with mclient and modify the tables with sql - then I open a mal session and query the bats/create the additional bats? My question is: can I open a mal session and somehow query/modify the sql catalog with the functions in the sql module? For example, is it legal to do this: $ mclient -l mal -d demo -H mal>include sql; mal>sql.start(); mal>bat1 := sql.bind("sys","_tables","id",1); The last statement hangs the client. If this is not allowed/possible in mclient, would I be able to do that with an embedded server, which is started with embedded_mal? > Anyway this means you'll have to get in depth knowledge on m5 and sql. > Which afcourse you can only obtain by reading all the manuals etc and > playing with the code, we won't have the time to give you this by email. Yes, this is what I'm trying to do and I didn't expect you to give me this be email. You (well, and the colleagues your organisation :) did enough to a) create this amazing piece of software b) make it open source, so everyone can learn from it c) write some extensive documentation (which is not usual for OSS) and I appreciate your doing so! Unfortunately, the first days are the hardest. So, bear with me if I ask stupid questions. I hope I could clarify what I would like to do: 1) use module sql (function in MonetDB-SQL/sql/src/backends/monet5/sql.mal) to do most of the work 2) use some additional bats to store information related to the objects If this is a bad idea, I would appreciate any hint. Thanks a lot Kambiz

1 1

[Monetdb-developers] MAL access to tables created with SQL
by darabi＠web.de 13 Oct '07

13 Oct '07

Hello, how is it possible to access in a MAL client session (mclient -l mal) the BATs of tables which were created previously in sql? I have tried different versions of bat.new, and bbp.bind. I tried different naming schemes (e.g. "tablename_columnname" which was mentioned somewhere in the docs), but I always get Referenced BAT not found And when I try to use sql.bind, I get SQLException:mvc:MAL_SCENARIO_PARSER not initialized Another question: is there a way of seeing the MAL translation of sql queries? EXPLAIN works great for SELECT statements, but it doesn't translate DDL expressions like CREATE TABLE. Many thanks Kambiz

4 8

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/test/BugDay_2005-10-06_2.8/Tests MapiClient-dump.SF-905851.SQL.py, SQL_2-20, 1.1.2.1, 1.1.2.2
by Stefan Manegold 13 Oct '07

13 Oct '07

Hi, this any all other tests that were ported from .sh/.bat to .py yesterday now fail on platforms that have a pyhton installed without subprocess; cf., http://monetdb.cwi.nl/testing/projects/monetdb/Stable/sql/.mTests103/index_… To solve the problem, in all newly added .py scripts, ======== import subprocess ======== should be replaced by ======== try: import subprocess except ImportError: # use private copy for old Python versions import MonetDB.subprocess26 as subprocess ======== as in Mtest.py[.in] to fall-back to the subprocess that we provide with MonetDB in case there is none available on the respective system. However, before checking in, please coordinate with Sjoerd, first! Stefan On Fri, Oct 12, 2007 at 09:53:41AM +0000, Romulo Goncalves wrote: > Update of /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.8/Tests > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv31626 > > Modified Files: > Tag: SQL_2-20 > MapiClient-dump.SF-905851.SQL.py > Log Message: > On windows the stdout and stderr of a python subprocess is not automatically redirect to > main process stdout and stderr. > The proper way is to use Popen and then use a PIPE to read the stderr > and stdout from the subprocess. > > NOTE: This script was tested on windows and Linux, and the test became green on both platforms. ;) > > > > Index: MapiClient-dump.SF-905851.SQL.py > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/test/BugDay_2005-10-06_2.8/Tests/MapiClient-dump.SF-905851.SQL.py,v > retrieving revision 1.1.2.1 > retrieving revision 1.1.2.2 > diff -u -d -r1.1.2.1 -r1.1.2.2 > --- MapiClient-dump.SF-905851.SQL.py 11 Oct 2007 14:40:50 -0000 1.1.2.1 > +++ MapiClient-dump.SF-905851.SQL.py 12 Oct 2007 09:53:39 -0000 1.1.2.2 > @@ -1,15 +1,21 @@ > -import os, sys > +import os, sys, subprocess > + > + > +def client(cmd): > + clt = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) > + sys.stdout.write(clt.stdout.read()) > + clt.stdout.close() > + sys.stderr.write(clt.stderr.read()) > + clt.stderr.close() > + > > > def main(): > clcmd = ("%s/bin/Mlog -x " % os.getenv('MONETDB_PREFIX')) + str(os.getenv('SQL_CLIENT')) + "< %s" % ('%s/JdbcClient_create_tables.sql' % os.getenv('RELSRCDIR')) > clcmd1 = ("%s/bin/Mlog -x " % os.getenv('MONETDB_PREFIX')) + str(os.getenv('SQL_CLIENT')) + "< %s" % ('%s/JdbcClient_inserts_selects.sql' % os.getenv('RELSRCDIR')) > clcmd2 = ("%s/bin/Mlog -x " % os.getenv('MONETDB_PREFIX')) + str(os.getenv('SQL_DUMP')) > - clt = os.popen(clcmd, 'w') > - clt.close() > - clt1 = os.popen(clcmd1, 'w') > - clt1.close() > - clt2 = os.popen(clcmd2, 'w') > - clt2.close() > + client(clcmd) > + client(clcmd1) > + client(clcmd2) > > main() > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

[Monetdb-developers] clients on windows 32...
by Romulo Goncalves 13 Oct '07

13 Oct '07

Why the option \h - show the readline history mclient.exe -lsql --help Is not available on windows? Is an design feature or a bug? Regards, Romulo

6 16

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/tools mcheckpoint, , 1.2, 1.3
by Sjoerd Mullender 12 Oct '07

12 Oct '07

How does this checkin message cover the change made here? On 2007-10-12 19:40, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/tools > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv9755/tools > > Modified Files: > mcheckpoint > Log Message: > Use a stricter runtime test to trap possible BAT bind mismatchs. > > > Index: mcheckpoint > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/tools/mcheckpoint,v > retrieving revision 1.2 > retrieving revision 1.3 > diff -u -d -r1.2 -r1.3 > --- mcheckpoint 9 Sep 2007 10:00:17 -0000 1.2 > +++ mcheckpoint 12 Oct 2007 17:40:25 -0000 1.3 > @@ -20,9 +20,8 @@ > function usage() > { > echo "Usage: mcheckpoint [options] " > - echo " --dbname=<database_name>" > + echo " --database=<database_name>" > echo " --ascii=<yes/no> default no" > - echo " --tool=<mapi/jdbc> default mapi" > echo " --dir=<directory>" > echo " --help" > exit 0 > @@ -99,8 +98,8 @@ > die "--tool: '${1#--tool=}' not understood, use 'mapi' or 'jdbc'" > esac > ;; > - --dbname=*) > - DBNAME="${1#--dbname=}" > + --database=*) > + DBNAME="${1#--database=}" > ;; > --dir=*) > CHKPNTDIR="${1#--dir=}" > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0