October 2008 - developers-list

[Monetdb-developers] communication problem between Mserver and mclient on OS X
by Jan Rittinger 31 Oct '08

31 Oct '08

Hi all, can you tell me what I need to change to make Mserver communicate with mclient on OS X. All I get is silence... $ Mserver --dbinit='module(pathfinder); mil_start();' # MonetDB Server v4.25.0 # based on GDK v1.25.0 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-, MonetDB B.V.. All rights reserved. # Compiled for i686-apple-darwin9.5.0/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah # MonetDB/XQuery module v0.25.0 loaded (default back-end is 'algebra') # XRPC administrative console at http://127.0.0.1:50001/admin MonetDB> $ mclient -lx q01.xq ^C (after 2 minutes) $ mclient -lmil q01.mil ^C (after 2 minutes) Running the same query through Mserver directly completes in under a second. $ pf q01.xq > q01.mil $ Mserver q01.mil # MonetDB Server v4.25.0 # based on GDK v1.25.0 # Copyright (c) 1993-July 2008, CWI. All rights reserved. # Copyright (c) August 2008-, MonetDB B.V.. All rights reserved. # Compiled for i686-apple-darwin9.5.0/32bit with 32bit OIDs; dynamically linked. # Visit http://monetdb.cwi.nl/ for further information. # PF/Tijah module v0.9.0 loaded. http://dbappl.cs.utwente.nl/pftijah # MonetDB/XQuery module v0.25.0 loaded (default back-end is 'algebra') # XRPC administrative console at http://127.0.0.1:50001/admin <?xml version="1.0" encoding="utf-8"?> <XQueryResult>Ayonca Vijaykrishnan</XQueryResult> MonetDB> Thanks, Jan -- Jan Rittinger Lehrstuhl Datenbanken und Informationssysteme Wilhelm-Schickard-Institut für Informatik Eberhard-Karls-Universität Tübingen http://www-db.informatik.uni-tuebingen.de/team/rittinger

2 4

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB4/src/modules/contrib malalgebra.mx, MonetDB_4-24, 1.8.2.1, 1.8.2.2
by Sjoerd Mullender 29 Oct '08

29 Oct '08

This type of change is not allowed on a stable branch. Changes of API are *not* allowed. Since we're going to abandon this stable branch soon, I'll let it pass. On 2008-10-28 18:49, Peter Boncz wrote: > Update of /cvsroot/monetdb/MonetDB4/src/modules/contrib > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv901/src/modules/contrib > > Modified Files: > Tag: MonetDB_4-24 > malalgebra.mx > Log Message: > changes to minimize sampling overhead (mostly in XML text index) > > - introduce bandmergejoin(), and leftmergejoin()/bandmergejoin() with a cutoff > > > > U malalgebra.mx > Index: malalgebra.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB4/src/modules/contrib/malalgebra.mx,v > retrieving revision 1.8.2.1 > retrieving revision 1.8.2.2 > diff -u -d -r1.8.2.1 -r1.8.2.2 > --- malalgebra.mx 10 Oct 2008 08:42:22 -0000 1.8.2.1 > +++ malalgebra.mx 28 Oct 2008 17:49:34 -0000 1.8.2.2 > @@ -43,7 +43,6 @@ > @:leftjoin(fetch,\nCAUTION: positional matches are assumed not to be out-of-bounds!!)@ > @:leftjoin(merge)@ > @:leftjoin(hash)@ > - > .COMMAND leftthetajoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] right, int mode) : > BAT[any::1,any::3] = CMDleftthetajoin; > "Hook directly into the 'leftthetajoin' implementation of the join. > @@ -58,6 +57,22 @@ > Also, for each left tuple, all matching right tuples will appear in their order > of appearrance in the right BAT. This property is handy for XQuery processing." > > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] inner, > + any::2 minus, any::2 plus, bit l_in, bit h_in) : > + BAT[any::1,any::3] = CMDbandmergejoin; > + "The bandjoin algorithm, but forced to use (left) mergejoin rather than nested loop" > + > + .COMMAND bandmergejoin ( BAT[any::1,any::2] outer, BAT[any::2,any::3] inner, > + any::2 minus, any::2 plus, bit l_in, bit h_in, lng limit) : > + BAT[lng,BAT] = CMDbandmergejoin_limit; > + "A bandjoin that uses the merge algorithm and limits the result size. > + Result generation is cut off after this point and the > + result [lng,BAT] holds [estimated_full_resultsize,limited_result]" > + > + .COMMAND leftmergejoin ( BAT[any::1,any::2] left, BAT[any::2,any::3] right, lng limit) : > + BAT[lng,BAT] = CMDleftmergejoin_limit; > + "leftmergejoin, but limit the result (by cutting off result generation) > + return a BAT[lng,BAT] containing [full_result_estimate,limited_result]" > > @= select > .COMMAND ord_@1select ( BAT[any::1,any::2] b, any::2 low, any::2 high) : > @@ -119,6 +134,7 @@ > #ifndef __MALALGEBRA_H__ > #define __MALALGEBRA_H__ > #include "gdk.h" > +#include "gdk_rangejoin.h" > > /* nothing much */ > > @@ -240,6 +256,40 @@ > return (*result = BATnlthetajoin(l, r, *mode, (size_t) * estimate)) ? GDK_SUCCEED : GDK_FAIL; > } > > +int > +CMDbandmergejoin(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi) > +{ > + return (*result = BATbandmergejoin(left, right, minus, plus, *li, *hi)) ? GDK_SUCCEED : GDK_FAIL; > +} > + > +int > +CMDbandmergejoin_limit(BAT **result, BAT *left, BAT *right, ptr minus, ptr plus, bit *li, bit *hi, lng* limit) > +{ > + size_t cutoff = *limit; > + *result = BATbandmergejoin_limit(left, right, minus, plus, *li, *hi, &cutoff); > + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid, FALSE); > + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > +int > +CMDleftmergejoin_limit(BAT **result, BAT *left, BAT *right, lng* limit) > +{ > + size_t cutoff = *limit; > + *result = BATleftmergejoin_limit(left, right, cutoff, &cutoff); > + if (*result) { > + bat bid = (*result)->batCacheid; > + *result = BATnew(TYPE_lng, TYPE_bat, 1); > + if (*result) BUNins(*result, &cutoff, &bid, FALSE); > + BBPunfix(bid); > + } > + return (*result)?GDK_SUCCEED:GDK_FAIL; > +} > + > @= selectcmd > int CMDord_@1select1(BAT **result, BAT* b, ptr value) { > return (*result = BAT_select_(b, value, 0, TRUE, TRUE, @2, FALSE, TRUE))?GDK_SUCCEED:GDK_FAIL; > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

4 11

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal bbp.mx, , 1.127, 1.128
by Stefan Manegold 27 Oct '08

27 Oct '08

On Mon, Oct 27, 2008 at 06:35:34AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv4663 > > Modified Files: > bbp.mx > Log Message: > Silence compilers > > > U bbp.mx > Index: bbp.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/bbp.mx,v > retrieving revision 1.127 > retrieving revision 1.128 > diff -u -d -r1.127 -r1.128 > --- bbp.mx 25 Oct 2008 22:08:09 -0000 1.127 > +++ bbp.mx 27 Oct 2008 06:35:32 -0000 1.128 > @@ -867,7 +867,7 @@ > int i; > char buf[MAXPATHLEN]; > char cwd[MAXPATHLEN]; > - getcwd(cwd, MAXPATHLEN); > + (void) getcwd(cwd, MAXPATHLEN); This is not "sufficient". With optimization enabled, gcc still (correctly) complains as follows: ======== gcc -DHAVE_CONFIG_H -I. -I../../.. -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal -I../atoms -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../atoms -I../kernel -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../kernel -I../../mal -I/ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/../../mal -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-optimize_--disable-debug_--disable-assert/include/MonetDB -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-optimize_--disable-debug_--disable-assert/include/MonetDB/mapilib -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-optimize_--disable-debug_--disable-assert/include/MonetDB -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-optimize_--disable-debug_--disable-assert/include/MonetDB/common -I/ufs/manegold/_/scratch0/Monet/Testing/Current/prefix.--enable-strict_--disable-optimize_--disable-debug_--disable-assert/include/MonetDB/gdk -I/usr/include/libxml2 -DLIBBBP -O2 -Wall -Wextra -std=c99 -Werror-implicit-function-declaration -Werror -Wpointer-arith -Wdeclaration-after-statement -Wundef -Wp,-D_FORTIFY_SOURCE=2 -D_REENTRANT -c bbp.c -fPIC -DPIC -o .libs/lib_bbp_la-bbp.o cc1: warnings being treated as errors /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/bbp.mx: In function 'CMDbbpLocation': /ufs/manegold/_/scratch0/Monet/Testing/Current/source/MonetDB5/src/modules/mal/bbp.mx:874: warning: ignoring return value of 'getcwd', declared with attribute warn_unused_result make[8]: *** [lib_bbp_la-bbp.lo] Error 1 ======== getcwd() can fail (e.g., incase the current directpry is not readable for the user; see `man getcwd` for details), and hence, the return value needs to be checked, and proper measures need to be taken in case getcwd() does fail (e.g., have default fall-back or trigger warning or error). Stefan > b = BATnew(TYPE_int, TYPE_str, BBPsize); > if (b == 0) > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_posix.mx, , 1.157, 1.158
by Stefan Manegold 25 Oct '08

25 Oct '08

On Sat, Oct 25, 2008 at 07:30:50AM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv30373 > > Modified Files: > gdk_posix.mx > Log Message: > The memory map table maintained for the vmtrim > should be aligned to the number of mapped files. > In more recent applications, such as the SkyServer > this may be around 2000 BATs easily. > For the time being the limit of 100 is increased to 4096 > > The remainder are additions of '#' in error messages > to prepare for problems in the API. just an idea (without background knowledge): shouldn't we introduce a new symbol/chracter in the protocol to indicate warnings and informative message? '!' indicates errors (obviously never to be ignored) '#' indicates (usually to be ignored) comments '?' to be added to indicate warnings (and "informative messages)? '?' to be added for ... ?? Stefan > > > > U gdk_posix.mx > Index: gdk_posix.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_posix.mx,v > retrieving revision 1.157 > retrieving revision 1.158 > diff -u -d -r1.157 -r1.158 > --- gdk_posix.mx 10 Oct 2008 08:50:35 -0000 1.157 > +++ gdk_posix.mx 25 Oct 2008 07:30:48 -0000 1.158 > @@ -389,7 +389,7 @@ > * hope the VM paging algorithm behaves better than Linux which just runs off > * the cliff and if MonetDB does not prevent RSS from being too high, enters coma. > * > - * We will only eb able to sensibly test this on Windows64. On Windows32, mmap sizes > + * We will only be able to sensibly test this on Windows64. On Windows32, mmap sizes > * do not significantly exceed RAM sizes so MonetDB swapping actually will not happen > * (of course, you've got this nasty problem of VM fragemntation and failing mmaps instead). > * > @@ -488,6 +488,10 @@ > * to the cursors (i.e. pointers to b->batBuns->free or the cursors > * in radix-cluster), which an enhanced version of this thread might > * take into account. > + * > + * [Kersten] The memory map table should be aligned to the number of > + * mapped files. In more recent applications, such as the SkyServer > + * this may be around 2000 BATs easily. > */ > > #ifdef HAVE_PTHREAD_H > @@ -519,7 +523,7 @@ > #endif > > #define MT_MMAP_TILE (1<<27) > -#define MT_MMAP_BUFSIZE 100 > +#define MT_MMAP_BUFSIZE 4096 > MT_mmap_t MT_mmap_tab[MT_MMAP_BUFSIZE]; > int MT_mmap_cur = -1, MT_mmap_first = -1, MT_mmap_free = 0; > > @@ -598,7 +602,7 @@ > if (MT_mmap_cur == -1) > MT_mmap_cur = i; > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "MT_mmap_new: %s fd=%d\n", path, fd); > + stream_printf(GDKerr, "#MT_mmap_new: %s fd=%d\n", path, fd); > #endif > strncpy(MT_mmap_tab[i].path, path, 128); > MT_mmap_tab[i].base = base; > @@ -642,13 +646,13 @@ > /* tell the OS quite clearly that you want to drop this */ > ret = posix_fadvise(MT_mmap_tab[victim].fd, 0LL, MT_mmap_tab[victim].len, POSIX_FADV_DONTNEED); > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "MT_mmap_del: posix_fadvise(%s,fd=%d,%uMB,POSIX_FADV_DONTNEED) = %d\n", MT_mmap_tab[victim].path, MT_mmap_tab[victim].fd, (unsigned int) (MT_mmap_tab[victim].len >> 20), ret); > + stream_printf(GDKerr, "#MT_mmap_del: posix_fadvise(%s,fd=%d,%uMB,POSIX_FADV_DONTNEED) = %d\n", MT_mmap_tab[victim].path, MT_mmap_tab[victim].fd, (unsigned int) (MT_mmap_tab[victim].len >> 20), ret); > #endif > } > #endif > ret = close(MT_mmap_tab[victim].fd); > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "MT_mmap_del: close(%s fd=%d) = %d\n", MT_mmap_tab[victim].path, MT_mmap_tab[victim].fd, ret); > + stream_printf(GDKerr, "#MT_mmap_del: close(%s fd=%d) = %d\n", MT_mmap_tab[victim].path, MT_mmap_tab[victim].fd, ret); > #endif > MT_mmap_tab[victim].next = MT_mmap_free; > MT_mmap_empty(victim); > @@ -674,7 +678,7 @@ > if (MT_mmap_tab[i].fd >= 0) { > ret = posix_fadvise(MT_mmap_tab[i].fd, 0, len, advice); > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "MT_fadvise: posix_fadvise(%s,fd=%d,%uMB,%d) = %d\n", MT_mmap_tab[i].path, MT_mmap_tab[i].fd, (unsigned int) (len >> 20), advice, ret); > + stream_printf(GDKerr, "#MT_fadvise: posix_fadvise(%s,fd=%d,%uMB,%d) = %d\n", MT_mmap_tab[i].path, MT_mmap_tab[i].fd, (unsigned int) (len >> 20), advice, ret); > #endif > } > } > @@ -697,7 +701,7 @@ > int ret = posix_madvise(MT_mmap_tab[i].base + off, MT_MMAP_TILE, POSIX_MADV_DONTNEED); > > if (err) { > - stream_printf(err, "MT_mmap_unload_tile: posix_madvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > + stream_printf(err, "#MT_mmap_unload_tile: posix_madvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > MT_mmap_tab[i].path, (unsigned int) (off >> 20), > (unsigned int) (MT_MMAP_TILE >> 20), MT_mmap_tab[i].fd, ret); > } > @@ -706,7 +710,7 @@ > /* tell the OS quite clearly that you want to drop this */ > ret = posix_fadvise(MT_mmap_tab[i].fd, off, MT_MMAP_TILE, POSIX_FADV_DONTNEED); > if (err) { > - stream_printf(err, "MT_mmap_unload_tile: posix_fadvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > + stream_printf(err, "#MT_mmap_unload_tile: posix_fadvise(%s,off=%uMB,%uMB,fd=%d,POSIX_MADV_DONTNEED) = %d\n", > MT_mmap_tab[i].path, (unsigned int) (off >> 20), > (unsigned int) (MT_MMAP_TILE >> 20), MT_mmap_tab[i].fd, ret); > } > @@ -721,7 +725,7 @@ > > /* save to disk an 128MB tile, and observe how long this takes */ > if (err) { > - stream_printf(err, "MT_mmap_save_tile: msync(%s,off=%uM,%u,SYNC)...\n", > + stream_printf(err, "#MT_mmap_save_tile: msync(%s,off=%uM,%u,SYNC)...\n", > MT_mmap_tab[i].path, (unsigned int) (tile >> 20), > (unsigned int) (MT_MMAP_TILE >> 20)); > } > @@ -729,7 +733,7 @@ > ret = MT_msync(MT_mmap_tab[i].base, tile, MT_MMAP_TILE, MMAP_SYNC); > t = GDKms() - t; > if (err) { > - stream_printf(err, "MT_mmap_save_tile: msync(%s,tile=%uM,%uM,SYNC) = %d (%dms)\n", > + stream_printf(err, "#MT_mmap_save_tile: msync(%s,tile=%uM,%uM,SYNC) = %d (%dms)\n", > MT_mmap_tab[i].path, (unsigned int) (tile >> 20), > (unsigned int) (MT_MMAP_TILE >> 20), ret, t); > } > @@ -768,7 +772,7 @@ > > (void) pthread_mutex_lock(&MT_mmap_lock); > if (err) { > - stream_printf(err, "MT_mmap_trim(%u MB): rss = %u MB\n", (unsigned int) ((target) >> 20), (unsigned int) (rss >> 20)); > + stream_printf(err, "#MT_mmap_trim(%u MB): rss = %u MB\n", (unsigned int) ((target) >> 20), (unsigned int) (rss >> 20)); > } > if (rss > target) { > /* try to selectively unload pages from the writable regions */ > @@ -1085,7 +1089,7 @@ > int ret = munmap(p, len); > > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "munmap(" LLFMT "," LLFMT ",%d) = %d\n", (long long) p, (long long) len, ret); > + stream_printf(GDKerr, "#munmap(" LLFMT "," LLFMT ",%d) = %d\n", (long long) p, (long long) len, ret); > #endif > MT_mmap_del(p, len); > return ret; > @@ -1097,7 +1101,7 @@ > int ret = msync(((char*) p)+off, len, (mode & MMAP_SYNC) ? MS_SYNC : ((mode & MMAP_ASYNC) ? MS_ASYNC : MS_INVALIDATE)); > > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "msync(" LLFMT "," LLFMT ",%s) = %d\n", (long long) p, (long long) len, (mode & MMAP_SYNC) ? "MS_SYNC" : ((mode & MMAP_ASYNC) ? "MS_ASYNC" : "MS_INVALIDATE"), ret); > + stream_printf(GDKerr, "#msync(" LLFMT "," LLFMT ",%s) = %d\n", (long long) p, (long long) len, (mode & MMAP_SYNC) ? "MS_SYNC" : ((mode & MMAP_ASYNC) ? "MS_ASYNC" : "MS_INVALIDATE"), ret); > #endif > if (ret < 0) > return errno; > @@ -1110,7 +1114,7 @@ > int ret = posix_madvise(p, len, advise); > > #ifdef MMAP_DEBUG > - stream_printf(GDKerr, "posix_madvise(" PTRFMT "," SZFMT ",%d) = %d\n", PTRFMTCAST p, len, advise, ret); > + stream_printf(GDKerr, "#posix_madvise(" PTRFMT "," SZFMT ",%d) = %d\n", PTRFMTCAST p, len, advise, ret); > #endif > if (MT_fadvise(p, len, advise)) > ret = -1; > @@ -1468,7 +1472,7 @@ > } > if (heapstatus == _HEAPBADPTR || heapstatus == _HEAPBADBEGIN || heapstatus == _HEAPBADNODE) { > > - stream_printf(GDKerr, "mallinfo(): heap is corrupt."); > + stream_printf(GDKerr, "#mallinfo(): heap is corrupt."); > } > _heapmin(); > return _ret; > @@ -1604,7 +1608,7 @@ > return; > > if (!FindClose(dir->find_file_handle)) > - stream_printf(GDKerr, "rewinddir(): FindClose() failed\n"); > + stream_printf(GDKerr, "#rewinddir(): FindClose() failed\n"); > > mask = malloc(strlen(dir->dir_name) + 3); > sprintf(mask, "%s\\*", dir->dir_name); > @@ -2116,7 +2120,7 @@ > p = (void *) VirtualAlloc(a, size, MEM_COMMIT | mode, PAGE_READWRITE); > (void) pthread_mutex_unlock(&MT_mmap_lock); > if (p == NULL) { > - stream_printf(GDKerr, "VirtualAlloc(" PTRFMT "," SZFMT ",MEM_COMMIT,PAGE_READWRITE): failed\n", PTRFMTCAST a, size); > + stream_printf(GDKerr, "#VirtualAlloc(" PTRFMT "," SZFMT ",MEM_COMMIT,PAGE_READWRITE): failed\n", PTRFMTCAST a, size); > } > return p; > } > @@ -2126,9 +2130,9 @@ > MT_vmfree(void *p, size_t size) > { > if (VirtualFree(p, size, MEM_DECOMMIT) == 0) > - stream_printf(GDKerr, "VirtualFree(" PTRFMT "," SZFMT ",MEM_DECOMMIT): failed\n", PTRFMTCAST p, size); > + stream_printf(GDKerr, "#VirtualFree(" PTRFMT "," SZFMT ",MEM_DECOMMIT): failed\n", PTRFMTCAST p, size); > if (VirtualFree(p, 0, MEM_RELEASE) == 0) > - stream_printf(GDKerr, "VirtualFree(" PTRFMT ",0,MEM_RELEASE): failed\n", PTRFMTCAST p); > + stream_printf(GDKerr, "#VirtualFree(" PTRFMT ",0,MEM_RELEASE): failed\n", PTRFMTCAST p); > } > > void * > @@ -2149,7 +2153,7 @@ > size_t ret = VirtualFree(p + newsize, oldsize - newsize, MEM_DECOMMIT); > > if (ret == 0) > - stream_printf(GDKerr, "VirtualFree(" PTRFMT "," SSZFMT ",MEM_DECOMMIT): failed\n", PTRFMTCAST(p + newsize), (ssize_t) (oldsize - newsize)); > + stream_printf(GDKerr, "#VirtualFree(" PTRFMT "," SSZFMT ",MEM_DECOMMIT): failed\n", PTRFMTCAST(p + newsize), (ssize_t) (oldsize - newsize)); > } else if (oldsize < newsize) { > (void) pthread_mutex_lock(&MT_mmap_lock); > a = (char *) VirtualAlloc(p, newsize, MEM_COMMIT, PAGE_READWRITE); > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 2

[Monetdb-developers] slow query log
by Stefan de Konink 21 Oct '08

21 Oct '08

Is there currently a syntax to log every incoming SQL query to a file? I found options like: 1) ODBCDEBUG [1] 2) mapi_log(...) [2] But is there something easy to capture all incoming query with their timings; so a real slow queries log can be created? In the Skyserver presentation [3] I see 'Query log' analysis; I'm really interested in how this is realised. I would like to do an analysis on all queries Mapserver generates by its GDAL/OGR layer, since I have implemented this interface using ODBC (yes, I was too lazy to implement a native connector) some logging would be possible, but not concurrently, hence the file would be overwritten. If nothing is in place, I'm happy to give it a try and implement it. Stefan [1] <http://article.gmane.org/gmane.comp.db.monetdb.devel/1381> [2] <http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/Mapi-Function-Refe…> [3] <www.win.tue.nl/~tcalders/presentations/ivanova.ppt>

3 2

[Monetdb-developers] x100 development
by Ross Bates 17 Oct '08

17 Oct '08

Hi All - is there any chance we could get a comment or two from the core monetdb developers about the future of x100? I'm just curious if it's going to be an extension, a separate product, commercial, open source, etc..... To be honest I'm fine with whatever the answer is... you could tell me it's never going to see the light of day and that would be ok. Just wondering. Best Regards, Ross

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime shredder.mx, , 1.142, 1.143
by Sjoerd Mullender 16 Oct '08

16 Oct '08

On 2008-10-16 08:21, Jan Flokstra wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv11903 > > Modified Files: > shredder.mx > Log Message: > - fix strncat() size overflow compilation problem on some architectures. > > > > U shredder.mx > Index: shredder.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/shredder.mx,v > retrieving revision 1.142 > retrieving revision 1.143 > diff -u -d -r1.142 -r1.143 > --- shredder.mx 15 Oct 2008 12:45:12 -0000 1.142 > +++ shredder.mx 16 Oct 2008 06:21:44 -0000 1.143 > @@ -1324,7 +1324,7 @@ > > const xmlChar *abs_SystemID; > > - char locbuff[FILENAME_MAX]; > + char locbuff[FILENAME_MAX+1]; > > if (SystemID && shredCtx->base_dir && !isURL((char*)SystemID) && !((*SystemID=='/') || (*SystemID=='\\'))) { > /* is is a relative name which should be prefixed with the > @@ -1333,7 +1333,7 @@ > /* base_dir */ > locbuff[0] = '\0'; > strncat(&locbuff[0],shredCtx->base_dir,FILENAME_MAX); > - strncat(&locbuff[0],(char*)SystemID,FILENAME_MAX); > + strncat(&locbuff[0],(char*)SystemID,FILENAME_MAX-strlen(shredCtx->base_dir)); > abs_SystemID = (const xmlChar *)&locbuff[0]; > } else > abs_SystemID = SystemID; > This is not very good from a defensive programming point of view, nor from an efficiency point of view. Plus, the compiler complains about the second strncat. Better, instead of the three lines starting with locbuff[0] = '\0'; do snprintf(locbuff, sizeof(locbuff), "%s%s", shredCtx->base_dir, (char *) SystemID); The problem is, strlen(shredCtx->base_dir) could be greater than FILENAME_MAX, in which case the third argument to the second strncat could overflow and become extremely large (it's an unsigned, so it won't be negative). -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/conf monetdb5.conf.in, , 1.34, 1.35
by Fabian Groffen 13 Oct '08

13 Oct '08

On 12-10-2008 09:38:04 +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/conf > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv19179 > > Modified Files: > monetdb5.conf.in > Log Message: > Move the MonetDB related files to a clearly marked achorpoint default > -mero_msglog=@localstatedir@@DIRSEP@log@DIRSEP@merovingian.log > -mero_errlog=@localstatedir@@DIRSEP@log@DIRSEP@merovingian.log > +mero_msglog=@localstatedir@@DIRSEP@MonetDB5@DIRSEP@merovingian.log > +mero_errlog=@localstatedir@@DIRSEP@MonetDB5@DIRSEP@merovingian.log May I suggest to leave in "log" in here. Traditionally, logfiles are stored in /var/log. No real need to deviate from that, I think? /var/log/MonetDB5 would be ok, I guess. > -mero_pidfile=@localstatedir@@DIRSEP@run@DIRSEP@merovingian.pid > +mero_pidfile=@localstatedir@@DIRSEP@MonetDB5@DIRSEP@merovingian.pid Pidfiles are traditionally stored in /var/run. I think this is not a good idea. Note that both changes don't make "slotting" of monetdb any more possible than it is now.

4 5

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_logger.mx, , 1.26, 1.27
by Stefan Manegold 12 Oct '08

12 Oct '08

On Sun, Oct 12, 2008 at 10:24:45AM +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB/src/gdk > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv23370/src/gdk > > Modified Files: > gdk_logger.mx > Log Message: > default log directories moved to ${gdk_dbfarm} > Logger now extends the logdir with ${dbname}/$log_name > where $log_name == sql or xquery (passed by the call to logger_create) > (so now everything is again under the dbfarm base directory by default > but can still be set to anything else using the logdir variables) Niels, (mainly to be able to anticipate how your changes will affect testing, and whether I need to do any changes in Mtest.py, TestTools, and/or */RunMtest*) in case we do use/set sql_logdir and/or xquery_logdir, what wil be the resulting path for the log? ${*_logdir} or ${*_logdir}/${dbname}/${log_name} ? in other words, is the default for sql_logdir & xquery_logdir now ${gdk_dbfarm} or ${gdk_dbfarm}/${dbname}/${log_name} ? Thanks! Stefan > Needed to patch mo_substitute (now rewrites all ${varname} and for windows > also all %varname%), same for monet_options.py > > > > U gdk_logger.mx > Index: gdk_logger.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_logger.mx,v > retrieving revision 1.26 > retrieving revision 1.27 > diff -u -d -r1.26 -r1.27 > --- gdk_logger.mx 10 Oct 2008 08:50:31 -0000 1.26 > +++ gdk_logger.mx 12 Oct 2008 10:24:43 -0000 1.27 > @@ -1037,7 +1037,7 @@ > > lg->tid = 0; > > - snprintf(filename, BUFSIZ, "%s%c%s%c", logdir, DIR_SEP, dbname, DIR_SEP); > + snprintf(filename, BUFSIZ, "%s%c%s%c%s%c", logdir, DIR_SEP, dbname, DIR_SEP, fn, DIR_SEP); > lg->fn = GDKstrdup(fn); > lg->dir = GDKstrdup(filename); > lg->phandler = phandler; > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 merovingian.mx, , 1.58, 1.59 monetdb.mx, , 1.27, 1.28
by Fabian Groffen 12 Oct '08

12 Oct '08

On 12-10-2008 10:31:31 +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/sql/src/backends/monet5 > In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv23781 > > Modified Files: > merovingian.mx monetdb.mx > Log Message: > Use a known location for the lock file. known? > - snprintf(lockfile, 512, "%s/.merovingian_lock", dbfarm); > + /* the lock file is placed in the same directory as the pid */ > + strcpy(lockfile, pidfilename); > + c= strrchr(lockfile,'/'); > + if (c) > + strcpy(c+1,".merovingian_lock"); > + } else > + strcat(lockfile, ".merovingian_lock"); Why would you move out the lockfile from the dbfarm, and put it in /var/run instead? merovingian manages a dbfarm, hence the most obvious place to lock is the dbfarm itself. You now allow a user to use a different pidfile, but the same dbfarm and get two merovingians managing the same dbfarm, which seems like NOT a good idea to me. Below changes seem unrelated to the commit message to me. It indeed fixes one bug. However, if you change the input prompt to ignore an empty response, you probably want to change the documentation accordingly, as it states that the default is No, while you now require the user to at least type something. (Yes I'm picky, because everything was very well thought over.) > U monetdb.mx > Index: monetdb.mx > =================================================================== > RCS file: /cvsroot/monetdb/sql/src/backends/monet5/monetdb.mx,v > retrieving revision 1.27 > retrieving revision 1.28 > diff -u -d -r1.27 -r1.28 > --- monetdb.mx 10 Oct 2008 08:54:15 -0000 1.27 > +++ monetdb.mx 12 Oct 2008 10:31:29 -0000 1.28 > @@ -24,7 +24,7 @@ > The monetdb utility is designed to be the interface for the DBA to the > dbfarm. Creating or deleting databases next to retrieving status > information about them are the primary goals of this tool. > - > +@{ > @h > #define TOOLKIT_VERSION "0.4" > > @@ -282,11 +282,13 @@ > } > /* demo | state | uptime | health */ > secondsToString(avg, uplog.avguptime, 0); > - printf("%-14s %s %12s %3d%%, %3s %s\n", > - dbname, > - state, uptime, > - 100 - (uplog.crashcntr * 100 / uplog.startcntr), > - avg, crash); > + printf("%-14s %s %12s", > + dbname, state, uptime); > + if (uplog.startcntr) > + printf(" %3d%%, %3s %s", > + 100 - (uplog.crashcntr * 100 / uplog.startcntr), > + avg, crash); > + printf("\n"); > } else if (mode == 2) { > /* long mode */ > char *state; > @@ -1076,9 +1078,10 @@ > printf("you are about to remove database '%s'\n", dbname); > printf("ALL data in this database will get lost, " > "are you sure? [y/N] "); > - if (scanf("%c", &answ) >= 1 && (answ == 'y' || answ == 'Y')) { > - /* do it! */ > - } else { > + answ = getchar(); > + while (answ!= '\n' && getchar()!= '\n') > + ; > + if (answ != 'y' && answ != 'Y') { > printf("battle control terminated\n"); > exit(0); > } > @@ -1361,4 +1364,5 @@ > return(0); > } > > +@} > /* vim:set ts=4 sw=4 noexpandtab: */ > > > ------------------------------------------------------------------------- > This SF.Net email is sponsored by the Moblin Your Move Developer's challenge > Build the coolest Linux based applications with Moblin SDK & win great prizes > Grand prize is a trip for two to an Open Source event anywhere in the world > http://moblin-contest.org/redirect.php?banner_id=100&url=/ > _______________________________________________ > Monetdb-sql-checkins mailing list > Monetdb-sql-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-sql-checkins

2 2