February 2007 - developers-list

[Monetdb-developers] windows compilaton...
by Romulo Goncalves 26 Feb '07

26 Feb '07

During the compilation of clients (cvs head) I got the following errors: cd "MonetSQLdb" && nmake /nologo "prefix=C:\cygwin\home\goncalve\MonetDB\MonetDB_SQL\clients\NT" all cl -GF -W3 -wd4273 -wd4102 -MD -nologo -Zi -Ox -I. -I.\..\..\.. "-IC:\Pthreads\include" -I.\..\..\..\..\src\python\Cim pl "-I..\..\mapilib" -I.\..\..\..\..\..\MonetDB\NT\include -I.\..\..\..\..\..\MonetDB\NT\include\common -I.\..\..\..\..\..\Mon etDB\NT\include\gdk -IC:\Python25\include -LD -Fe_MapiLib.pyd MapiLib.py.obj /link ..\..\mapilib\libMapi.lib C:\Python25\libs \python25.lib Creating library _MapiLib.lib and object _MapiLib.exp MapiLib.py.obj : error LNK2019: unresolved external symbol __imp__mapi_setBlocking referenced in function __wrap_mapi_setBlock ing MapiLib.py.obj : error LNK2019: unresolved external symbol __imp__mapi_is_blocked referenced in function __wrap_mapi_is_blocke d _MapiLib.pyd : fatal error LNK1120: 2 unresolved externals NMAKE : fatal error U1077: 'cl' : return code '0x2' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: 'cd' : return code '0x2' Stop. NMAKE : fatal error U1077: '"E:\Visual Studio .NET 2003\VC7\BIN\nmake.exe"' : return code '0x2' Stop. I already updated and compiled buildtools and MonetDB. Maybe Niels or Fabian know the reason of this error. Regards, Romulo

3 8

[Monetdb-developers] check-ins
by Henning Rode 26 Feb '07

26 Feb '07

hej amsterdam, what's the status of the release? or more important, when are we allowed to do check-ins again? and second question: which machine on the test web is actually running icc? i remember we checked the test-web on friday and besides the rare sun-os all systems looked pretty green on our test-cases. so, where exactly should we have spotted the icc compilation problems? best -henning

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime/xrpc/export export.xq, 1.1.2.1, 1.1.2.2
by Sjoerd Mullender 25 Feb '07

25 Feb '07

On 02/25/2007 11:02 PM, Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/runtime/xrpc/export > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv16912 > > Modified Files: > Tag: XQuery_0-16 > export.xq > Log Message: > A copy-paste error in the NS identifier. > > Since this change won't affect compilation, I check it in without > explicit permission from Sjoerd. :P It does, however, affect the binary and source distributions that we make, and so you should have asked permission. But you're in luck: I'm going to use tonight's build. > > > Index: export.xq > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/xrpc/export/export.xq,v > retrieving revision 1.1.2.1 > retrieving revision 1.1.2.2 > diff -u -d -r1.1.2.1 -r1.1.2.2 > --- export.xq 21 Feb 2007 16:25:48 -0000 1.1.2.1 > +++ export.xq 25 Feb 2007 22:02:54 -0000 1.1.2.2 > @@ -11,5 +11,5 @@ > > (: =================== HTTP =================== :) > > -declare function admin:GET($uri as xs:string) > +declare function export:GET($uri as xs:string) > { doc($uri) }; > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] buildtools/autogen/autogen am.py, 1.30.2.1, 1.30.2.2
by Sjoerd Mullender 25 Feb '07

25 Feb '07

On 02/25/2007 01:51 PM, Niels Nes wrote: > Update of /cvsroot/monetdb/buildtools/autogen/autogen > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv2017/autogen/autogen > > Modified Files: > Tag: MonetDB_1-16 > am.py > Log Message: > we need to distribute *.def files as the rules to build them are part of > the rules.mk file (which isn't distributed) Did you really mean to check in those print statements? > > Index: am.py > =================================================================== > RCS file: /cvsroot/monetdb/buildtools/autogen/autogen/am.py,v > retrieving revision 1.30.2.1 > retrieving revision 1.30.2.2 > diff -u -d -r1.30.2.1 -r1.30.2.2 > --- am.py 13 Feb 2007 10:13:23 -0000 1.30.2.1 > +++ am.py 25 Feb 2007 12:51:11 -0000 1.30.2.2 > @@ -19,7 +19,7 @@ > from codegen import find_org > > #automake_ext = ['c', 'h', 'y', 'l', 'glue.c'] > -automake_ext = ['o', 'lo', 'c', 'h', 'tab.c', 'tab.h', 'yy.c', 'glue.c', 'proto.h', 'py.c', 'pm.c', ''] > +automake_ext = ['o', 'lo', 'c', 'h', 'tab.c', 'tab.h', 'yy.c', 'glue.c', 'proto.h', 'py.c', 'pm.c', 'def', ''] > buildtools_ext = ['mx', 'm', 'y', 'l', 'brg'] > > am_assign = "+=" > @@ -153,9 +153,11 @@ > pf = f > > # built source if has dep and ext != cur ext > + print(pf) > if not(cond) and deps.has_key(pf) and pf not in am['BUILT_SOURCES']: > pfb, pfext = split_filename(pf) > sfb, sfext = split_filename(deps[pf][0]) > + print('test %s' % pf) > if sfext != pfext: > if pfext in automake_ext: > dist = None > @@ -622,6 +624,7 @@ > def am_library(fd, var, libmap, am): > name = var[4:] > > + print("am_library\n"); > sep = "" > pref = 'lib' > if libmap.has_key("NAME"): > @@ -702,6 +705,7 @@ > deps = [] > for target in libmap['TARGETS']: > t, ext = split_filename(target) > + print(t,ext) > if ext in scripts_ext: > if target not in SCRIPTS: > SCRIPTS.append(target) > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime Makefile.ag, 1.70, 1.71 levelsteps.mx, 1.12, 1.13 ll_staircasejoin.mx, 1.52, 1.53 ll_upwards.mx, 1.19, 1.20 pathfinder.mx, 1.297, 1.298 shredder.mx, 1.116, 1.117
by Jens Teubner 23 Feb '07

23 Feb '07

On Fri, Feb 23, 2007 at 12:51:45PM +0000, Niels Nes wrote: > [...] > > +@h > +#ifndef _LEVELSTEPS_H_ > +#define _LEVELSTEPS_H_ > + > +#define XML_DEPTH_MAX 128 > + > +#endif /* _LEVELSTEPS_H_ */ > > [...] It is a bad habit to use macro names that start with an underscore. Such names are reserved for the C library only. For a reference, see, e.g., http://www.gnu.org/software/libc/manual/html_node/Reserved-Names.html Jens -- Jens Teubner Technische Universitaet Muenchen, Department of Informatics D-85748 Garching, Germany Tel: +49 89 289-17259 Fax: +49 89 289-17263 Emacs is a nice OS - but it lacks a good text editor. That's why I am using Vim.

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah pftijah.mx, 1.92.2.17, 1.92.2.18
by Henning Rode 23 Feb '07

23 Feb '07

hej roel, zijn die niet sowieso al gesorteerd? tenminste moeten ze dat eigenlijk altijd zijn. groet -henning Roel van Os wrote: > Update of /cvsroot/monetdb/pathfinder/modules/pftijah > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv12912 > > Modified Files: > Tag: XQuery_0-16 > pftijah.mx > Log Message: > PF/Tijah: fix some sortedness issues. These only seem to occur on large-scale > querying, not on the current Tests. > > > Index: pftijah.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/pftijah.mx,v > retrieving revision 1.92.2.17 > retrieving revision 1.92.2.18 > diff -u -d -r1.92.2.17 -r1.92.2.18 > --- pftijah.mx 23 Feb 2007 09:10:04 -0000 1.92.2.17 > +++ pftijah.mx 23 Feb 2007 12:12:57 -0000 1.92.2.18 > @@ -1371,7 +1371,7 @@ > } > > > - var ctx := region; > + var ctx := region.sort(); > var pre_size := load( "tj_" + collName + "_size1"); > var desc := scj_desc( pre_size, ctx, count(pre_size) ); > var result := nodes( desc ).project( dbl(scoreBase) ); > @@ -1589,7 +1589,7 @@ > var pre_size := bat("tj_" + collName + "_size" + ind); > > # evaluate doc/term (anc/desc) relationship > - var elem_tid := _containing_desc3(left.mark(0@0), pre_tid, pre_size); > + var elem_tid := _containing_desc3(left.sort().mark(0@0), pre_tid, pre_size); > pre_tid := nil; > if (elem_tid.count() = 0) {return new(oid,dbl);} > > @@ -1632,7 +1632,7 @@ > > # evaluate doc/term (anc/desc) relationship > var t2 := time(); > - var elem_tid := _containing_desc3(left.mark(0@0), pre_tid, pre_size); > + var elem_tid := _containing_desc3(left.sort().mark(0@0), pre_tid, pre_size); > var t3 := time(); > pre_tid := nil; > if (elem_tid.count() = 0) {return new(oid,dbl);} > @@ -1679,7 +1679,7 @@ > > # evaluate doc/term (anc/desc) relationship > var t2 := time(); > - var elem_tid := _containing_desc(left.mark(0@0), pre_tid, pre_size); > + var elem_tid := _containing_desc(left.sort().mark(0@0), pre_tid, pre_size); > var t3 := time(); > pre_tid := nil; > if (elem_tid.count() = 0) {return new(oid,dbl);} > @@ -1870,7 +1870,7 @@ > var pre_size := bat("tj_" + collName + "_size1"); > > # See which document contain the query terms we create a bat of [doc, term-id]: > - var doc_tid := _containing_desc(ctx.mark(0(a)0).sort(), tid_pre.reverse(), pre_size); > + var doc_tid := _containing_desc(ctx.sort().mark(0@0), tid_pre.reverse(), pre_size); > > # len(doc): [doc, size] > var doc_len := [dbl](bat("tj_" + collName + "_size1").semijoin(doc_tid)); > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins

1 0

[Monetdb-developers] SQL5 compilation error
by Ying Zhang 22 Feb '07

22 Feb '07

Hi, (Current branch) checkout and compile SQL results in the following error: cc1: warnings being treated as errors ../../../../src/backends/monet5/sql_scenario.mx: In function 'monet5_freestack': ../../../../src/backends/monet5/sql_scenario.mx:94: warning: passing argument 1 of 'GDKfree' makes pointer from integer without a cast make[7]: *** [lib_sql_la-sql_scenario.lo] Error 1 Regards, Jennie

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime shttpd.c, 1.19.4.1, 1.19.4.2 shttpd.h, 1.6.6.1, 1.6.6.2 xrpc_server.mx, 1.7.2.7, 1.7.2.8
by Ying Zhang 21 Feb '07

21 Feb '07

Sorry, I forgot to mention that you need to update and rebuild MonetDB4 because there are some changes in conf/MonetDB4.conf.in Regards, Jennie On Wed, Feb 21, 2007 at 04:22:52PM +0000, Ying Zhang wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv31043 > > Modified Files: > Tag: XQuery_0-16 > shttpd.c shttpd.h xrpc_server.mx > Log Message: > implementation of > - xrpc_open (only listen to localhost or not) > - xrpc_port (default port nr) > - xrpc_trusted (prefix of module locations) > - xrpc_admin (hosts that have (web) access to the functions defined in > <datadir>/MonetDB/xrpc/admin/admin.xq) > > > > Index: shttpd.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/shttpd.c,v > retrieving revision 1.19.4.1 > retrieving revision 1.19.4.2 > diff -u -d -r1.19.4.1 -r1.19.4.2 > --- shttpd.c 15 Feb 2007 01:46:13 -0000 1.19.4.1 > +++ shttpd.c 21 Feb 2007 16:22:49 -0000 1.19.4.2 > @@ -332,11 +332,11 @@ > * change enum that goes below! > */ > {'d', "document_root", 0, NULL, OPT_STR, {0}, NULL }, > - {'i', "index_files", 0, NULL, OPT_STR, {0}, INDEX_FILES }, > - {'D', "list_directories",0,NULL, OPT_BOOL, {0}, "1" }, > - {'c', "cgi_extention", 0, NULL, OPT_STR, {0}, CGI_EXTENTION }, > - {'N', "server_name", 0, NULL, OPT_STR, {0}, AUTH_REALM }, > - {'p', "listen_port", 0, NULL, OPT_INT, {0}, LISTEN_PORT }, > + {'i', "index_files", 1, NULL, OPT_STR, {INDEX_FILES}, INDEX_FILES }, > + {'D', "list_directories",1,NULL, OPT_BOOL, {0}, "0" }, > + {'c', "cgi_extention", 1, NULL, OPT_STR, {CGI_EXTENTION}, CGI_EXTENTION }, > + {'N', "server_name", 1, NULL, OPT_STR, {AUTH_REALM}, AUTH_REALM }, > + {'p', "listen_port", 1, NULL, OPT_INT, {LISTEN_PORT}, LISTEN_PORT }, > {'l', "access_log", 0, NULL, OPT_STR, {0}, NULL }, > {'e', "error_log", 0, NULL, OPT_STR, {0}, NULL }, > {'m', "mime_types", 0, NULL, OPT_STR, {0}, NULL }, > @@ -346,7 +346,7 @@ > {'I', "inetd_mode", 0, NULL, OPT_BOOL, {0}, "0" }, > {'u', "runtime_uid", 0, NULL, OPT_STR, {0}, NULL }, > {'V', "show_stats", 0, NULL, OPT_BOOL, {0}, "0" }, > - {'C', "config_file", 0, NULL, OPT_STR, {0}, CONFIG }, > + {'C', "config_file", 1, NULL, OPT_STR, {CONFIG}, CONFIG }, > {0, NULL, 0, NULL, OPT_BOOL, {0}, NULL } > }; > > @@ -1080,7 +1080,7 @@ > * Setup listening socket on given port, return socket > */ > shttpd_socket > -shttpd_open_port(int port) > +shttpd_open_port(int port, int accept_any) > { > shttpd_socket ret; > SOCKET sock; > @@ -1092,10 +1092,10 @@ > if (port == 0) > port = INTOPT(OPT_LISTENPORT); > > - sa.len = sizeof(sa.u.sin); > - sa.u.sin.sin_family = AF_INET; > - sa.u.sin.sin_port = htons((uint16_t) port); > - sa.u.sin.sin_addr.s_addr = htonl(INADDR_ANY); > + sa.len = sizeof(sa.u.sin); > + sa.u.sin.sin_family = AF_INET; > + sa.u.sin.sin_port = htons((uint16_t) port); > + sa.u.sin.sin_addr.s_addr = accept_any?htonl(INADDR_ANY):htonl(INADDR_LOOPBACK); > > if ((sock = socket(PF_INET, SOCK_STREAM, 6)) == -1) > elog(ERR_FATAL, "shttpd_open_port: socket: %s",strerror(ERRNO)); > @@ -3242,6 +3242,12 @@ > return arg->connection->uri; > } > > +struct in_addr > +shttpd_get_inAddr(struct shttpd_callback_arg *arg) > +{ > + return arg->connection->sa.u.sin.sin_addr; > +} > + > void > shttpd_finish(struct shttpd_callback_arg *arg) > { > > Index: shttpd.h > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/shttpd.h,v > retrieving revision 1.6.6.1 > retrieving revision 1.6.6.2 > diff -u -d -r1.6.6.1 -r1.6.6.2 > --- shttpd.h 15 Feb 2007 01:46:13 -0000 1.6.6.1 > +++ shttpd.h 21 Feb 2007 16:22:50 -0000 1.6.6.2 > @@ -159,7 +159,7 @@ > extern void shttpd_setopt(const char *variable, const char *value); > extern void shttpd_addmimetype(const char *ext, const char *mime); > extern void shttpd_register_mountpoint(const char *uri, const char *system_path); > -extern shttpd_socket shttpd_open_port(int port); > +extern shttpd_socket shttpd_open_port(int port, int accept_any); > extern void shttpd_register_url(const char *url, > shttpd_callback_t callback, void *callback_data); > extern void shttpd_protect_url(const char *url, const char *file); > @@ -172,6 +172,7 @@ > extern char *shttpd_get_method(struct shttpd_callback_arg *arg); > extern char *shttpd_get_uri(struct shttpd_callback_arg *arg); > extern int shttpd_get_socket(struct shttpd_callback_arg *arg); > +extern struct in_addr shttpd_get_inAddr(struct shttpd_callback_arg *arg); > extern void shttpd_finish(struct shttpd_callback_arg *arg); > > #ifdef MT > > Index: xrpc_server.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/xrpc_server.mx,v > retrieving revision 1.7.2.7 > retrieving revision 1.7.2.8 > diff -u -d -r1.7.2.7 -r1.7.2.8 > --- xrpc_server.mx 17 Feb 2007 01:10:38 -0000 1.7.2.7 > +++ xrpc_server.mx 21 Feb 2007 16:22:50 -0000 1.7.2.8 > @@ -40,61 +40,123 @@ > > @- HTTP server function(s) > @m > -.COMMAND httpd_start(int port, str option) : void = CMDhttpd_start; > +.COMMAND rpcd_start(int port, bit open, str options) : void = CMDrpcd_start; > "Start the HTTP server for RPC calls on the specified port." > > .EPILOGUE = xrpc_epilogue; > .END xrpc_server; > > @mil > [...1118 lines suppressed...] > > /* Open listening socket */ > - ctx = shttpd_open_port(*port); > + ctx = shttpd_open_port(xrpc_port, *open); > + listen_socket = ctx.sock; > > /* Serve connections infinitely until someone kills us */ > for ( ; rpcd_running; ) shttpd_poll(&ctx, 200); > @@ -905,8 +1199,10 @@ > return GDK_SUCCEED; > } > > -void xrpc_epilogue() { > +void xrpc_epilogue() > +{ > shttpd_fini(); /* Shut down the HTTP server. */ > rpcd_running = 0; /* Stop RPC server */ > + close(listen_socket); > } > /* vim:set shiftwidth=4 expandtab: */ > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/modules/pftijah porter_dutch.c, NONE, 1.1.2.1 porter_dutch.h, NONE, 1.1.2.1 porter_eng.c, NONE, 1.1.2.1 porter_eng.h, NONE, 1.1.2.1 Makefile.ag, 1.16.2.2, 1.16.2.3 pftijah_stem.mx, 1.11.2.1, 1.11.2.2
by Sjoerd Mullender 21 Feb '07

21 Feb '07

Is this a bug fix or a new feature? On 2007-02-21 10:47, Jan Flokstra wrote: > Update of /cvsroot/monetdb/pathfinder/modules/pftijah > In directory sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv590 > > Modified Files: > Tag: XQuery_0-16 > Makefile.ag pftijah_stem.mx > Added Files: > Tag: XQuery_0-16 > porter_dutch.c porter_dutch.h porter_eng.c porter_eng.h > Log Message: > - add 2 new stemmers: > > porter-english > porter-dutch > > The 2 new stemmers are handcrafted and not generated like the snowball > stemmers. First tests indicate they are much faster and indexing time > is speedup 25% > > > > --- NEW FILE: porter_dutch.c --- > /***************************************************************************** > UPLIFT > > This file contains a version of Porter's algorithm for DUTCH based on the > implementation by Frakes (see below). > > The code is a result of the UPLIFT project (Utrecht Project: Linguistic > Information for Free Text Retrieval) and is based on the joint efforts of > Renee Pohlmann and Wessel Kraaij. > > Research Institute for Language and Speech, Utrecht University 1994. > > Copyright (C) 1995 Wessel Kraaij & Renee Pohlmann > > email: Wessel.Kraaij(a)let.ruu.nl Renee.C.Pohlmann(a)let.ru.nl > > This library is free software; you can redistribute it and/or > modify it under the terms of the GNU Library General Public > License as published by the Free Software Foundation; either > [...984 lines suppressed...] > rule = ReplaceSuffix( word, step1c_rules ); > rule = ReplaceInfix( word, step1b_rules ); > if (rule > 0) cleanup = 1; > if ( (502 == rule) ) > rule = ReplaceSuffix( word, step1c_rules ); > rule = ReplaceSuffix( word, step7_rules); > if (rule > 0) cleanup = 1; > if ( (cleanup == 1) ) > (void)ReplaceSuffix( word, step6_rules ); > > /* Part 3: Return an indication of successful stemming */ > > return( TRUE ); > > } /* Stem */ > > > > > > > --- NEW FILE: porter_dutch.h --- > > /******************************* stem.h *********************************** > > UPLIFT > > Purpose: > Header file for a version of Porter's algorithm for DUTCH based on the > implementation by Frakes as documented in: > Porter, M.F., "An Algorithm For Suffix Stripping," > Program 14 (3), July 1980, pp. 130-137 > > The code is a result of the UPLIFT project (Utrecht Project: Linguistic > Information for Free Text Retrieval) and is based on the joint efforts of > Renee Pohlmann and Wessel Kraaij. > > Research Institute for Language and Speech, Utrecht University 1994. > > Copyright (C) 1995 Wessel Kraaij & Renee Pohlmann > > email: Wessel.Kraaij(a)let.ruu.nl Renee.C.Pohlmann(a)let.ru.nl > > This library is free software; you can redistribute it and/or > modify it under the terms of the GNU Library General Public > License as published by the Free Software Foundation; either > version 2 of the License, or (at your option) any later version. > > This library is distributed in the hope that it will be useful, > but WITHOUT ANY WARRANTY; without even the implied warranty of > MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU > Library General Public License for more details. > > You should have received a copy of the GNU Library General Public > License along with this library; if not, write to the Free > Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA. > > > **/ > > #ifndef DSTEM_H > #define DSTEM_H > > /******************************************************************************/ > /**************************** Public Routines *****************************/ > > #ifdef __STDC__ > > extern int dStem( char *word ); > /* returns: 1 --> non-word containing numbers and hyphens, not stemmed > e.g. 12-5-1995 > 2 --> word was not changed, e.g. 'loop' --> 'loop' > 3 --> word was changed */ > > #else > > extern int Stem(); > > #endif > > #endif > > > --- NEW FILE: porter_eng.c --- > /******************************* stem.c *********************************** > > Purpose: Implementation of the Porter stemming algorithm documented > in: Porter, M.F., "An Algorithm For Suffix Stripping," > Program 14 (3), July 1980, pp. 130-137. > > Provenance: Written by B. Frakes and C. Cox, 1986. > Changed by C. Fox, 1990. > - made measure function a DFA > - restructured structs > - renamed functions and variables > - restricted function and variable scopes > Changed by C. Fox, July, 1991. > - added ANSI C declarations > - branch tested to 90% coverage > Changed by Arjen P. de Vries, March, 1996 > - fixed mismatch between prototype and function > - removed unnecessary includes > > Notes: This code will make little sense without the the Porter > article. The stemming function converts its input to > lower case. > **/ > > /************************ Standard Include Files *************************/ > > #include <pf_config.h> > > #include <string.h> > #include <ctype.h> > > /*****************************************************************************/ > /***************** Private Defines and Data Structures *******************/ > > #include "porter_eng.h" > > #define FALSE 0 > #define TRUE 1 > #define EOS '\0' > > #define IsVowel(c) ('a'==(c)||'e'==(c)||'i'==(c)||'o'==(c)||'u'==(c)) > > typedef struct { > int id; /* returned if rule fired */ > char *old_end; /* suffix replaced */ > char *new_end; /* suffix replacement */ > int old_offset; /* from end of word to start of suffix */ > int new_offset; /* from beginning to end of new suffix */ > int min_root_size; /* min root word size for replacement */ > int (*condition)(); /* the replacement test function */ > } RuleList; > > static char LAMBDA[1] = ""; /* the constant empty string */ > static char *end; /* pointer to the end of the word */ > > /*****************************************************************************/ > /******************** Private Function Declarations **********************/ > > #ifdef __STDC__ > > static int WordSize( char *word ); > static int ContainsVowel( char *word ); > static int EndsWithCVC( char *word ); > static int AddAnE( char *word ); > static int RemoveAnE( char *word ); > static int ReplaceEnd( char *word, RuleList *rule ); > > #else > > static int WordSize( /* word */ ); > static int ContainsVowel( /* word */ ); > static int EndsWithCVC( /* word */ ); > static int AddAnE( /* word */ ); > static int RemoveAnE( /* word */ ); > static int ReplaceEnd( /* word, rule */ ); > > #endif > > /******************************************************************************/ > /***************** Initialized Private Data Structures ********************/ > > static RuleList step1a_rules[] = > { > {101, "sses", "ss", 3, 1, -1, NULL}, > {102, "ies", "i", 2, 0, -1, NULL}, > {103, "ss", "ss", 1, 1, -1, NULL}, > {104, "s", LAMBDA, 0, -1, -1, NULL}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step1b_rules[] = > { > {105, "eed", "ee", 2, 1, 0, NULL}, > {106, "ed", LAMBDA, 1, -1, -1, ContainsVowel}, > {107, "ing", LAMBDA, 2, -1, -1, ContainsVowel}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step1b1_rules[] = > { > {108, "at", "ate", 1, 2, -1, NULL}, > {109, "bl", "ble", 1, 2, -1, NULL}, > {110, "iz", "ize", 1, 2, -1, NULL}, > {111, "bb", "b", 1, 0, -1, NULL}, > {112, "dd", "d", 1, 0, -1, NULL}, > {113, "ff", "f", 1, 0, -1, NULL}, > {114, "gg", "g", 1, 0, -1, NULL}, > {115, "mm", "m", 1, 0, -1, NULL}, > {116, "nn", "n", 1, 0, -1, NULL}, > {117, "pp", "p", 1, 0, -1, NULL}, > {118, "rr", "r", 1, 0, -1, NULL}, > {119, "tt", "t", 1, 0, -1, NULL}, > {120, "ww", "w", 1, 0, -1, NULL}, > {121, "xx", "x", 1, 0, -1, NULL}, > {122, LAMBDA, "e", -1, 0, -1, AddAnE}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step1c_rules[] = > { > {123, "y", "i", 0, 0, -1, ContainsVowel}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step2_rules[] = > { > {203, "ational", "ate", 6, 2, 0, NULL}, > {204, "tional", "tion", 5, 3, 0, NULL}, > {205, "enci", "ence", 3, 3, 0, NULL}, > {206, "anci", "ance", 3, 3, 0, NULL}, > {207, "izer", "ize", 3, 2, 0, NULL}, > {208, "abli", "able", 3, 3, 0, NULL}, > {209, "alli", "al", 3, 1, 0, NULL}, > {210, "entli", "ent", 4, 2, 0, NULL}, > {211, "eli", "e", 2, 0, 0, NULL}, > {213, "ousli", "ous", 4, 2, 0, NULL}, > {214, "ization", "ize", 6, 2, 0, NULL}, > {215, "ation", "ate", 4, 2, 0, NULL}, > {216, "ator", "ate", 3, 2, 0, NULL}, > {217, "alism", "al", 4, 1, 0, NULL}, > {218, "iveness", "ive", 6, 2, 0, NULL}, > {219, "fulnes", "ful", 5, 2, 0, NULL}, > {220, "ousness", "ous", 6, 2, 0, NULL}, > {221, "aliti", "al", 4, 1, 0, NULL}, > {222, "iviti", "ive", 4, 2, 0, NULL}, > {223, "biliti", "ble", 5, 2, 0, NULL}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step3_rules[] = > { > {301, "icate", "ic", 4, 1, 0, NULL}, > {302, "ative", LAMBDA, 4, -1, 0, NULL}, > {303, "alize", "al", 4, 1, 0, NULL}, > {304, "iciti", "ic", 4, 1, 0, NULL}, > {305, "ical", "ic", 3, 1, 0, NULL}, > {308, "ful", LAMBDA, 2, -1, 0, NULL}, > {309, "ness", LAMBDA, 3, -1, 0, NULL}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step4_rules[] = > { > {401, "al", LAMBDA, 1, -1, 1, NULL}, > {402, "ance", LAMBDA, 3, -1, 1, NULL}, > {403, "ence", LAMBDA, 3, -1, 1, NULL}, > {405, "er", LAMBDA, 1, -1, 1, NULL}, > {406, "ic", LAMBDA, 1, -1, 1, NULL}, > {407, "able", LAMBDA, 3, -1, 1, NULL}, > {408, "ible", LAMBDA, 3, -1, 1, NULL}, > {409, "ant", LAMBDA, 2, -1, 1, NULL}, > {410, "ement", LAMBDA, 4, -1, 1, NULL}, > {411, "ment", LAMBDA, 3, -1, 1, NULL}, > {412, "ent", LAMBDA, 2, -1, 1, NULL}, > {423, "sion", "s", 3, 0, 1, NULL}, > {424, "tion", "t", 3, 0, 1, NULL}, > {415, "ou", LAMBDA, 1, -1, 1, NULL}, > {416, "ism", LAMBDA, 2, -1, 1, NULL}, > {417, "ate", LAMBDA, 2, -1, 1, NULL}, > {418, "iti", LAMBDA, 2, -1, 1, NULL}, > {419, "ous", LAMBDA, 2, -1, 1, NULL}, > {420, "ive", LAMBDA, 2, -1, 1, NULL}, > {421, "ize", LAMBDA, 2, -1, 1, NULL}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step5a_rules[] = > { > {501, "e", LAMBDA, 0, -1, 1, NULL}, > {502, "e", LAMBDA, 0, -1, -1, RemoveAnE}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > static RuleList step5b_rules[] = > { > {503, "ll", "l", 1, 0, 1, NULL}, > {000, NULL, NULL, 0, 0, 0, NULL} > }; > > /*****************************************************************************/ > /******************** Private Function Declarations **********************/ > > /*FN*************************************************************************** > > WordSize( word ) > > Returns: int -- a weird count of word size in adjusted syllables > > Purpose: Count syllables in a special way: count the number > vowel-consonant pairs in a word, disregarding initial > consonants and final vowels. The letter "y" counts as a > consonant at the beginning of a word and when it has a vowel > in front of it; otherwise (when it follows a consonant) it > is treated as a vowel. For example, the WordSize of "cat" > is 1, of "any" is 1, of "amount" is 2, of "anything" is 3. > > Plan: Run a DFA to compute the word size > > Notes: The easiest and fastest way to compute this funny measure is > with a finite state machine. The initial state 0 checks > the first letter. If it is a vowel, then the machine changes > to state 1, which is the "last letter was a vowel" state. > If the first letter is a consonant or y, then it changes > to state 2, the "last letter was a consonant state". In > state 1, a y is treated as a consonant (since it follows > a vowel), but in state 2, y is treated as a vowel (since > it follows a consonant. The result counter is incremented > on the transition from state 1 to state 2, since this > transition only occurs after a vowel-consonant pair, which > is what we are counting. > **/ > > static int > WordSize( word ) > char *word; /* in: word having its WordSize taken */ > { > register int result; /* WordSize of the word */ > register int state; /* current state in machine */ > > result = 0; > state = 0; > > /* Run a DFA to compute the word size */ > while ( EOS != *word ) > { > switch ( state ) > { > case 0: state = (IsVowel(*word)) ? 1 : 2; > break; > case 1: state = (IsVowel(*word)) ? 1 : 2; > if ( 2 == state ) result++; > break; > case 2: state = (IsVowel(*word) || ('y' == *word)) ? 1 : 2; > break; > } > word++; > } > > return( result ); > > } /* WordSize */ > > /*FN************************************************************************** > > ContainsVowel( word ) > > Returns: int -- TRUE (1) if the word parameter contains a vowel, > FALSE (0) otherwise. > > Purpose: Some of the rewrite rules apply only to a root containing > a vowel, where a vowel is one of "aeiou" or y with a > consonant in front of it. > > Plan: Obviously, under the definition of a vowel, a word contains > a vowel iff either its first letter is one of "aeiou", or > any of its other letters are "aeiouy". The plan is to > test this condition. > > Notes: None > **/ > > static int > ContainsVowel( word ) > char *word; /* in: buffer with word checked */ > { > > if ( EOS == *word ) > return( FALSE ); > else > return( IsVowel(*word) || (NULL != strpbrk(word+1,"aeiouy")) ); > > > } /* ContainsVowel */ > > /*FN************************************************************************** > > EndsWithCVC( word ) > > Returns: int -- TRUE (1) if the current word ends with a > consonant-vowel-consonant combination, and the second > consonant is not w, x, or y, FALSE (0) otherwise. > > Purpose: Some of the rewrite rules apply only to a root with > this characteristic. > > Plan: Look at the last three characters. > > Notes: None > **/ > > static int > EndsWithCVC( word ) > char *word; /* in: buffer with the word checked */ > { > int length; /* for finding the last three characters */ > > if ( (length = strlen(word)) < 2 ) > return( FALSE ); > else > { > end = word + length - 1; > return( (NULL == strchr("aeiouwxy",*end--)) /* consonant */ > && (NULL != strchr("aeiouy", *end--)) /* vowel */ > && (NULL == strchr("aeiou", *end )) ); /* consonant */ > } > > } /* EndsWithCVC */ > > /*FN************************************************************************** > > AddAnE( word ) > > Returns: int -- TRUE (1) if the current word meets special conditions > for adding an e. > > Purpose: Rule 122 applies only to a root with this characteristic. > > Plan: Check for size of 1 and a consonant-vowel-consonant ending. > > Notes: None > **/ > > static int > AddAnE( word ) > char *word; > { > > return( (1 == WordSize(word)) && EndsWithCVC(word) ); > > } /* AddAnE */ > > /*FN************************************************************************** > > RemoveAnE( word ) > > Returns: int -- TRUE (1) if the current word meets special conditions > for removing an e. > > Purpose: Rule 502 applies only to a root with this characteristic. > > Plan: Check for size of 1 and no consonant-vowel-consonant ending. > > Notes: None > **/ > > static int > RemoveAnE( word ) > char *word; > { > > return( (1 == WordSize(word)) && !EndsWithCVC(word) ); > > } /* RemoveAnE */ > > /*FN************************************************************************** > > ReplaceEnd( word, rule ) > > Returns: int -- the id for the rule fired, 0 is none is fired > > Purpose: Apply a set of rules to replace the suffix of a word > > Plan: Loop through the rule set until a match meeting all conditions > is found. If a rule fires, return its id, otherwise return 0. > Connditions on the length of the root are checked as part of this > function's processing because this check is so often made. > > Notes: This is the main routine driving the stemmer. It goes through > a set of suffix replacement rules looking for a match on the > current suffix. When it finds one, if the root of the word > is long enough, and it meets whatever other conditions are > required, then the suffix is replaced, and the function returns. > **/ > > static int > ReplaceEnd( word, rule ) > char *word; /* in/out: buffer with the stemmed word */ > RuleList *rule; /* in: data structure with replacement rules */ > { > register char *ending; /* set to start of possible stemmed suffix */ > char tmp_ch; /* save replaced character when testing */ > > while ( 0 != rule->id ) > { > ending = end - rule->old_offset; > if ( word <= ending ) > if ( 0 == strcmp(ending,rule->old_end) ) > { > tmp_ch = *ending; > *ending = EOS; > if ( rule->min_root_size < WordSize(word) ) > if ( !rule->condition || (*rule->condition)(word) ) > { > (void)strcat( word, rule->new_end ); > end = ending + rule->new_offset; > break; > } > *ending = tmp_ch; > } > rule++; > } > > return( rule->id ); > > } /* ReplaceEnd */ > > /*****************************************************************************/ > /********************* Public Function Declarations **********************/ > > /*FN*************************************************************************** > > Stem( word ) > > Returns: int -- FALSE (0) if the word contains non-alphabetic characters > and hence is not stemmed, TRUE (1) otherwise > > Purpose: Stem a word > > Plan: Part 1: Check to ensure the word is all alphabetic > Part 2: Run through the Porter algorithm > Part 3: Return an indication of successful stemming > > Notes: This function implements the Porter stemming algorithm, with > a few additions here and there. See: > > Porter, M.F., "An Algorithm For Suffix Stripping," > Program 14 (3), July 1980, pp. 130-137. > > Porter's algorithm is an ad hoc set of rewrite rules with > various conditions on rule firing. The terminology of > "step 1a" and so on, is taken directly from Porter's > article, which unfortunately gives almost no justification > for the various steps. Thus this function more or less > faithfully refects the opaque presentation in the article. > Changes from the article amount to a few additions to the > rewrite rules; these are marked in the RuleList data > structures with comments. > **/ > > int > Stem( word ) > char *word; /* in/out: the word stemmed */ > { > int rule; /* which rule is fired in replacing an end */ > > /* Part 1: Check to ensure the word is all alphabetic */ > for ( end = word; *end != EOS; end++ ) > if ( !isalpha(*end) ) return( FALSE ); > else *end = tolower( *end ); > end--; > > /* Part 2: Run through the Porter algorithm */ > (void)ReplaceEnd( word, step1a_rules ); > rule = ReplaceEnd( word, step1b_rules ); > if ( (106 == rule) || (107 == rule) ) > (void)ReplaceEnd( word, step1b1_rules ); > (void)ReplaceEnd( word, step1c_rules ); > > (void)ReplaceEnd( word, step2_rules ); > > (void)ReplaceEnd( word, step3_rules ); > > (void)ReplaceEnd( word, step4_rules ); > > (void)ReplaceEnd( word, step5a_rules ); > (void)ReplaceEnd( word, step5b_rules ); > > /* Part 3: Return an indication of successful stemming */ > return( TRUE ); > > } /* Stem */ > > Index: Makefile.ag > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/Makefile.ag,v > retrieving revision 1.16.2.2 > retrieving revision 1.16.2.3 > diff -u -d -r1.16.2.2 -r1.16.2.3 > --- Makefile.ag 24 Jan 2007 23:12:28 -0000 1.16.2.2 > +++ Makefile.ag 21 Feb 2007 09:47:09 -0000 1.16.2.3 > @@ -52,6 +52,10 @@ > stem_UTF_8_english.c \ > stem_UTF_8_porter.h \ > stem_UTF_8_porter.c \ > + porter_eng.h \ > + porter_eng.c \ > + porter_dutch.h \ > + porter_dutch.c \ > pftijah_tokenize.l \ > pftijah_util.mx \ > serialize_pftijah.mx \ > > Index: pftijah_stem.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/pftijah_stem.mx,v > retrieving revision 1.11.2.1 > retrieving revision 1.11.2.2 > diff -u -d -r1.11.2.1 -r1.11.2.2 > --- pftijah_stem.mx 5 Feb 2007 08:05:08 -0000 1.11.2.1 > +++ pftijah_stem.mx 21 Feb 2007 09:47:09 -0000 1.11.2.2 > @@ -628,6 +628,78 @@ > static tjStemCtx snb_dutch = { "snowball-dutch", snb_dutch_init, snb_dutch_stem, snb_dutch_clear, NULL , NULL_STOPWORDS }; > > /* > + * > + * a fast porter stemmer experiment > + * > + */ > + > +#include "porter_eng.h" > + > +static int porter_english_init(void* ctx, char* s) { > + (void)ctx; > + (void)s; > + return 1; > +} > + > +static const char* porter_english_stem(void* ctx, const char* s) { > + (void)ctx; > +#ifdef STEMMING_DEBUG > + stream_printf(GDKout,"+ stem-porter_english(%s)=",s); > +#endif > + if ( Stem( s ) ) { > +#ifdef STEMMING_DEBUG > + stream_printf(GDKout,"\"%s\"\n",s); > +#endif > + return s; > + } else { > +#ifdef STEMMING_DEBUG > + stream_printf(GDKout,"\<FAIL>\"%s\"\n",s); > +#endif > + return s; > + } > +} > + > +static int porter_english_clear(void* ctx) { > + (void)ctx; > + return 1; > +} > + > +static tjStemCtx porter_english = { "porter-english", porter_english_init, porter_english_stem, porter_english_clear, NULL , ENGLISH_STOPWORDS }; > + > +#include "porter_dutch.h" > + > +static int porter_dutch_init(void* ctx, char* s) { > + (void)ctx; > + (void)s; > + return 1; > +} > + > +static const char* porter_dutch_stem(void* ctx, const char* s) { > + (void)ctx; > +#ifdef STEMMING_DEBUG > + stream_printf(GDKout,"+ stem-porter_dutch(%s)=",s); > +#endif > + dStem( s ); > +#ifdef STEMMING_DEBUG > + stream_printf(GDKout,"\"%s\"\n",s); > +#endif > + return s; > +} > + > +static int porter_dutch_clear(void* ctx) { > + (void)ctx; > + return 1; > +} > + > +static tjStemCtx porter_dutch = { "porter-dutch", porter_dutch_init, porter_dutch_stem, porter_dutch_clear, NULL , NULL_STOPWORDS }; > + > +/* > + * > + * end of fast porter stemmer experiment > + * > + */ > + > +/* > * The getStemmingContext() function is the extern function called to get > * the stemming context. > */ > @@ -648,6 +720,10 @@ > res = & snb_english; > else if ( strcmp(name,"snowball-dutch")==0 ) > res = & snb_dutch; > + else if ( strcmp(name,"porter-english")==0 ) > + res = & porter_english; > + else if ( strcmp(name,"porter-dutch")==0 ) > + res = & porter_dutch; > else { > stream_printf(GDKerr,"# WARNING: stemmer \"%s\" unknown, using nostemming\n",name); > res = & nostemming; > > --- NEW FILE: porter_eng.h --- > > /******************************* stem.h *********************************** > > Purpose: Header file for an implementation of the Porter stemming > algorithm. > > Notes: This module implemnts a fast stemming function whose results > are about as good as any other. > **/ > > #ifndef STEM_H > #define STEM_H > > /******************************************************************************/ > /**************************** Public Routines *****************************/ > > #ifdef __STDC__ > > extern int Stem( char *word ); /* returns 1 on success, 0 otherwise */ > > #else > > extern int Stem(); > > #endif > > #endif > > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- Sjoerd Mullender

2 1

Re: [Monetdb-developers] HELP ! Excel ODBC Demo cannot be found
by Martin Kersten 19 Feb '07

19 Feb '07

Mark Simms wrote: > http://monetdb.cwi.nl/TechDocs/APIs/ODBC/index.html > > The above link is dead. > > > ------------------------------------------------------------------------ > *From:* Mark Simms [mailto:marksimms@verizon.net] > *Sent:* Monday, February 19, 2007 12:36 PM > *To:* 'monet(a)cwi.nl' > *Subject:* HELP ! Excel ODBC Demo cannot be found > > http://monetdb.cwi.nl/MonetDB/Documentation/The-ODBC-Library.html#The-ODBC-… > > Can you please check out this page and fix the link to the Excel > Example ? > The link appears to be wrong. > I cannot find the Excel example anywhere on this site ! Thanks. This one wasn't catch by the link checker. We'll look into it shortly. regards, Martin

1 0