- developers-list - monetdb.org

[Monetdb-developers] SQL:2003 OLAP/windowing functions?
by Andrew Piskorski 25 Jan '06

25 Jan '06

Hello, I believe MonetDB/SQL does not currently support the SQL:2003 OLAP (aka, windowing) functions. Do you have any plans to add such support, or thoughts on how to do so? Coul this support be added via MIL, and if so how difficult do you think that would be? E.g., is it something that a dedicated end-user/devloper might plausibly attempt, or would it require deep knowledge of and hacking on the MonetDB core itself? These windowing functions were added as an amendement to the SQL:1999 spec, and SQL:2003 includes that, I think as features T611, "Elementrary OLAP functions" and T612, "Advanced OLAP functions". Apparently Fred Zemke of Oracle was the author of those SQL specs, and IBM also supported it, so the SQL:2003 syntax and behavior is probably identical to what Oracle and DB2 already have. PostgreSQL 8.0 doesn't yet support it, and I don't know if/when they plan to. Oracle calls this their "analytic" SQL functions, and has had it starting in 8i with improvements in 9i. I've used it extensively on Oracle, and although its syntax seems kind of klunky and non-intuitive, the functionality is EXTREMELY useful. Here are a bunch of links about the SQL OLAP/windowing functions: http://www.wintercorp.com/rwintercolumns/SQL_99snewolapfunctions.html http://www.ncb.ernet.in/education/modules/dbms/SQL99/OLAP-99-154r2.pdf http://www.wiscorp.com/sql/SQL2003Features.pdf http://troels.arvin.dk/db/rdbms/#select-limit-offset http://www.postgresql.org/docs/8.0/interactive/features.html http://en.wikipedia.org/wiki/SQL http://www.sigmod.org/sigmod/record/issues/0403/E.JimAndrew-standard.pdf http://www.oracle.com/oramag/oracle/01-jul/o41industry.html And here are a few more examples of using it. "lead" and "lag" are just the beginning, there is more. I find "dense_rank" particularly useful: http://www.mail-archive.com/sqlite-users@sqlite.org/msg08223.html http://www.mail-archive.com/sqlite-users@sqlite.org/msg10855.html -- Andrew Piskorski <atp(a)piskorski.com> http://www.piskorski.com/

2 1

[Monetdb-developers] FYI: only Fedore Core RPMs for upcoming MonetDB 4.10 release
by Stefan Manegold 22 Jan '06

22 Jan '06

Dear all, I just wanted to let you now that the only binary RPMs we will provide with the upcoming MonetDB 4.10 release will be made on Fedora Core 4 systems. On all other machines that we currently use with RPM-bases systems (SuSE, SLES & RHEL), too many tools are missing to build a full-fledged source distribution and/or all RPMs. We might consider having these machines upgraded *after* the initial release. Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

[Monetdb-developers] Feature Freeze!
by Stefan Manegold 21 Jan '06

21 Jan '06

Dear all, to ensure smooth processing in the final stages before we will release MonetDB 4.10 (hopefully on Monday), I now declare a Feature Freeze i.e., as of now, no more checkin sare allowed to *development trunks* of any the MonetDB related repositories on SF (buildtools, MonetDB, sql, pathfinder), unless they are bug fixes! I plan to fork-off the new release branches later tonight or tomorrow. I let you know, once I'm done with that. *Only then*, you may again check-in new features to the *development trunks*; *all* bugfixes must then go to the new release branches. Thanks you very much for your cooperation. Regards, Stefan ps: Your well come to take your share of the following tasks: 1) Take care of all open bugs and fix them: https://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=4… 2) Re-check all bug reports that are set to "pending" and add proper test scripts for them: https://sourceforge.net/tracker/index.php?func=browse&group_id=56967&atid=4… If you do so, please let me know! Thanks in advance! -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1

Fwd: * Re: * [Monetdb-developers] MonetDB on FreeBSD?
by Arnoud Vermeer 20 Jan '06

20 Jan '06

Hi, I've included the full alarm.glue.c in the following url: http://www.freshway.biz/files/MonetDB-alarm.glue.c.txt I hope it helps. Thanks, Arnoud

3 9

Re: * [Monetdb-developers] MonetDB on FreeBSD?
by Stefan Manegold 20 Jan '06

20 Jan '06

> > > configure: WARNING: sys/file.h: present but cannot be compiled > > > configure: WARNING: sys/file.h: check for missing prerequisite > > headers? > > > configure: WARNING: sys/file.h: see the Autoconf documentation > > > configure: WARNING: sys/file.h: section "Present But Cannot Be > > Compiled" > > > > > > configure: WARNING: sys/file.h: proceeding with the preprocessor's > > result > > > configure: WARNING: sys/file.h: in the future, the compiler will take > > > precedence > > > configure: WARNING: ## ------------------------------------------ ## > > > > > configure: WARNING: ## Report this to the AC_PACKAGE_NAME lists. ## > > > configure: WARNING: ## ------------------------------------------ ## > > > > are these the only error messages you get from configure? > > yes, I've seen them before at other compiles to. Does compile most of the > time, inspite of the error. yes. I actually don't think that these are the reason for the errors you get. first was, that MonetDB requires GNU make instead of BSD make. That's solved. > The gmake fails: > standard ./configure (same output as in earlier posted version) > gmake > gmake.txt > gmake console output > gmake_console.txt > > Adding --enable-bits=64 to the ./configure doesn't change the output. > > http://www.freshway.biz/files/MonetDB-gmake.txt > http://www.freshway.biz/files/MonetDB-gmake_console.txt > > Any ideas? not really. I'm a bit scared by all the "overriding commands for target" messages, but the real problems seems to be this: ======= In file included from stream.mx:264: /usr/include/netinet/ip.h:49: error: syntax error before "u_int" /usr/include/netinet/ip.h:148: error: syntax error before "u_char" /usr/include/netinet/ip.h:200: error: syntax error before "u_char" stream.mx: In function ocket_open': stream.mx:1198: error: P_TOS' undeclared (first use in this function) stream.mx:1198: error: (Each undeclared identifier is reported only once stream.mx:1198: error: for each function it appears in.) gmake[4]: *** [libstream_la-stream.lo] Error 1 gmake[3]: *** [all] Error 2 gmake[2]: *** [all-recursive] Error 1 gmake[1]: *** [all-recursive] Error 1 gmake: *** [all] Error 2 ======== I recall that there were some changes/fixes related to FreeBSD, which might not be in the release tarball ("MonetDB-4.8.2.tar.gz"). But we are about to release MonetDB 4.10 on Monday; hence, could you download the latest release candidate "MonetDB-4.9.3.tar.gz" from http://monetdb.cwi.nl/testing/projects/monetdb/Current/.DailyBuilds./Source… and check whether this works (better)? Stefan > Greetings, > > Arnoud Vermeer > FreshWay Innovations -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: * [Monetdb-developers] MonetDB on FreeBSD?
by Stefan Manegold 19 Jan '06

19 Jan '06

Arnoud, which version of gcc are you using? `gcc --version` Stefan On Thu, Jan 19, 2006 at 10:57:58PM +0100, Arnoud Vermeer wrote: > Hi Stefan, > > In the new version, I've got a new error. The gmake command lasts longer > though, so I guess the previous problem was fixed in the newer version. > > This is the current error I get at a standard ./configure && gmake > > /bin/sh ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I. > -I../../.. -I. -I../../common -I../../gdk -I../../monet -DLIBALARM > -g -O2 -std=c99 -Wall -W -fno-strict-aliasing > -Werror-implicit-function-declaration -Werror -Wno-unused-function > -Wno-unused-label -D_REENTRANT -c -o lib_alarm_la-alarm.glue.lo `test -f ' > alarm.glue.c' || echo './'`alarm.glue.c > gcc -DHAVE_CONFIG_H -I. -I. -I../../.. -I. -I../../common -I../../gdk > -I../../monet -DLIBALARM -g -O2 -std=c99 -Wall -W -fno-strict-aliasing > -Werror-implicit-function-declaration -Werror -Wno-unused-function > -Wno-unused-label -D_REENTRANT -c alarm.glue.c -fPIC -DPIC -o > .libs/lib_alarm_la-alarm.glue.o > alarm.glue.c: In function `CMDctime_unpack559103634': > alarm.glue.c:185: warning: assignment discards qualifiers from pointer > target type > gmake[4]: *** [lib_alarm_la-alarm.glue.lo] Error 1 > gmake[4]: Leaving directory `/usr/home/funzoneq/MonetDB-4.9.3 > /src/modules/plain' > gmake[3]: *** [check] Error 2 > gmake[3]: Leaving directory `/usr/home/funzoneq/MonetDB-4.9.3 > /src/modules/plain' > gmake[2]: *** [check-recursive] Error 1 > gmake[2]: Leaving directory `/usr/home/funzoneq/MonetDB-4.9.3/src/modules' > gmake[1]: *** [check-recursive] Error 1 > gmake[1]: Leaving directory `/usr/home/funzoneq/MonetDB-4.9.3/src' > gmake: *** [check-recursive] Error 1 > > I also get a lot of overrides from rpm.mk, but that doesn't concern me that > much hence FreeBSD doesn't know rpm's. It seems the new error is located in > alarm.glue.c. > Although I have had a small bit of experience with C++ programming (first > year student Information Sciences at Windesheim College, Zwolle, The > Netherlands), I don't have a clue what the error means. > > Do you have any clue? > > Thanks, > > Arnoud Vermeer > > On 1/19/06, Stefan Manegold <Stefan.Manegold(a)cwi.nl> wrote: > > > > > > > configure: WARNING: sys/file.h: present but cannot be compiled > > > > > configure: WARNING: sys/file.h: check for missing prerequisite > > > > headers? > > > > > configure: WARNING: sys/file.h: see the Autoconf documentation > > > > > configure: WARNING: sys/file.h: section "Present But Cannot Be > > > > Compiled" > > > > > > > > > > configure: WARNING: sys/file.h: proceeding with the preprocessor's > > > > result > > > > > configure: WARNING: sys/file.h: in the future, the compiler will > > take > > > > > precedence > > > > > configure: WARNING: ## > > ------------------------------------------ ## > > > > > > > > > configure: WARNING: ## Report this to the AC_PACKAGE_NAME > > lists. ## > > > > > configure: WARNING: ## > > ------------------------------------------ ## > > > > > > > > are these the only error messages you get from configure? > > > > > > yes, I've seen them before at other compiles to. Does compile most of > > the > > > time, inspite of the error. > > > > yes. I actually don't think that these are the reason for the errors you > > get. > > > > first was, that MonetDB requires GNU make instead of BSD make. That's > > solved. > > > > > The gmake fails: > > > standard ./configure (same output as in earlier posted version) > > > gmake > gmake.txt > > > gmake console output > gmake_console.txt > > > > > > Adding --enable-bits=64 to the ./configure doesn't change the output. > > > > > > http://www.freshway.biz/files/MonetDB-gmake.txt > > > http://www.freshway.biz/files/MonetDB-gmake_console.txt > > > > > > Any ideas? > > > > not really. > > > > I'm a bit scared by all the "overriding commands for target" messages, but > > the real problems seems to be this: > > ======= > > In file included from stream.mx:264: > > /usr/include/netinet/ip.h:49: error: syntax error before "u_int" > > /usr/include/netinet/ip.h:148: error: syntax error before "u_char" > > /usr/include/netinet/ip.h:200: error: syntax error before "u_char" > > stream.mx: In function ocket_open': > > stream.mx:1198: error: P_TOS' undeclared (first use in this function) > > stream.mx:1198: error: (Each undeclared identifier is reported only once > > stream.mx:1198: error: for each function it appears in.) > > gmake[4]: *** [libstream_la-stream.lo] Error 1 > > gmake[3]: *** [all] Error 2 > > gmake[2]: *** [all-recursive] Error 1 > > gmake[1]: *** [all-recursive] Error 1 > > gmake: *** [all] Error 2 > > ======== > > > > I recall that there were some changes/fixes related to FreeBSD, which > > might > > not be in the release tarball ("MonetDB-4.8.2.tar.gz"). > > > > But we are about to release MonetDB 4.10 on Monday; > > hence, could you download the latest release candidate " > > MonetDB-4.9.3.tar.gz" > > from > > http://monetdb.cwi.nl/testing/projects/monetdb/Current/.DailyBuilds./Source… > > and check whether this works (better)? > > > > Stefan > > > > > Greetings, > > > > > > Arnoud Vermeer > > > FreshWay Innovations > > > > -- > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 1

[Monetdb-developers] MonetDB on FreeBSD?
by Arnoud Vermeer 19 Jan '06

19 Jan '06

Hello, I'm trying to compile MonetDB on FreeBSD 5.4. The machine is a dell single xeon EMT64 3.0 ghz with 2 gb of memory. The ./configure works just fine. I've included the output in configure.txt It does however put out the following text: configure: WARNING: sys/file.h: present but cannot be compiled configure: WARNING: sys/file.h: check for missing prerequisite headers? configure: WARNING: sys/file.h: see the Autoconf documentation configure: WARNING: sys/file.h: section "Present But Cannot Be Compiled" configure: WARNING: sys/file.h: proceeding with the preprocessor's result configure: WARNING: sys/file.h: in the future, the compiler will take precedence configure: WARNING: ## ------------------------------------------ ## configure: WARNING: ## Report this to the AC_PACKAGE_NAME lists. ## configure: WARNING: ## ------------------------------------------ ## When I try to run make, it returns the following error: "Makefile", line 981: Could not find ./*.mk "Makefile", line 982: Could not find ./*.mk make: fatal errors encountered -- cannot continue It is these two lines in the Makefile: include $(top_srcdir)/*.mk include $(top_builddir)/*.mk when I ls src/* I get no *.mk just: Makefile Makefile.am <http://makefile.am/> Makefile.msc gdk mel monet tools Makefile.ag <http://makefile.ag/> Makefile.in <http://makefile.in/> common mapi modules testing utils Includes: http://www.freshway.biz/files/MonetDB-configure.txt http://www.freshway.biz/files/MonetDB-makefile.txt Can anybody help me further along? Kind regards, Arnoud Vermeer FreshWay Innovations The Netherlands

3 4

Re: [Monetdb-developers] RFC: Upcoming release of MonetDB 4.10: Burkowski Code
by Stefan Manegold 19 Jan '06

19 Jan '06

On Tue, Jan 17, 2006 at 10:47:33PM +0100, Martin Kersten wrote: > > > Wouter Alink wrote: > >My opinion, for what it is worth... > > >I would like to ask to leave the code in, because it (probably) would not > >harm anybody, and I would be happy. > But are you planning to stick to the stable release then? or > do you switch to the development head directly? > Does your users work with that stuff from the release? > > I think that it can be removed from the release and stay > in the development branch. This would make you happy and > does not distract outsiders to use it too early. Exactly that was my proposal: leave the Burkowski code in the development runk, but keep it out of the release. Since the release will not be configure with `--enable-burk` (most; see * below) of the respective code will not be compiled and the features will not be usable with the (binary) release, anyway. Moreover, if I understand the comments of Jan & Jens (separate mail discussion between Wouter, Jens, Jan & me) correctly, the current burK implementation/integration does cause some significant(?) performance penalty in the burG generated code, which will harm the performance of XQuery translation significantly(?). Hence, unless Wouter has some clients that need to use the release with Burkowski features (i.e., self-compiled from the sources), I make a strong point to keep the Burkowski code out of the release --- the next release (then with Burkowski code) will (must ;-)) be ready within 2 months from now. (*) "Even" for the development trunk, I'd suggest to try an use the macro BURKOWSKI (and an AC_CONDITIONAL that we should add), that is/are set with the --enable-burk configure switch, to remove ALL Burkowski related code from the compilation; i.e., (if possible) also the declarative parts in compiler/algebra/core2alg.brg, compiler/core/coreopt.brg, compiler/core/simplify.brg, compiler/semantics/typecheck.brg, as well as the changes in compiler/parser/parser.y, compiler/parser/scanner.l, & runtime/pathfinder.mx. In fact, Jan & Jens suggest to even reduce the changes in (at least) compiler/algebra/core2alg.brg to a minimum. Maybe, Jan & Jens can help with the *.brg file (also optimizing the indices), I could help with runtime/pathfinder.mx, and the AC_CONDITIONAL to avoid compilation of pf_burk unless configure with --enable-burk. If we agree on keeping Burk out of the release, we have almost tow month to implement these improvements in the development trunk ;-) Kind Regards, Stefan ps: the release must be ready by Sunday Jan 22, 23:59:59, hence a quick reply is appreciated; sorry for the hurry, but "external constraints" forced us on short notice to finish the release a few days earlier than planned ... -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 2

[Monetdb-developers] RFC: Upcoming release of MonetDB 4.10
by Stefan Manegold 17 Jan '06

17 Jan '06

Dear Developers, Enclosed you find the draft cover letter for the upcoming MonetDB release. Please provide any comments that from your perspective are relevant to mention. In preparing the distribution content we have finally decided not to include anything of the Xupdate functionality, SOAP, or other XML modules. The primary reason is that implementation and validation of these components have not reached a point were we are confident of their stability. These issues will be moved forward to the next release planned for March 2006 when the system will be presented at EDBT. Nevertheless, we can be proud of the progress made. Especially interaction with the server using a large number of small SQL update queries has been boosted to the point where we are in the same arena as MySQL and PostgreSQL using JDBC connectivity. The code freeze date is Friday, Jan 13 2006. We plan to release MonetDB v4.10 on Tuesday, Jan 24 2006. Regards, The MonetDB Development Team ------------------- Cover release 4.10 ---------- The MonetDB release 4.10 marks the end of a period where much attention was given to improving the performance of simple SQL applications. These improvements came from careful analysis of student programs and participating in the "c't" Magazine's database contest featuring the DVDstore benchmark. It led to re-consideration of MAPI -- the communication protocol -- aiming for less overhead and complex structures to be parsed. To facilitate this, JDBC has been largely be rewritten at core level to streamline the communication flow even more. Not only JDBC has benefited from the changes made, all MAPI driven communication has been sped up. A second Perl library has been prepared, which more directly uses the MAPI library. It has been registered at the official Perl archive for distribution (ICAN). ODBC has been upgraded to support server side prepared statements like JDBC. More meta-data is now available for columns in JDBC. MonetDB/SQL has been extended with SQL:03's sequences support. This allows for identity columns, serials (PostgreSQL) or auto_increments (MySQL) in MonetDB. A complete rewrite of the catalog and transaction handling removed the overhead encountered in small interactions. The SQL transaction scheme is based on three BATs (stable,inserts,deletes) which are merged at the end of a transaction. MonetDB/XQuery performance has been boosted using query/result caching and a rewrite of the code to manage the internal representation of MIL programs. Further, MonetDB/XQuery no also supports XQuery module definitions. XQuery is currently actively used in large scale applications and is stable. The MonetDB software base has been split to ease source distribution. The tools to prepare the source compilation (Mx), source code generators (MEL, Burg) and makefile generators (autogen) are assembled in the buildtools package. They are not needed to recompile source code distributed. ! TODO: ! Actually make the latter work. ! For now also the source distributions (still) require the buildtools package. In the kernel area we have added the 'constant' module. It provides functions to work with BATs whose tail consist of a constant value. During this period we have solved numerous bugs reported by our users. In addition we organized 3 bug testing days to encapsulate the problems reported into scripts that became part of our nightly testing scheme. Several smaller performance improvements have been made to the inner core of the server. One semantic error has been the cause of confusion. Bats with signature [void,void] are now allowed. A final clean-up of the respective semantic is currently being implemented; a detailed description of the changes will follow soon. Lack of manpower, expiring access to external machines, ceasing system maintenance and decrepit hardware forced us to stop nightly testing on IBM AIX, SGI IRIX and Sun Solaris. Thus, all active support for these (aging) platforms had to be suspended, too, but could easily be revived on request, once we (re-)gain access to such systems. Higher on our wishlist are more modern systems, especially 64-bit Windows and 64-bit Mac OS X. We welcome any donation in this area: access to respective systems, hardware, and/or (especially in case of the pending 64-bit Windows port of MonetDB) development-expertise and -time. The next release of MonetDB version 4 is scheduled for March 2006. By that time we also plan to release the alpha version of the next generation of MonetDB, version 5. Regards, The MonetDB Development Team -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5 12

[Monetdb-developers] CVS sql tree
by Fabian Groffen 23 Dec '05

23 Dec '05

Hi all, Unfortunately I seem to have broken SF's CVS system somehow. I tried to approve a test, but got in a 'hang' situation, which I killed after an hour of waiting. This didn't release some lockfile on the server, hence: | cvs diff: [17:27:03] waiting for mr-meltdown's lock in | /cvsroot/monetdb/sql/src/test/BugTracker/Tests Regardless how long I wait, it keeps spitting out this message. I made sure no connections are open with the SourceForge CVS server from my side (e.g. ps ax | grep sf.net) so I guess somehow I managed to melt down the system there causing a stale lockfile. If anyone knows how to resolve it, please speak up or fix it. I'm affraid one of the regular SF downtimes will not solve this stale lock thing.

2 3