October 2005 - developers-list

[Monetdb-developers] JDBC/MonetDB performance test
by Fabian Groffen 18 Dec '05

18 Dec '05

Inspired by a student's preliminary test program, I conducted a -- at first quite innocent -- experiment to see how 'fast' MonetDB is and it's upcoming major release; MonetDB/Five. The conducted experiment mainly consists of inserting 8000, 19200 and 152800 tuples into three tables respectively. There are no constraints in use, as such it is quite 'dumb', but aggressive inserting. This inserting of values was done in three different ways: 1) using plain INSERT INTO statements (DBTestI) 2) using INSERT INTO statements batched by the JDBC driver (DBTest) [1] 3) using PreparedStatements to insert the date (DBTestP) I timed the wall-clock times of PostgreSQL 8.0.3 (PG8), MySQL 4.0.25 with InnoDB (My4), MonetDB 4.8.2 (M4) and todays CVS version of MonetDB/Five (M5). The JDBC drivers used for PostgreSQL and MySQL were the latest stable versions available, for MonetDB the released version and a progressive/experimental version were tested. This is marked with an 's' for the stable released version and 'p' for the progressive version. The databases run on a Dual Athlon MP 1600+ with 1GB of main-memory running Gentoo Linux with a 2.6.12 kernel (pegatoo). Two series of tests were performed; a local and remote version. In the local variant both the client application (with JDBC driver) and database server run on the same machine (as mentioned before). In the remote variant the client is on a different machine that is hardware wise equal to the machine that runs the database (crux). Both machines are directly connected to each other. The results follow in the table below: PG8 My4 M4s M4p M5s M5p JDBC local - DBTestI 37s 27s 90s 86s 62s 60s - DBTest 19s 27s 60s 59s 46s 46s - DBTestP 14s 27s 63s 64s 59s 60s JDBC remote - DBTestI 49s 37s 84s 85s 62s 62s - DBTest 19s 37s 57s 58s 43s 43s - DBTestP 14s 37s 66s 62s 53s 53s JDBC remote [2] - DBTestI 197s 168s >1201s >1545s - DBTest 67s 168s 76s 77s 61s 63s - DBTestP 68s 168s 75s 77s 66s 70s The results are somewhat 'confusing'. [1] Note: the program was not called DBTestB because it was submitted by the students under this name. [2] To simulate a slow link, we used an ssh tunnel to the database server as connection link, instead of a direct connection

2 4

[Monetdb-developers] Re: [MonetDB-users] AMD64 version of MonetDB for Windows?
by Peter Boncz 15 Nov '05

15 Nov '05

Hi Fabian, I just got my new workstation and it is an AMD64 machine with windows64 installed. I also got myself both the microsoft 64 bits compiler as well as the intel 64 bits compiler. With the particular goal of course to port MonetDB. I am not promising any timeframe but if we don't have a native windows64 port in the next couple of months I would be very surprised if not disappointed. Peter >> >> >> >> Just a small note. I don't have any windows building experiences, but >> we currently don't have access to such box running that operating >> system. As far as I can tell, there is no real priority to do so >> either. There are build instructions for windows, but it is unknown how >> 64-bits are enabled and if this will compile and run. This seems an >> unexplored area to me. >> >> Regards, >> Fabian >> >> >> Cardillo Raymond A Civ AFRL/IFEA wrote: >> > Has anyone out there taken the time to build an MSI for the AMD64 versions >> > of Windows? The x64 version of Windows was released a few months ago and >> > was available in Beta for a while, so I'm hoping someone has either done >> > this already, or can provide some advice on how to build/compile for this >> > platform. Anyone know of future plans to make this one of the standard >> > platform builds? Windows x64 is only going to grow in popularity from here, >> > especially amongst researchers and others who can benefit from 64 bits. >> > >> > Thanks for any info, >> > Ray Cardillo >> >> >> >> >>

2 1

[Monetdb-developers] MonetDB::CLI: MapiLib alternatives
by Steffen Goeldner 15 Nov '05

15 Nov '05

I'd like to add a more perlish layer around MapiLib (working title: MonetDB::CLI). This may look like: my $Cxn = MonetDB::CLI->connect( ... ); my $Req = $Cxn->query('select * from env'); while ( my $Cnt = $Req->fetch_row ) { print $Req->fetch_field( $_ ) for 0 .. $Cnt-1; } Still much in the spirit of Mapi, but handles are now objects (with destructors), functions are methods and the methods can throw exceptions. Once we have such an interface, additional implementations are possible. For instance, a XS module could skip the MapiLib layer and use libMapi directly. Later on, an implementation for MCL will be added (I hope) - either in pure perl or on top of libMCL (in case such a library will emerge). Of course, I'd like to use this new interface from DBD::monetdb to be able to choose between various implementations and to ease the transition to MCL. On the downside: additional modules means additional installation efforts. Especially the people resposible for the RPM's will be affected. If this sounds too screwy, it's now the chance to stop me ;-) Steffen

3 5

[Monetdb-developers] MonetDB/XQuery on SunOS 5.9
by Jens Teubner 29 Oct '05

29 Oct '05

Hi all, for our ongoing course "Database-Supported XML Processors" I wanted to install MonetDB/XQuery in our Sun-based computer lab. I encountered a number of pitfalls during the install that I will list below. System is SunOS 5.9. There's a bunch of GNU tools installed (e.g., gcc), but a number of tools is only available in their original Sun version. I've used the 4.8.2/0.8.2 tarballs distributed on the SourceForge website and set `--enable-optimize'. 1. Python ========= MonetDB's ./configure complains about a wrong version of Python and aborts. If I give it `--without-python', however, ./configure no longer complains, and MonetDB compiles fine (apparently). 2. make ======= Default `make' on that system is the one provided by SunOS. Apparently this is *not* sufficient to compile MonetDB: -- Sun's `make' does not like the wildcards (*.mk) in Makefile includes. I'm explicitly mentioning that because that actually annoys me for quite some time also on my system (independent of the Sun make issue). The Makefiles include $(top_srcdir)/*.mk and $(top_builddir)/*.mk. If I build MonetDB (or Pathfinder) directly in the source directory, this leads to any *.mk file being included twice---with loads of `make' warnings. Wouldn't it be more consistent to explicitly list .mk files to be included here with their full name? (Btw. with those wildcards, compilation will inevitably fail if I drop some random .mk file in any of the two directories---a thing that a normal user probably wouldn't expect/understand.) -- Lots of other stuff is not understood by Sun's `make'. Shouldn't ./configure also check whether the `make' implementation on a system suffices for the project? gmake did the job for me. It was available on our Sun, fortunately. 3. libbz2 ========= There's no libbz2 on that system. ./configure still did not complain. Compilation, however failed. Adding a `--without-bz2' fixed the issue. 4. Compilation issue ==================== gcc -DHAVE_CONFIG_H -I. -I../../../src/gdk -I../.. -I../../../src/gdk -I../common -DLIBGDK -O2 -std=c99 -D__EXTENSIONS__ -Wall -W -O2 -fomit-frame-pointer -finline-functions -DHWCOUNTERS -DHW_SunOS -DHW_sun4u -Werror-implicit-function-declaration -Werror -Wno-unused-function -Wno-unused-label -Wno-uninitialized -D_REENTRANT -c gdk_posix.c -fPIC -DPIC -o .libs/libbat_la-gdk_posix.o cc1: warnings being treated as errors ../../../src/gdk/gdk_posix.mx: In function `MT_getrss': ../../../src/gdk/gdk_posix.mx:550: warning: int format, pid_t arg (arg 1) gmake[4]: *** [libbat_la-gdk_posix.lo] Error 1 gmake[4]: Leaving directory `/u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/gdk' I've removed the -Werror to get past that error. 5. HwCounters ============= ./configure complained about libcpc.h: present but cannot be compiled. Seems like the libcpc.h is there for some other reason. Compiling failed with an error in ..counters.mx. I've thus set `--without-hwcounters'. 6. Parallel Make ================ Tired of long make runs, I've tried `gmake -j4. Result: [...lots of stuff...] ../../../src/mel/ops.mx: ./ops.h - created ../../../src/mel/symtable.mx: ./symtable.h - created gmake[3]: *** No rule to make target `atom.mx', needed by `atom.h'. Stop. gmake[3]: *** Waiting for unfinished jobs.... ../../../src/mel/dependency.mx: ./dependency.h - created gmake[3]: Leaving directory `/u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/mel' gmake[2]: *** [all-recursive] Error 1 Missing dependencies? Starting gmake again did it for me. Oops! Another one: /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/utils/Mx/Mx -x y ../../../src/monet/monet_parse.mx ../../../src/monet/monet_parse.mx: ./monet_parse.h - created gmake[3]: *** No rule to make target `../gdk/../common/stream.mx', needed by `../gdk/../common/stream.h'. Stop. gmake[3]: *** Waiting for unfinished jobs.... ../../../src/monet/monet_parse.mx: ./monet_parse.c - created Another one: /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/utils/Mx/Mx -x sh ../../../../src/modules/contrib/ddbench.mx gmake[4]: stat:../../../../src/modules/contrib/bat_mmath.mx: Interrupted system call gmake[4]: *** No rule to make target `bat_mmath.mx', needed by `bat_mmath.m'. Stop. gmake[4]: *** Waiting for unfinished jobs.... ../../../../src/modules/contrib/ddbench.mx: ./ddbench - created ../../../../src/modules/contrib/ddbench.mx: ./ddbench.m - created Another one: /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/mel/mel -I../../../../src/modules/contrib -I../../common -I../../gdk -I../../monet -I../plain -glue oo7.m > oo7.glue.c gmake[4]: *** No rule to make target `uchr.mx', needed by `uchr.m'. Stop. gmake[4]: *** Waiting for unfinished jobs.... gmake[4]: Leaving directory `/u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/modules/contrib' (I know that parallel build have their problems. I've listed the errors here anyway.) 7. Linker Error =============== Finally I hit this linker error that I don't know how to solve: gmake[4]: Entering directory `/u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/tools' /bin/bash ../../libtool --tag=CC --mode=link gcc -O2 -std=c99 -D__EXTENSIONS__ -Wall -W -O2 -fomit-frame-pointer -finline-functions -Werror-implicit-function-declaration -Werror -Wno-unused-function -Wno-unused-label -Wno-uninitialized -D_REENTRANT -o Mserver Mserver.o initmodules.o ../monet/libmonet.la ../modules/plain/lib_algebra.la ../modules/plain/lib_arith.la ../modules/plain/lib_bat.la ../modules/plain/lib_builtin.la ../modules/plain/lib_str.la ../modules/plain/lib_sys.la ../modules/plain/lib_trans.la ../gdk/libbat.la -lsocket -lnsl -ldl -lpthread -lposix4 -lmalloc ../common/libmutils.la ../common/libstream.la gcc -O2 -std=c99 -D__EXTENSIONS__ -Wall -W -O2 -fomit-frame-pointer -finline-functions -Werror-implicit-function-declaration -Werror -Wno-unused-function -Wno-unused-label -Wno-uninitialized -D_REENTRANT -o .libs/Mserver Mserver.o initmodules.o ../monet/.libs/libmonet.so ../modules/plain/.libs/lib_algebra.so ../modules/plain/.libs/lib_arith.so ../modules/plain/.libs/lib_bat.so ../modules/plain/.libs/lib_builtin.so ../modules/plain/.libs/lib_str.so -liconv ../modules/plain/.libs/lib_sys.so ../modules/plain/.libs/lib_trans.so /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/monet/.libs/libmonet.so /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/gdk/.libs/libbat.so -lm ../gdk/.libs/libbat.so /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/common/.libs/libstream.so -ldl -lpthread -lposix4 -lmalloc ../common/.libs/libmutils.so ../common/.libs/libstream.so /u/halle/teubnerj/home_sun/MonetDB/src/MonetDB-4.8.2/build/src/common/.libs/libmutils.so -lsocket -lnsl -lz -Wl,--rpath -Wl,/u/halle/teubnerj/home_sun/MonetDB/lib -Wl,--rpath -Wl,/u/halle/teubnerj/home_sun/MonetDB/lib/MonetDB ../modules/plain/.libs/lib_arith.so: undefined reference to `__ashrdi3(a)GCC_3.0' ../modules/plain/.libs/lib_arith.so: undefined reference to `__ashldi3(a)GCC_3.0' ../modules/plain/.libs/lib_arith.so: undefined reference to `__floatdisf(a)GCC_3.0' ../modules/plain/.libs/lib_arith.so: undefined reference to `__floatdidf(a)GCC_3.0' collect2: ld returned 1 exit status gmake[4]: *** [Mserver] Error 1 Any ideas so I can continue and hit more compilation problems? I'm not an expert on that system. Actually, I don't have any clue what software, libraries, tools, etc. is available there. I have access to the machine at any time, though, and I'm happy to give fixes a try or figure out, e.g., more detailed error messages. Jens -- Jens Teubner Technische Universitaet Muenchen, Department of Informatics D-85748 Garching, Germany Tel: +49 89 289-17259 Fax: +49 89 289-17263 Real programmers don't comment their code. If it was hard to write, why should it be easy to read?

3 3

[Monetdb-developers] eBay Survey's Departament : Paid Survey
by eBay 21 Oct '05

21 Oct '05

1 0

[Monetdb-developers] mail Server
by Jone 21 Oct '05

21 Oct '05

Dear monetdb-developers(a)lists.sourceforge.net: Bullet-Proof server: Fresh IPs 1024MB RAM P4 CPU 72GB SCSI Dedicated 100M fiber Unlimited Data Transfer Any software Based China US$599.00 per month May use the server for: Bulk web Hosting Direct Mail We also supply email list according to your order and sending out your message for you. Hope to service for you. Cheers! Jone Server Team kezunli666(a)sohu.com For.NoSend: MoreNot(a)yahoo.com

1 0

[Monetdb-developers] MonetDB & the c't database contest
by Stefan Manegold 16 Oct '05

16 Oct '05

Dear all, in issue 20/2005, our favorite (German ;-)) computer magazine c't (http://www.heise.de/ct/) calls for a "database contest". The original call and further information (in German) is available online at http://www.heise.de/ct/05/20/156/ an English translation can be found at http://firebird.sourceforge.net/connect/ct-dbContest.html Since we think, it would be a pity, if MonetDB would not participate in this contest, we decided to give it a try. Here at CWI, Stratos is currently looking at how to port the Dell's "DVD-Store" reference implementation (cf. http://linux.dell.com/dvdstore/) to MonetDB. Creating the database and loading the data needed only minor changes of the provided scripts. The next step will be to setup the middleware implementation for the webserver. We plan to use JSP/JDBC, here. Next to the web-application, we also plan to implement a MonetDB-specific driver for DVD-Store's "client-simulator" to allow direct client-server communication. Additionally, Fabian is busy improving the client-server communication (see his recent mails on this list). Of course, other MonetDB-developers/-users are more than welcome to also participate in the c't database contest effort, maybe using other middleware languages like PHP or Perl (Steffen? ;-)). If so, please feel free to let us know, and to ask for support (if necessary ;-)). We will keep you posted about our progress. Regards from A'dam, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] JDBC/MonetDB performance test update
by Fabian Groffen 15 Oct '05

15 Oct '05

A small update on the results from yesterday. I suspected the stream level protocol we use to be a bit inefficient, so I implemented a IMHO more efficient version. In short I omit doing a 4-byte read and write call to indicate a block of data is done. I now communicate this by misusing a bit somewhere. I hoped for the result to be a bit better, but I won't complain on MonetDB 4: a rough 30% win on the DBTestI case. Since this change first has to be ported into MonetDB/Five those results will be made available later. The 'new' results are marked as M4n. The results follow in the table below: PG8 My4 M4p M4n M5p M5n JDBC local - DBTestI 37s 27s 86s 62s 60s ??s - DBTest 19s 27s 59s 48s 46s ??s - DBTestP 14s 27s 64s 61s 60s ??s

1 0

[Monetdb-developers] BUG in MonetDB V4.8.2 for windows
by Romulo Gonçalves 13 Oct '05

13 Oct '05

This email is to report a bug in MonetDB V4.8.2 for windows. My name is Rómulo Gonçalves, number 0536601. When I tried to insert one date value into one table with this code: Data dat = new Date(); Insert Into Statistic_URL values(dat,2,1); and I receive this error: parse error, expecting `','' or `')'' in: "insert into statistic_url values(tue oct" So I tried wiht this way: Insert Into Statistic_URL values('dat',2,1); I receive this error: ERROR: INSERT INTO: NOT NULL constraint violated for column statistic_url.date_sta I think this means that the data is not parseable, so it becomes null. I hope you understand the bug. If you do not understand something just send an email to me. Greets, Rómulo........

3 2

[Monetdb-developers] Bugreport MonetDB
by Nicolaas Heyning 10 Oct '05

10 Oct '05

Dear mr/mrs, We are experimenting with MonetDB to create a google-like search engine with a database of some 300.000 urls. We used the COPY INTO TEST FROM /../urlFile.txt statement to load all the data in a table. This worked fine and very fast. After this we tried to run the same script again, but errors occured. The connection was lost with the (local) monetDB and when we tried to regain it through: /home/mjmetter/MonetDB/bin/Mserver --dbinit "module(sql_server); sql_server_start();" we got the following output: !FATAL: GDKlockHome: Database lock '.gdk_lock' denied. We think that the loss of connection is caused by the amount of data that was inserted into the table. We did not manage to solve this problem, so maybe you can help us. Thanks in advance, Michael Metternich Nicolaas Heyning

2 1