- developers-list - monetdb.org

[Monetdb-developers] script postgresSQL to Monetdb
by Roberto De Virgilio 09 Jun '08

09 Jun '08

Hi i have this problem.. i have to convert a dump file (.sql) of a PostgreSQL Db into a dump file for MonetDB to create the same DB of postgres into MonetDB... do you help me with a script? thanks a lot ____________________________________________________________________________________________________________ Roberto De Virgilio Ph.D Università Roma Tre Dipartimento di Informatica e Automazione Via della Vasca Navale 79, 00146 Roma Phone: +39-06-55173229 Fax: +39-06-5573030 E-mail: rde79(a)yahoo.com Home Page: http://mais.dia.uniroma3.it/rodevirg Room: 219

1 0

[Monetdb-developers] windows problem...
by Romulo Goncalves 09 Jun '08

09 Jun '08

I removed all the dbfarm, sql_logs and all the lib_* files.... Clean installation and I continue to get: E:\MonetDB\Current\sql\NT>RunMserver5.bat !WARNING: GDKlockHome: created directory E:\MonetDB\Current\\sql\NT\dbfarm\demo\ !WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # MonetDB server v5.6.0, based on kernel v1.24.0 # Serving database 'demo' # Compiled for i686-pc-win32/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-2008 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see E:\MonetDB\Current\MonetDB5\NT\etc\monetdb5.conf) !FATAL: MALException:addUser:user 'monetdb' already exists E:\MonetDB\Current\sql\NT> It seems the bats are somewhere, the problem is where. The search of windows does not detect any dbfarm... Any suggestion? Romulo

2 1

[Monetdb-developers] Fastest way to calculate median?
by Rt Ibmer 09 Jun '08

09 Jun '08

Hi - I have an int column with several million rows of data. What is the fastest way to write a query to get the median value of the column? Apparently there is no median() function in SQL. A web search revealed lots of various talk about using stored procedures and fancy functions. None of this looked like it would be very efficient. I searched these archives but didn't see a single mention of median. Thank you!

2 1

[Monetdb-developers] Bug report - COPY RECORDS and encoded data
by Rt Ibmer 09 Jun '08

09 Jun '08

I was going crazy trying to figure out why my COPY RECORDS command wasn't working. Turns out it is caused from funny (encoded?) characters in the string for one of the fields. For instance one of the fields contained this "?a?as?e??, 18 ?a??? 2001 10:24:00 pµ" How did the data get this way? Well I exported it from SQL Server and I guess that db is set up to store encoded characters. When I did the export I told SQL Server to make the file ASCII since I figured that is what COPY RECORDS expects. So apparently SQL Server is writing the characters this way into the string. Then MonetDB (latest version) cannot handle this. So what happens is that the COPY x RECORDS command just returns immediately after I try this with this data in it. After that all my SQL statements in mclient return immediately. So I can say "select count(*) from foo;" and it just comes immediately back with the sql> prompt. So something with these funny characters sends mclient into a funk. I have to exit and restart it to get it going again. Any advice on how to handle this? Thank you.

3 2

[Monetdb-developers] Calculate disk space for table?
by Rt Ibmer 08 Jun '08

08 Jun '08

Hi - how can I tell how much disk storage space on of my tables is using? I tried one rough way but not sure if it is accurate. Basically I did a linux du command and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data? There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way. Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods? And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making? Thank you!

3 3

[Monetdb-developers] Division by zero on this query - why?
by Rt Ibmer 08 Jun '08

08 Jun '08

Any ideas why the following query would result in a division by zero exception? sql> select prodcode, avg(price) from sales group by prodcode; !ArithmeticException:batcalc./:Division by zero Timer 24326.296 msec 0 rows Thank you.

1 0

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime pf_support.mx, XQuery_0-24, 1.299.2.3, 1.299.2.4
by Stefan Manegold 06 Jun '08

06 Jun '08

On Fri, Jun 06, 2008 at 09:00:53PM +0000, Lefteris Sidirourgos wrote: > Update of /cvsroot/monetdb/pathfinder/runtime > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv9671/runtime > > Modified Files: > Tag: XQuery_0-24 > pf_support.mx > Log Message: > A rather aggresive fix for the order of attributes and nodes in updates. > > Now we reverse in algebra the order of insert as last and insert before at runtime like the MPS. This fix the bug [1981844] PF/alg: result order differs from mps. Also it fix the test Update/update.xq (which makes unnecessary the algebra specific output). Also other failing tests of updates are fixed in tests/BugTrack. > > However, now the test insert_test_order_seq fails:( However, this is a different error that should be addressed with a diffrent fix. > > To be continued... > > > U pf_support.mx > Index: pf_support.mx > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/runtime/pf_support.mx,v > retrieving revision 1.299.2.3 > retrieving revision 1.299.2.4 > diff -u -d -r1.299.2.3 -r1.299.2.4 > --- pf_support.mx 29 May 2008 16:45:15 -0000 1.299.2.3 > +++ pf_support.mx 6 Jun 2008 21:00:46 -0000 1.299.2.4 > @@ -3321,9 +3321,27 @@ > { > var inserts := command.ord_select(UPDATE_INSERT_FIRST, UPDATE_REPLACENODE); # [i,insertcommand] > if (inserts.count() > 0) { > + > + # UPDATE_INSERT_LAST and UPDATE_INSERT_BEFORE commands must be executed in reverse order > + var order := inserts.ord_uselect(UPDATE_INSERT_LAST, UPDATE_INSERT_BEFORE).copy().mark(0@0); > + if (order.count() > 1) { > + pre_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + pre_cont_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_cont_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + attr_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + attr_cont_tgt.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_cont_tgt).access(BAT_WRITE).revert().tmark(0@0))); > + pre_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_ins).access(BAT_WRITE).revert().tmark(0@0))); > + pre_cont_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(pre_cont_ins).access(BAT_WRITE).revert().tmark(0@0))); > + attr_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_ins).access(BAT_WRITE).revert().tmark(0@0))); > + attr_cont_ins.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(attr_cont_ins).access(BAT_WRITE).revert().tmark(0@0))); > + inserts.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(inserts).access(BAT_WRITE).revert().tmark(0@0))); > + command.access(BAT_WRITE).replace(order.leftjoin(order.mirror().leftjoin(command).access(BAT_WRITE).revert().tmark(0@0))); > + } Lefteris, thanks for the fix! I'm not completely sure, whether I understand what your above code does, or better, whether I understand how is does what *I think* it is supposed to do, i.e. revert the order of BUNs that are related to UPDATE_INSERT_LAST and UPDATE_INSERT_BEFORE commands in all BATs. However given my idea of what it is supposed to do, wouldn't the following solution look "simpler" and be more efficient? ======== var order := inserts.ord_uselect(UPDATE_INSERT_LAST, UPDATE_INSERT_BEFORE); if (order.count() > 1) { var redro := order.copy().access(BAT_WRITE).revert().access(BAT_READ); var order_redro := order.mark(0@0).leftfetchjoin(redro.tmark(0@0)); pre_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(pre_tgt) ).access(BAT_READ); pre_cont_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(pre_cont_tgt) ).access(BAT_READ); attr_tgt .access(BAT_WRITE).replace( order_redro.leftjoin(attr_tgt) ).access(BAT_READ); attr_cont_tgt.access(BAT_WRITE).replace( order_redro.leftjoin(attr_cont_tgt) ).access(BAT_READ); pre_ins .access(BAT_WRITE).replace( order_redro.leftjoin(pre_ins) ).access(BAT_READ); pre_cont_ins .access(BAT_WRITE).replace( order_redro.leftjoin(pre_cont_ins) ).access(BAT_READ); attr_ins .access(BAT_WRITE).replace( order_redro.leftjoin(attr_ins) ).access(BAT_READ); attr_cont_ins.access(BAT_WRITE).replace( order_redro.leftjoin(attr_cont_ins) ).access(BAT_READ); inserts .access(BAT_WRITE).replace( order_redro.leftjoin(inserts) ).access(BAT_READ); command .access(BAT_WRITE).replace( order_redro.leftjoin(command) ).access(BAT_READ); } ======== ... just an idea; might be completely wrong ... let's think about anc check it tomorrow or "live" on Monday ... Good night > + > var inserts_pre_ins := inserts.mirror().leftfetchjoin(pre_ins); # [i,PRE] > var inserts_elems := inserts_pre_ins.ord_uselect(oid_nil, oid_nil).hmark(0@0); # [j,i] inserts on elements > var inserts_attrs := inserts_pre_ins.ord_uselect(oid_nil).hmark(0@0); # [k,i] inserts on attributes > + > + > if (inserts_attrs.count() > 0) { > do_update_insert_attributes(ws, inserts_attrs.leftjoin(pre_tgt), inserts_attrs.leftjoin(pre_cont_tgt), inserts_attrs.leftjoin(attr_ins), inserts_attrs.leftjoin(attr_cont_ins)); > } > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-sql-checkins] sql/src/backends/monet5 sql_optimizer.mx, , 1.182, 1.183 sql_scenario.mx, , 1.308, 1.309
by Stefan Manegold 06 Jun '08

06 Jun '08

On Fri, Jun 06, 2008 at 08:25:18PM +0000, Martin Kersten wrote: > Update of /cvsroot/monetdb/sql/src/backends/monet5 > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv27738 > > Modified Files: > sql_optimizer.mx sql_scenario.mx > Log Message: > The first nightly test of the recycler ... will have to wait until Sunday: To have more time for (hopefully) migrating MonetDB testing to a new "server", I decided to run only Stable testing tonight, and "skip" Current testing for one night, assuming that Stable testing is currently more "mission-critical" than Current testing. Sorry for the inconveniences this might cause ... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal recycle.mx, , 1.27, 1.28
by Romulo Goncalves 06 Jun '08

06 Jun '08

Milena Ivanova wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv27013/mal > > Modified Files: > recycle.mx > Log Message: > Reset is called after an update instruction to clean the recycle cache. > > > U recycle.mx > Index: recycle.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/recycle.mx,v > retrieving revision 1.27 > retrieving revision 1.28 > diff -u -d -r1.27 -r1.28 > --- recycle.mx 5 Jun 2008 08:37:43 -0000 1.27 > +++ recycle.mx 6 Jun 2008 14:00:22 -0000 1.28 > @@ -70,7 +70,7 @@ > command getCachePolicy():int > address RECYCLEgetCache; > > -pattern reset():void > +pattern reset(bid:bat):void Is the optimized aware of this change? It might call for the command with wrong number of arguments and it might break several sql tests... Romulo > address RECYCLEresetCMD > comment "Reset off all recycled variables"; > > @@ -78,11 +78,6 @@ > address RECYCLEshutdownWrap > comment "Clear the recycle cache"; > > -command restore():void > -address RECYCLErestore > -comment "Restore the recycle cache to the start of the call"; > - > - > command monitor(f:int):void > address RECYCLEmonitor > comment "start/stop the monitoring (printing) of the recycler info (storage size used and number of statements retained)"; > @@ -207,7 +202,7 @@ > > /* and dump the statistics */ > if (recycleBlk) > - for(i=0; i< recycleBlk->stop; i++){ > + for(i=1; i< recycleBlk->stop; i++){ > stream_printf(cntxt->fdout,"# %4d lru= %8d cnt= %8d ticks= %9d rd= %9d wr= %9d %s\n", i, > recycleBlk->profiler[i].clk, > recycleBlk->profiler[i].counter, > > > ------------------------------------------------------------------------- > Check out the new SourceForge.net Marketplace. > It's the best place to buy or sell services for > just about anything Open Source. > http://sourceforge.net/services/buy/index.php > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 0

[Monetdb-developers] Default of "--enable-strict/--disable-strict" in release branch
by Stefan Manegold 06 Jun '08

06 Jun '08

Dear all, I do see the point that we do not necessarily distribute released source tarballs with "--enable-strict" as default, as this might "unnecessarily" hinder compilation with compilers and/or environments that "we don't know yet". However, nightly testing does explicitly use --enable-strict both both the development trunk ("Current") and the release branch ("Stable") to ensure "correctness" of the code. This leads to the "inconvenient" situation that, during bug fixing, developers that do not explicitly use "--enable-strict" compile with the default "--disable-strict" and consequently might miss small (but "nasty") problems that then cause nightly testing to fail. To solve this "dilemma", I came up with the idea to make the default setting "more flexible" and "automatically detect" whether a developer is compiling from CVS or a user is compiling from source tarball(s). In the former case, the default is "--enable-strict", it the latter it is "--disable-strict". Here's my patch: ======== --- buildtools/conf/MonetDB.m4 24 May 2008 17:13:41 -0000 1.75.2.1 +++ buildtools/conf/MonetDB.m4 6 Jun 2008 06:14:32 -0000 @@ -17,7 +17,15 @@ dnl Defaults that differ between development trunk and release branch: AC_DEFUN([AM_MONETDB_DEFAULTS], [ -dft_strict=no +if test -f "$srcdir"/vertoo.data; then + dnl Developers compiling from CVS: + dnl default is --enable-strict + dft_strict=yes +else + dnl Users compiling from source tarball(s): + dnl default is --disable-strict + dft_strict=no +fi dft_assert=no dft_optimi=yes dft_warning=no ======== In case there are no (strong) objections, I'd like to check this in later today. Open question: shall we only do this on the release branch and keep "--enable-strict" as default in all cases in the development trunk, or shall we apply the same strategy o the development trunk, too? Yours, Stefan On Fri, Jun 06, 2008 at 06:45:45AM +0200, Testweb Compilation wrote: > opt_icol.c:376: warning: enumeration value 'la_dummy' not handled in switch > > 374 if (L(p)->kind == la_twig && > 375 !PFprop_icol (p->prop, L(p)->sem.iter_item.item)) > 376 switch (LL(p)->kind) { > 377 > 378 case la_docnode: > > /home/monet/testing/monet.GNU.32.32.d.2271/pathfinder/compiler/algebra/opt/opt_icol.c:376: warning: enumeration value 'la_string_join' not handled in switch > /home/monet/testing/monet.GNU.32.32.d.2271/pathfinder/compiler/algebra/opt/opt_icol.c:376: warning: enumeration value 'la_dummy' not handled in switch > make[8]: *** [libopt_la-opt_icol.lo] Error 1 > > http://www.ins.cwi.nl/~monet/permastore/2008-06-06/Stable/make-pathfinder-G… > > CVS committers: boncz(a)cwi.nl jan.rittinger(a)in.tum.de lsidir(a)cwi.nl manegold(a)cwi.nl sjoerd(a)cwi.nl zhang(a)cwi.nl > > An identical error message was detected on the following systems: > GNU.32.32.d-Debian4.0 GNU.32.32.d-Fedora6 GNU.32.32.d-Fedora8 GNU.64.32.d-Fedora8 GNU.64.64.d-Fedora8 GNU.64.64.s-Fedora8 > > -- > At this time you cannot configure Testing mail, sorry > You are receiving this mail because: > - you checked in some code on pathfinder (Stable) > - you are a fixed recipient > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

4 8