- developers-list - monetdb.org

[Monetdb-developers] Segmentation Fault upon sql.start()
by Stefan de Konink 22 Oct '07

22 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 After loading in a bunch of data the monetdb server was Killed due to memory starvation. With swap MonetDB performed the huge amount of one insert in 10 seconds, so that wasn't an option too. I restarted it, and tried to 'load the data again'. Backtrace: gdb ./mserver5 GNU gdb 6.7 Copyright (C) 2007 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu"... Using host libthread_db library "/lib/libthread_db.so.1". (gdb) run Starting program: /opt/monetdb/bin/mserver5 warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff275fe000 [Thread debugging using libthread_db enabled] [New Thread 0x2afd85f0f2a0 (LWP 23929)] # MonetDB Server v5.0.0 # Copyright (c) 1993-2007 CWI, all rights reserved # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # dbname:osm # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /opt/monetdb/etc/monetdb5.conf) [New Thread 0x40800950 (LWP 23932)] >include sql; >sql.start(); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x2afd85f0f2a0 (LWP 23929)] 0x00002aaaaab40a8f in load_column () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so (gdb) bt #0 0x00002aaaaab40a8f in load_column () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #1 0x00002aaaaab41a71 in load_table () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #2 0x00002aaaaab433cc in load_trans () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #3 0x00002aaaaab46e15 in store_init () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #4 0x00002aaaaab1b95b in mvc_init () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #5 0x00002aaaaaad0b50 in SQLinit () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #6 0x00002afd8389f476 in initScenario () from /opt/monetdb/lib/libmal.so.0 #7 0x00002aaaaaad0ac8 in SQLsession () from /opt/monetdb/lib/MonetDB5/lib/lib_sql.so #8 0x00002afd8387842d in runMALsequence () from /opt/monetdb/lib/libmal.so.0 #9 0x00002afd83879dfd in runMAL () from /opt/monetdb/lib/libmal.so.0 #10 0x00002afd83871dc8 in MALengine () from /opt/monetdb/lib/libmal.so.0 #11 0x00002afd8389f082 in runScenarioBody () from /opt/monetdb/lib/libmal.so.0 #12 0x00002afd8389f0d2 in runScenario () from /opt/monetdb/lib/libmal.so.0 #13 0x00002afd838719e0 in MSserveClient () from /opt/monetdb/lib/libmal.so.0 #14 0x0000000000402c14 in main () (gdb) It work doesn't anymore... I can conclude, inserting a lot of data start to become a huge waste of time. If required I can recompile with the -g option and a lower amount of optimization. Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHHQTsYH1+F2Rqwn0RCjWKAJ9u/OEu8Sk6HTSY+v431bLjmylrCwCffsNN Rpf0Fs4XjrakYkdyEwkknWQ= =0UHS -----END PGP SIGNATURE-----

2 2

[Monetdb-developers] python development
by Sjoerd Mullender 22 Oct '07

22 Oct '07

For all you Python developers, a new rule is being enforced from now on: Python source files are not allowed to contain TAB characters or trailing white space. This is enforced at cvs commit time. -- Sjoerd Mullender

1 0

[Monetdb-developers] man-pages
by Fabian Groffen 22 Oct '07

22 Oct '07

We should put or html-ified (autogenerated) man-pages for merovingian, monetdb and mclient on our website, as easy to read documentation.

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/testing Mtest.py.in, MonetDB_1-20, 1.309.2.8, 1.309.2.9
by Fabian Groffen 22 Oct '07

22 Oct '07

On 21-10-2007 16:36:26 +0000, Niels Nes wrote: > Update of /cvsroot/monetdb/MonetDB/src/testing > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv16749/src/testing > > Modified Files: > Tag: MonetDB_1-20 > Mtest.py.in > Log Message: > export TST_FIVE, we sometimes need to know which server we test > > > Index: Mtest.py.in > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB/src/testing/Mtest.py.in,v > retrieving revision 1.309.2.8 > retrieving revision 1.309.2.9 > diff -u -d -r1.309.2.8 -r1.309.2.9 > --- Mtest.py.in 19 Oct 2007 13:41:24 -0000 1.309.2.8 > +++ Mtest.py.in 21 Oct 2007 16:36:21 -0000 1.309.2.9 > @@ -1237,6 +1237,7 @@ > os.environ['TST_MODS'] = str(env['TST_MODS']) > if par['M5']: > env['TST_FIVE'] = "Five" > + os.environ['TST_FIVE'] = "Five" This looks like an upcoming IndentationError to me. > return rtrn > ### GetBitsAndOIDsAndModsAndStatic(env) #

2 2

[Monetdb-developers] Mserver5 issues / Perl patches
by Stefan de Konink 21 Oct '07

21 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hi, [If the first part should be in monetdb-users please inform me] Currently I am still using the same setup, because I do not have the time to reimport my data using after the nightly builds. After my first big import in SQL resulting in 4*10^7 rows, I decided to move on. And continue my work. I have started mserver5 again, include sql; [no problems] sql.start(); [this command took 10 minutes to be processed] After this 10 minutes, the command works instantly when running the server again. Is this some kind of integrity checking? In my last e-mail I asked about the Perl issues, Niels pointed that the Perl sources were out of sync. I took the liberty to try to come up with a fix, please check as attached. I think no copyright can be claimed on 3 changes, so if this is the right solution :) Please apply :) Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHGqTsYH1+F2Rqwn0RCnZqAJ9l4IMVeGOCMm7JJN0BijnCvEIrAQCdEDd0 8zEe0HfX1ygvsJdawvOC63Q= =RqZ8 -----END PGP SIGNATURE-----

2 1

[Monetdb-developers] Perl documentation and connectivity
by Stefan de Konink 20 Oct '07

20 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Hello, I'm trying the first lines of the Perl example presented: http://monetdb.cwi.nl/projects/monetdb/SQL/Documentation/The-Perl-Library.h… # connect to the database: my $dsn = 'dbi:monetdb:database=test;host=localhost;port=50000;language=mil'; my $dbh = DBI->connect( $dsn, undef, undef, # no authentication in MIL { PrintError => 0, RaiseError => 1 } # turn on exception handling ); skinkie@xen01 ~/private $ perl test.pl Start a simple Monet MIL interaction data sources: dbi:monetdb: DBI connect('database=test;host=localhost;port=50000;language=mil','',...) failed: RuntimeError Usage: mapi_connect(host,port,username,password,lang,dbname); at test.pl line 15 (I have traced this back to: MonetDB/CLI/MapiLib.pm) Is there some trivial thing I have missed? (I'm definitely not a Perl hero so forgive me for my incompetence.) Yours Sincerely, Stefan de Konink ps. Running: # MonetDB Server v5.0.0 # Copyright (c) 1993-2007 CWI, all rights reserved # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # dbname:demo ps2. I know test!=demo -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHGU0jYH1+F2Rqwn0RCvlvAJ0XbNzyV8ySOyoNcKbmitNWUSwH1gCfaDma /kMe7PQ4nl9cRhMcKpun2Uw= =BP78 -----END PGP SIGNATURE-----

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/compiler/debug logdebug.c, XQuery_0-20, 1.72, 1.72.2.1
by Stefan Manegold 18 Oct '07

18 Oct '07

Sjoerd, not that I want to "worry" you, but just out of curiosity: did this one make it into the release? If not, I would not recomment to re-build the release once again --- unless Jan has very strong arguments that this is a strickt show-stopped (don't think so, since most tests work fine) ---, but just keep it for a potential bug-fix release. Stefan On Wed, Oct 17, 2007 at 04:29:31PM +0000, Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/compiler/debug > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv26773/debug > > Modified Files: > Tag: XQuery_0-20 > logdebug.c > Log Message: > -- Bugfix: Added missing 'break;'. > > > Index: logdebug.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/compiler/debug/logdebug.c,v > retrieving revision 1.72 > retrieving revision 1.72.2.1 > diff -u -d -r1.72 -r1.72.2.1 > --- logdebug.c 18 Sep 2007 11:46:58 -0000 1.72 > +++ logdebug.c 17 Oct 2007 16:29:28 -0000 1.72.2.1 > @@ -1494,6 +1494,7 @@ > PFatt_str (n->sem.pos_sel.part)); > > PFarray_printf (xml, " </content>\n"); > + break; > > case la_fun_1to1: > PFarray_printf (xml, > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 1

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/src/sqlhelpers/xmlshred encoding.c, XQuery_0-20, 1.8.4.3, 1.8.4.4
by Sjoerd Mullender 18 Oct '07

18 Oct '07

Jan Rittinger wrote: > Update of /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv19535 > > Modified Files: > Tag: XQuery_0-20 > encoding.c > Log Message: > -- Bugfix: Added missing quotes. > > > Index: encoding.c > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/src/sqlhelpers/xmlshred/encoding.c,v > retrieving revision 1.8.4.3 > retrieving revision 1.8.4.4 > diff -u -d -r1.8.4.3 -r1.8.4.4 > --- encoding.c 9 Oct 2007 19:54:07 -0000 1.8.4.3 > +++ encoding.c 18 Oct 2007 08:02:23 -0000 1.8.4.4 > @@ -146,7 +146,7 @@ > if (shredstate.names_separate) > fprintf(out, "%i", tuple.name_id); > else > - fprintf(out, "%s", (char*)tuple.name); > + fprintf(out, "\"%s\"", (char*)tuple.name); > } > break; > case 't': > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins Too late. I have built the installers, RPMs, and tar balls. This change is not in it. Unless it is a show stopper, in which case I will be extremely unhappy, but seeing that it is in the sqlhelpers directory, I don't believe that. -- Sjoerd Mullender

2 1

[Monetdb-developers] next release: mps or algebra?
by Peter Boncz 17 Oct '07

17 Oct '07

Stefan suggested rightly to move this discussion to the mailing list. I also changed the language into english for that purpose. Maurice' original questions were: (a) whether the new release would be algebra-only (short answer: hopefully). (b) whether our talk of a new release also refers to submodules, such as probxml and pftijah (short answer: yes) (c) whether all issues on the wiki, see: http://www.pathfinder-xquery.org/wiki/index.php/Jupiter:_The_Algebra_Release needed for this action, have already been resolved (short answer: no) On Tue, Oct 16, 2007 at 05:23:43PM +0200, Peter Boncz wrote: > Hi Maurice, et al, > > The release of (hopefully) this week will still be mps based. I also > anticipate at least one bug-fix release (ie 0.20.2), or even more. However, > the next major release (0.22) should be algebra based, in my opinion. I am > certainly reluctant to keep investing any more effort in mps. I would aim > for an mps-free release, yes. > > The time-goal of that release is "XMAS 2007", as far as we are concerned. > For the work of built-in functions I have good hope that with the help of > Lefteris we can come a long way. > > I hope and think updates can also be ported by joerd in that time-frame. As > for pftijah, if Jan Flokstra has time, it should be possible to port that as > well in 2-3 months. > > There are some hairy issues, one of them being free-form recursion (needed > by you), another one being precompiled modules (needed by XRPC). The > resolution of these needs assistance from Munich, and maybe related. It needs > a method to compile certain functions into scripted procedures (here MIL) > and calling them. One algorithm is to do this for all functions in modules, > as well as for all recursive functions. The issue to resolve is the > representation of the parameters. > > Depending on how much time Jan Rittinger has available for this, we will > have to find creative solutions for this. Indeed, a backup solution would be > to keep handling certain queries for the time being with mps, but that > surely is ugly. > > Peter > > -----Original Message----- > From: Keulen, M. van (Maurice) [mailto:m.vankeulen@utwente.nl] > Sent: dinsdag 16 oktober 2007 16:38 > To: Peter Boncz [CWI]; Stefan Manegold [CWI] > Cc: Djoerd Hiemstra; Jan Flokstra; Henning Rode; Abdel Kader, R. (Riham) > Subject: Nieuwe releases en mps-backend > > > Hoi, > > Sorry dat ik niet bij de Skype-sessie van vandaag kon zijn, maar Riham > vertelde me dat het onderwerp van nieuwe releases en de mps-backend ter > sprake is geweest. Het riep wat specifieke vragen bij me op die ze niet > allemaal kon beantwoorden: > > a.. Is het de bedoeling dat een nieuwe release de mps-backend helemaal > niet meer bevat of alleen maar de algebra-backend als default heeft? > b.. Als jullie praten over "nieuwe release", bedoel je dan alleen een > package van alle modules van MonetDB/XQuery? Maw zegt het niets over > eventuele `subreleases' van individuele modules? > > c.. Op de wiki hebben we een pagina met open issues voor de > algebra-release. Wat is de status van die open issues? Ik zie namelijk op de > pagina nergens dat bepaalde issues al opgelost/geïmplementeerd zijn. > Ik Cc deze e-mail even aan de PF/Tijah-mensen, want hoewel er levendige > discussies zijn hoe PF/Tijah naar de algebra-backend to porten, ik heb niet > het idee dat er oplossingen zijn die op korte termijn realiseerbaar zijn. > Evenzo het recursie-vraagstuk. Het lijkt me bijvoorbeeld voor de > zichtbaarheid naar buiten toe onwenselijk als deze features ineens zouden > wegvallen voor non-insiders. Of andere onwenselijke consequenties van het > niet meer (sub)releasen van modules die gebaseerd zijn op de mps-backend. > > Groetjes, > Maurice. > > -- > ---------------------------------------------------------------------- > Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology > Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands > Email: m.vankeulen(a)utwente.nl, Phone: +31 534893688, Fax: +31 534892927 > Room: ZI 3039, WWW: http://www.cs.utwente.nl/~keulen -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

5 8

[Monetdb-developers] FW: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, 1.143, 1.143.2.1
by Peter Boncz 17 Oct '07

17 Oct '07

Thanks Stefan, > I'll leave the further interpretation of the above results to > the interested recipient / reader. I will give it a try, then. > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s These h/SU results are strange, because a 5 x faster time due to a hash function with more collisions is.. puzzling. In any case, with 64-bits oids on a 8GB machine, the shredding generates very intense swapping, and Linux performance under swapping is known to be highly variable. Difficult to draw conclusions. I am still investigating the behavior of MT_mmap under load and it may well be that shredding performance can be improved by better memory management (less swapping). From the result below we know that it can be done in 25m. The not-yet checked in improvements in index construction will strongly reduce the QU[1] time by taking less memory; in the order of the 100m seen below. > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s These numbers show that MonetDB prefers to operate on RAM resident data. I am glad you reproduced my 25 minutes h/SU result. Must be a coicidence that the mini hash self-join benchmark on 50M string numbers ran in 26s with the new hash and 43s with the old hash (we see the same numbers in minutes here repeated for shredding). In my current version, QU[2] with the ordered indices is back at 17s. Reason for 1m15s is that you tend to loose the hash-table on the index bats due to swapping. So that includes hash table creation. Whether QU[1] is acceptable is highly questionable. Starting the database takes 1.5 hours!! I am seriously considering to switch to persistent ordered indices for the updatable case. For that to happen, the worksing set needs to be extended with persistent delta-bats on those indices, changes to these deltas need to be logged (in order to make the indices recoverable). > -----Original Message----- > From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] > Sent: Wednesday, October 17, 2007 1:54 PM > To: monetdb-developers(a)lists.sourceforge.net > Cc: Peter Boncz; Peter Boncz > Subject: Re: [Monetdb-checkins] MonetDB/src/gdk gdk_atoms.mx, > MonetDB_1-20, 1.134, 1.134.6.1 gdk_posix.mx, MonetDB_1-20, > 1.143, 1.143.2.1 > > > Just for the records: > > I finally managed to finsh my experiments regarding > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > and the related code changes. For those interested, here's > the detailed > story: > > > "S08-64" System (beo-24): > - 2x 64-bit Dual-Core Opteron270 @ 2 Ghz > - 8 GB memory > - MonetDB/XQuery 0.20, 64-bit, 64-bit OIDs, --enable-optimize > (gcc 4.1.2) > > "S16-32" System (core-1): > - 4x 64-bit Dual-Core Opteron870 @ 2 Ghz > - 16 GB memory > - MonetDB/XQuery 0.20, 64-bit, 32-bit OIDs, --enable-optimize > (gcc 4.1.2) > > Document: > http://mirror.openstreetmap.nl/planet/planet-071003.osm.bz2 > (extracted: 19 GB XML file) > > "SR" Shredding read-only: > pf:add-doc(".../planet-071003.osm","planet-071003.osm") > > "SU" Shredding updateable: > pf:add-doc(".../planet-071003.osm","planet-071003.osm","planet > -071003.osm",5) > > "QR"/"QU" Count query: > count(doc("planet-071003.osm")//*) > > Configurations: > m: without Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143 of gdk_posix.mx) > M: with Peter's mmap fix in gdk_posix.mx > (i.e., using rev. 1.143.2.1 of gdk_posix.mx) > h: without Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134 of gdk_atoms.mx) > H: with Peter's new string hash function in gdk_atoms.mx > (i.e., using rev. 1.134.6.1 of gdk_atoms.mx) > > > Results (wall-clock times): > > S08-64: > SR QR SU QU[1] QU[2] > mh 659m34s 1m09s 81m06s ERROR - > mH - - 383m17s ERROR - > Mh - - 77m03s 344m32s 1m34s > MH 644m01s 0m49s 390m42s 342m47s 1m36s > > S16-32: > SR QR SU QU[1] QU[2] > mh 127m59s 0m17s 43m14s ERROR - > mH 110m33s 0m16s 26m26s ERROR - > Mh 128m11s 0m18s 44m00s 100m50s 1m15s > MH 191m42s 0m17s 25m43s 101m37s 1m21s > > (NB: "SR" includes building of indices, while "SU" does not; > consequently, "QR" can exploit the indices built during > "SR", while "QU[1]" > has to build the indices first, and only "QU[2]" can exploit them.) > > > Apparently, the mmap fix in gdk_posix.mx seems to be > sufficient to prevent > the remap-ERROR reported (for a system & configuration > similar to "S08-64") > in > [ 1811229 ] [ADT] Adding large document, with update support > http://sourceforge.net/tracker/index.php?func=detail&aid=18112 > 29&group_id=56967&atid=482468 > > I'll leave the further interpretation of the above results to > the interested > recipient / reader. > > > Stefan > >

1 0