- developers-list - monetdb.org

[Monetdb-developers] MonetDB XQuery 0.18.2
by Sjoerd Mullender 19 Jun '07

19 Jun '07

A bug fix release for MonetDB XQuery 0.18.0 has been done. Two smallish bugs have been fixed in this release: - compiling an XQuery query took much longer than warranted due to a configuration error; - pftijah was accidentally omitted from the Windows builds. The promised MonetDB/SQL release will remain a promise. The current head of the Stable branch is still not yet stable, unfortunately. -- Sjoerd Mullender

1 0

Re: [Monetdb-developers] [Monetdb-checkins] MonetDB5/src/modules/mal tablet.mx, , 1.93, 1.94
by Romulo Goncalves 19 Jun '07

19 Jun '07

Martin Kersten wrote: > Update of /cvsroot/monetdb/MonetDB5/src/modules/mal > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv14370 > > Modified Files: > tablet.mx > Log Message: > There is no guarantee that the table descriptor has a BAT > assigned to the first field. So you may not perform a BATcount. After this checkin (I still have to check if I did not change the code). The crash in the mserver with the skyserver data is gone. I will run more experiments to check it. Regards, Romulo > > > Index: tablet.mx > =================================================================== > RCS file: /cvsroot/monetdb/MonetDB5/src/modules/mal/tablet.mx,v > retrieving revision 1.93 > retrieving revision 1.94 > diff -u -d -r1.93 -r1.94 > --- tablet.mx 16 Jun 2007 07:54:02 -0000 1.93 > +++ tablet.mx 19 Jun 2007 05:54:56 -0000 1.94 > @@ -1495,7 +1495,7 @@ > TABLETload_file(Tablet * as, bstream *b, stream *out) > { > int res = 0, done = 0; > - size_t i = 0; > + size_t i = 0, tuples=0; > char *sep = as->format[as->nr_attrs - 1].sep; > int seplen = as->format[as->nr_attrs - 1].seplen; > > @@ -1551,9 +1551,10 @@ > as->error=0; > GDKerror("TABLETload_file: read error " > "(after loading %d records)\n", > - BATcount(as->format[0].c)); > + tuples); > res = -1; > } > + tuples++; > break; > } > end = b->buf + b->len; > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

1 1

Re: [Monetdb-developers] [MonetDB-users] Importing large dataset (TPC-H SF100)
by Martin Kersten 18 Jun '07

18 Jun '07

Dear All, The problems reported on loading have our full attention. They are hard to reproduce. It takes a long time to reach the point were it breaks. BUT, we can reproduce (hopefully) the bug (It seems a memory overwrite). Due to a local science meeting and priority (they pay for the development of MonetDB), the concerted frontal attack on the bug will start upcoming Wednesday. Please stay with us. regards, Martin James Laken wrote: > I forgot to mention that the load script use the following formula: > COPY N RECORDS INTO table from 'path/to/importfile' ... where N is the > number of records in the inputfile. > > Regards, > Zoltan > > > > Colin Foss wrote: >> James, >> >> I've had success avoiding "swap-of-death" with COPY >> INTO by specifying the number of records on each >> statement. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

1 0

[Monetdb-developers] Importing large dataset (TPC-H SF100)
by James Laken 18 Jun '07

18 Jun '07

Hi, I am trying to import the TPC-H dataset (SF100) to the database without success. The import method is the same as in the benchmark/tpch, except the number of expected records in the load script and the load script execution (executed line-by-line in the console). The machine is an dual core AMD64 (64bit OS), with 4G ram and 8 disk raid0 for storage. The import process consume nearly all memory and nearly all the swap (please note that the number of expected records is specified after the COPY command). In fact I have to restart the server process after each table import to prevent the "swap to death" state. However it seems that the lineitem table import will fail, no matter what I do. I have tried it with different client (eg.: mjclient with -Xbatching mode), sliced lineitem.tlb file, with the same result. I have noticed that two or three hours after the issued lineitem COPY command the mserver5 process does not consume CPU any more. The attached strace shows me the following: [pid 4843] select(6, [5], NULL, NULL, {0, 500}) = 0 (Timeout) [pid 4843] select(6, [5], NULL, NULL, {0, 500}) = 0 (Timeout) [pid 4843] select(6, [5], NULL, NULL, {0, 500}) = 0 (Timeout) Any idea? Is there any way to import huge datasets (TB) in bulk mode to the database? For example the postgresql database has a feature that it can import the data without write ahead logging, nearly at disk speed. The dataset can be sliced up per column, so a direct column copy would be possible. Regards, J.

1 0

[Monetdb-developers] new release
by Sjoerd Mullender 15 Jun '07

15 Jun '07

Some problems have been found in the X.18.0 release that we've created in both SQL and XQuery. Because of that we're planning to do a minor release of just those two packages on Tuesday. In SQL, the error was a crash of the server when adding a foreign key to an existing table. In XQuery, the problem was mainly a performance issue (and missing pftijah functionality in the Windows builds). -- Sjoerd Mullender

1 0

[Monetdb-developers] Release is done
by Sjoerd Mullender 14 Jun '07

14 Jun '07

The new release (X.18.0 for various values of X, and 5.0.0) is available on Sourceforge. The CVS Stable branch is once again open for bug fixing (and *only* bug fixing). As soon as the web site has been updated, you can see the full release notes at http://monetdb.cwi.nl/projects/monetdb/Development/Releases/Version4.18/ind… Niels, can you update the web site? Martin, can you send the announcements? -- Sjoerd Mullender

2 1

[Monetdb-developers] Dealing with large tables
by Peter Boncz 14 Jun '07

14 Jun '07

Hi Augustin, The crash is a bug. You have not told us about your platform (Darwin, I presume) and whether you have compiled with 64-bits enabled. Considering the data is 'large', it may be an out-of-memory problem but then in any case it is a bug, because lack of resources should lead to the query failing rather than crashing. So, first provide us the missing platform info (is it 64 bits????), and second, recompile with debugging enabled (configure with: --enable-debug=yes --enable-optimize=no). Since this problem seems highly reproducable, then run again following my instructions: (0) start Mserver in gdb (gdb --args Mserver ...args''') (1) modify the MIL script to set the debugmask 19 (2) also insert some extra info() calls (3) when it crashes, type in the gdb console: bt (4) send us all the output Your modified MIL script should resemble: debugmask(19); var var_0 := [=](col1,col2).select(true).mirror(); var_0.info().print(); col1.info().print(); col1.delete(var_0); commit(); col2.info().print(); col2.delete(var_0); commit(); col3.info().print(); col3.delete(var_0); commit(); thanks in advance, Peter ------------------------------ Message: 6 Date: Thu, 14 Jun 2007 11:59:24 -0400 From: Agustin Schapira <schapira(a)cs.umass.edu> Subject: [Monetdb-developers] Dealing with large tables To: monetdb-developers(a)lists.sourceforge.net Message-ID: <9FAD9406-07DC-454A-9AB0-A9C9DFD05153(a)cs.umass.edu> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Dear Monet developers, Once again, here I am, writing for help ;-) We are working with a 'large' table, with 250 million rows, 11 columns. This takes up a lot of memory space, as you can imagine: a single column of 250 million 'ints' requires almost a gigabyte, and there's several of those; in fact, I think that the entire table takes up a bit over 6GB in memory (some columns are strings, and can be compressed). The problem is that we have a machine with 8GB, and every time we try to do something with that table, Monet (4.16.2) crashes. For example, I've been trying to delete rows from the table where two columns have the same value: var var_0 := [=](col1,col2).select(true).mirror(); col1.delete(var_0); commit(); col2.delete(var_0); commit(); col3.delete(var_0); commit(); ... After deleting from a few of those columns, Monet dies: !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault), please restart. !ERROR: (One potential cause could be that your disk might be full...) Do you know if there a fix for this? In principle, Monet should be able to unload the BATs that it has already deleted from, to release memory. Also, would doing a semijoin() instead of a delete() help? (I though that a semijoin would need to create a new BAT, thereby using even more memory!). And, in general: is Monet 5 capable of accessing BATs that don't entirely fit in memory? Would the new version fix this problem? Maybe it's time for us to adapt it... Thanks again for all your help, and regards from Amherst, -- Agustin ------------------------------ Message: 7 Date: Thu, 14 Jun 2007 18:29:05 +0200 From: Martin Kersten <Martin.Kersten(a)cwi.nl> Subject: Re: [Monetdb-developers] Dealing with large tables To: Agustin Schapira <schapira(a)cs.umass.edu> Cc: monetdb-developers(a)lists.sourceforge.net Message-ID: <46716CD1.8080104(a)cwi.nl> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Agustin Schapira wrote: > Dear Monet developers, > > Once again, here I am, writing for help ;-) > > We are working with a 'large' table, with 250 million rows, 11 > columns. This takes up a lot of memory space, as you can imagine: a > single column of 250 million 'ints' requires almost a gigabyte, and I would say 1GB is a minimum. What if you are on a 64bit machine? And also, if this table requires a hash index, then add another 1GB/table. > there's several of those; in fact, I think that the entire table > takes up a bit over 6GB in memory (some columns are strings, and can > be compressed). The problem is that we have a machine with 8GB, and > every time we try to do something with that table, Monet (4.16.2) > crashes. For example, I've been trying to delete rows from the table > where two columns have the same value: > > var var_0 := [=](col1,col2).select(true).mirror(); Hard to deduce what's happening. > col1.delete(var_0); > commit(); > col2.delete(var_0); > commit(); > col3.delete(var_0); > commit(); I would first try to make sure if the commit is not sitting in the way. Do you need to commit here ? > ... > > After deleting from a few of those columns, Monet dies: > > !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault), > please restart. > !ERROR: (One potential cause could be that your disk might be full...) > > > Do you know if there a fix for this? In principle, Monet should be > able to unload the BATs that it has already deleted from, to release > memory. Also, would doing a semijoin() instead of a delete() help? (I > though that a semijoin would need to create a new BAT, thereby using > even more memory!). And, in general: is Monet 5 capable of accessing > BATs that don't entirely fit in memory? Would the new version fix > this problem? Maybe it's time for us to adapt it... > > Thanks again for all your help, and regards from Amherst, > > -- Agustin > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Monetdb-developers mailing list > Monetdb-developers(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-developers ------------------------------ ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ ------------------------------ _______________________________________________ Monetdb-developers mailing list Monetdb-developers(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers End of Monetdb-developers Digest, Vol 13, Issue 6 *************************************************

1 0

[Monetdb-developers] Dealing with large tables
by Agustin Schapira 14 Jun '07

14 Jun '07

Dear Monet developers, Once again, here I am, writing for help ;-) We are working with a 'large' table, with 250 million rows, 11 columns. This takes up a lot of memory space, as you can imagine: a single column of 250 million 'ints' requires almost a gigabyte, and there's several of those; in fact, I think that the entire table takes up a bit over 6GB in memory (some columns are strings, and can be compressed). The problem is that we have a machine with 8GB, and every time we try to do something with that table, Monet (4.16.2) crashes. For example, I've been trying to delete rows from the table where two columns have the same value: var var_0 := [=](col1,col2).select(true).mirror(); col1.delete(var_0); commit(); col2.delete(var_0); commit(); col3.delete(var_0); commit(); ... After deleting from a few of those columns, Monet dies: !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault), please restart. !ERROR: (One potential cause could be that your disk might be full...) Do you know if there a fix for this? In principle, Monet should be able to unload the BATs that it has already deleted from, to release memory. Also, would doing a semijoin() instead of a delete() help? (I though that a semijoin would need to create a new BAT, thereby using even more memory!). And, in general: is Monet 5 capable of accessing BATs that don't entirely fit in memory? Would the new version fix this problem? Maybe it's time for us to adapt it... Thanks again for all your help, and regards from Amherst, -- Agustin

3 3

Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder MonetDB-XQuery.spec.in, XQuery_0-18, 1.21, 1.21.2.1
by Stefan Manegold 13 Jun '07

13 Jun '07

On Mon, Jun 11, 2007 at 01:37:14PM +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/pathfinder > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv2101 > > Modified Files: > Tag: XQuery_0-18 > MonetDB-XQuery.spec.in > Log Message: > There is no need to specify that the version of MonetDB4-server needs > to be below 5.0 because it is by definition. > Also, make the minimum requirement 4.18. good point --- I apperantly overlooked that one when updateing the vertoo configuration --- should we also handle this via vertoo? Stefan > Index: MonetDB-XQuery.spec.in > =================================================================== > RCS file: /cvsroot/monetdb/pathfinder/MonetDB-XQuery.spec.in,v > retrieving revision 1.21 > retrieving revision 1.21.2.1 > diff -u -d -r1.21 -r1.21.2.1 > --- MonetDB-XQuery.spec.in 11 Feb 2007 22:04:35 -0000 1.21 > +++ MonetDB-XQuery.spec.in 11 Jun 2007 13:37:11 -0000 1.21.2.1 > @@ -39,7 +39,7 @@ > %package devel > Summary: MonetDB XQuery development package > Group: System > -Requires: MonetDB4-server>=4.15<5.0 > +Requires: MonetDB4-server >= 4.18 > > %description devel > Add the MonetDB devel description here > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Monetdb-pf-checkins mailing list > Monetdb-pf-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 3

Re: [Monetdb-developers] [Monetdb-checkins] buildtools/conf MonetDB.m4, MonetDB_1-18, 1.16.2.6, 1.16.2.7
by Stefan Manegold 13 Jun '07

13 Jun '07

On Fri, Jun 08, 2007 at 02:03:41PM +0000, Sjoerd Mullender wrote: > Update of /cvsroot/monetdb/buildtools/conf > In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv19133 > > Modified Files: > Tag: MonetDB_1-18 > MonetDB.m4 > Log Message: > Ported to Fedora (no Core) 7. Thanks, Sjoerd! ... did you also update TestTools/.Mconfig.rc & TestTools/MdoServer accordingly? (sorry, due bad network and/but good and intensive networking, I haven't followed all check in message...) It's not curcial, until we have an F67 machine in our testing pool, but we shuld not "forget" it ... Greetings from a hot, busy, foggy Beijing! Stefan > > Index: MonetDB.m4 > =================================================================== > RCS file: /cvsroot/monetdb/buildtools/conf/MonetDB.m4,v > retrieving revision 1.16.2.6 > retrieving revision 1.16.2.7 > diff -u -d -r1.16.2.6 -r1.16.2.7 > --- MonetDB.m4 8 Jun 2007 08:19:19 -0000 1.16.2.6 > +++ MonetDB.m4 8 Jun 2007 14:03:38 -0000 1.16.2.7 > @@ -412,7 +412,7 @@ > dnl Please keep this aligned / in sync with TestTools/.Mconfig.rc & TestTools/MdoServer ! > if test -s /etc/fedora-release ; then > LINUX_DIST="`cat /etc/fedora-release | head -n1 \ > - | sed 's|^.*$Fedora$ Core.* release $[[0-9]][[^ \n]]*$$ .*$*$|\1:\2|'`" > + | sed 's|^.*$Fedora$.* release $[[0-9]][[^ \n]]*$$ .*$*$|\1:\2|'`" > elif test -s /etc/centos-release ; then > LINUX_DIST="`cat /etc/centos-release | head -n1 \ > | sed 's|^$CentOS$ release $[[0-9]][[^ \n]]*$$ .*$*$|\1:\2|'`" > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Monetdb-checkins mailing list > Monetdb-checkins(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-checkins > > -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0