February 2008 - users-list

[MonetDB-users] COPY TO corrupting data
by mobigital1 25 Feb '08

25 Feb '08

I am trying to load about 500M rows into a table with about 160 columns: column types: 7 columns of type "int" 2 columns of type "varchar(20)" 6 columns of type "varchar(10)" 1 columns of type "varchar(15)" 144 columns of type "real" on a system with 16 GB RAM , 64 bit windows server 2003 SP2. it takes some 20 hours to load the 100M rows, but when I reach that boundary, the server seems to crash and database becomes corrupt so that about 95M rows have 0 in numeric columns, blanks in varchars. The remaining 5M rows have real data. does anyone have any suggestions on what can be done to make it possible to load this data? -- View this message in context: http://www.nabble.com/COPY-TO-corrupting-data-tp15634267p15634267.html Sent from the monetdb-users mailing list archive at Nabble.com.

4 7

[MonetDB-users] MonetDB mserver5 crash
by Michael Schmidt 21 Feb '08

21 Feb '08

Hi, I'm trying to run a benchmark using the latest February stable (on an Ubuntu 7.04 - Feisty Fawn), and want to cleanup the database entirely between two single runs. Basically, my benchmark script is as follows: //////////////////// START CODE // kill all processes and remove databases) // (I know this is not the intended way, but using // > monetdb stop test // > monetdb destroy -f test // while not restarting merovingian basically results in // in the same error) > killall -9 merovingian > killall -9 mserver5 > killall -9 mclient > rm -Rf /usr/local/src/MonetDB/var/MonetDB5/dbfarm/* // start the merovingian server > merovingian > sleep 2 // wait until ports 50000 and 50001 are free, // as they might still be blocked (I inserted this piece // of code because I thought busy port allocation might // be the cause of failure) > [code works fine (ommited)] // create and start the database > monetdb create test; > sleep 2; > monetdb start test; > sleep 2; // load the document into the database (this part tends to fail) > mclient -lsql -dtest < document.sql //////////////////// END CODE The strange thing is that every second run fails due to a "Connection terminated" error when loading the document, i.e. mserver5 crashes away. Did anyone experience similar problems? Any hints would be appreciated. Thanks in advance, Michael

2 9

[MonetDB-users] Storing UUID, column of type Binary?
by McKennirey.Matthew 21 Feb '08

21 Feb '08

2 3

[MonetDB-users] Problem with View definition
by Michael Schmidt 21 Feb '08

21 Feb '08

Hi, I am running the latest (February) release of MonetDB, and I have a question concerning MonetDB view definitions. I have defined a schema consisting of simple triple-column tables, defined as follows CREATE TABLE bench_booktitle (subject INTEGER,object INTEGER,FOREIGN KEY (subject) REFERENCES Dictionary(ID),FOREIGN KEY (object) REFERENCES Dictionary(ID)); CREATE TABLE bench_cdrom (subject INTEGER,object INTEGER,FOREIGN KEY (subject) REFERENCES Dictionary(ID),FOREIGN KEY (object) REFERENCES Dictionary(ID)); ... using a simple Dictionary lookup table CREATE TABLE Dictionary (ID INTEGER,val VARCHAR(20000),PRIMARY KEY (ID)); Now, I'm trying to create a view ranging over all these tables. The problem with the view definition is that it takes a very long time and consumes large main memory resources (I aborted after 10 minutes, where the mserver5 process uses more than 1.2GB of main memory). It might be of interest that view creation takes place even before loading any data in the tables, i.e. all tables are empty. Maybe someone has an idea what's going on when creating the View. Why does it require so much memory and why does it take so long? Any idea how to fix this problem? Here is the view definition (which is rather straightforward): CREATE VIEW Triples AS ( SELECT subject,'bench:booktitle' AS predicate,object FROM bench_booktitle ) UNION ( SELECT subject,'bench:cdrom' AS predicate,object FROM bench_cdrom ) UNION ( SELECT subject,'swrc:pages' AS predicate,object FROM swrc_pages ) UNION ( SELECT subject,'foaf:homepage' AS predicate,object FROM foaf_homepage ) UNION ( SELECT subject,'swrc:editor' AS predicate,object FROM swrc_editor ) UNION ( SELECT subject,'swrc:number' AS predicate,object FROM swrc_number ) UNION ( SELECT subject,'dc:creator' AS predicate,object FROM dc_creator ) UNION ( SELECT subject,'rdfs:seeAlso' AS predicate,object FROM rdfs_seeAlso ) UNION ( SELECT subject,'dcterms:partOf' AS predicate,object FROM dcterms_partOf ) UNION ( SELECT subject,'dcterms:references' AS predicate,object FROM dcterms_references ) UNION ( SELECT subject,'dcterms:issued' AS predicate,object FROM dcterms_issued ) UNION ( SELECT subject,'swrc:volume' AS predicate,object FROM swrc_volume ) UNION ( SELECT subject,'dc:publisher' AS predicate,object FROM dc_publisher ) UNION ( SELECT subject,'swrc:note' AS predicate,object FROM swrc_note ) UNION ( SELECT subject,'swrc:chapter' AS predicate,object FROM swrc_chapter ) UNION ( SELECT subject,'swrc:address' AS predicate,object FROM swrc_address ) UNION ( SELECT subject,'swrc:series' AS predicate,object FROM swrc_series ) UNION ( SELECT subject,'swrc:month' AS predicate,object FROM swrc_month ) UNION ( SELECT subject,'foaf:name' AS predicate,object FROM foaf_name ) UNION ( SELECT subject,'rdf:type' AS predicate,object FROM rdf_type ) UNION ( SELECT subject,'dc:title' AS predicate,object FROM dc_title ) UNION ( SELECT subject,'swrc:journal' AS predicate,object FROM swrc_journal ) UNION ( SELECT subject,'bench:abstract' AS predicate,object FROM bench_abstract ) UNION ( SELECT subject,'swrc:isbn' AS predicate,object FROM swrc_isbn ); Kind regards, Michael Schmidt

2 4

Re: [MonetDB-users] New MonetDB Release
by Rahul Chopra 20 Feb '08

20 Feb '08

> > On Tue, Feb 19, 2008 at 11:35:14PM +0800, Rahul Chopra wrote: > > Congratulations on the release. > > > > I ran a TPC-H test on it. The majority of queries now seem slower > > Example: Query 21: 119 sec (was 18 sec), while others are closer, > > example: Query 13: 21 sec (was 18 sec). > > Which scale factor of TPC-H did you run? > Factor 10. The system has 24 GB of RAM, running SuSe 10 64-bit. The test system has been taken away from me, and the drives rebuilt, so I can't further help at the moment, but wanted to give you the heads up in case you folks did not have time to do much performance testing before this release. If you did, then there was probably just an issue in my environment for some reason. Thanks. > Niels > > > > Did you verify performance for this release? It could be > > something I misconfigured for the new release, so take this with > > a grain of salt. Thanks. > > > > > > > ----- Original Message ----- > > > From: "Stefan Manegold" <Stefan.Manegold(a)cwi.nl> > > > To: monetdb-announce(a)lists.sourceforge.net > > > Subject: [MonetDB-users] New MonetDB Release > > > Date: Tue, 19 Feb 2008 13:51:24 +0100 > > > > > > The MonetDB team at CWI is pleased to announce the new > > "Feb2008" release of > > > the MonetDB suite of programs. > > > > This release marks the roll forward of the binary storage > > layout to a single > > > column storage layout. This involved a major overhaul of the > > MonetDB Server > > > code. > > > > IMPORTANT NOTE: > > > The database format of this release is not compatible with that of any > > > previous release. Before installing this release, make a dump of your > > > database, and do a restore afterwards. > > > > Further changes include: > > > > SQL: The functional extensions to SQL are limited. The mostly concern > > > minor bugs and some of the optimizers have been improved to better exploit > > > re-occurring joins. The core cracker code has been cleaned up for better > > > inspection and sharing in the community. > > > > XQuery: Next to the original Pathfinder compiler (nick-named > > > "milprint_summer" or simply "MPS"), the development of the new improved and > > > enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. > > > While the original MPS version remains the default in this release of > > > MonetDB/XQuery, a preview of the new ALG version is now available without > > > recompilation. > > > > JDBC & XRPC: The Java code (JDBC driver & XRPC wrapper) has > > been moved from > > > the clients and pathfinder source trees to a dedicated java source tree in > > > CVS, which can simply be compiled using "ant". > > > > Apart from this, many bugs have been fixed, and many speed > > improvements have > > > been made. See > > > http://monetdb.cwi.nl/Development/Releases/Version4.22/ > > > for more details. > > > > The new release consists of the following packages: > > > > MonetDB Common 1.22.0 > > > MonetDB Clients 1.22.0 > > > MonetDB4 Server 4.22.0 > > > MonetDB XQuery 0.22.0 > > > MonetDB5 Server 5.4.0 > > > MonetDB SQL 2.22.0 > > > monetdb-1.7-jdbc.jar > > > > There is of course also a new "Super Source Ball", a tarball > > containing the > > > combined sources. > > > > All this is available on our SourceForge project page. See > > > https://sourceforge.net/project/showfiles.php?group_id=56967 for the > > > downloadable files. > > > > -- > > > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > ------------------------------------------------------------------------- > > > This SF.net email is sponsored by: Microsoft > > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > > _______________________________________________ > > > MonetDB-users mailing list > > > MonetDB-users(a)lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > > > > > > > > > > = > > Fishing Vacation Deals at Hyatt? > > Book Your Hyatt Fishing Vacation Now. Best Rates Online Guaranteed. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=a0e62a460a6115c26a11780… > > > > > > -- Powered by Outblaze > > > > ------------------------------------------------------------------------- > > This SF.net email is sponsored by: Microsoft > > Defy all challenges. Microsoft(R) Visual Studio 2008. > > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > > _______________________________________________ > > MonetDB-users mailing list > > MonetDB-users(a)lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/monetdb-users > > -- > > Niels Nes, Centre for Mathematics and Computer Science (CWI) > Kruislaan 413, 1098 SJ Amsterdam, The Netherlands > room C0.02, phone ++31 20 592-4098, fax ++31 20 592-4312 > url: http://www.cwi.nl/~niels e-mail: Niels.Nes(a)cwi.nl > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Search for products and services at: http://search.mail.com -- Powered by Outblaze

1 0

[MonetDB-users] New MonetDB Release
by Stefan Manegold 20 Feb '08

20 Feb '08

The MonetDB team at CWI is pleased to announce the new "Feb2008" release of the MonetDB suite of programs. This release marks the roll forward of the binary storage layout to a single column storage layout. This involved a major overhaul of the MonetDB Server code. IMPORTANT NOTE: The database format of this release is not compatible with that of any previous release. Before installing this release, make a dump of your database, and do a restore afterwards. Further changes include: SQL: The functional extensions to SQL are limited. The mostly concern minor bugs and some of the optimizers have been improved to better exploit re-occurring joins. The core cracker code has been cleaned up for better inspection and sharing in the community. XQuery: Next to the original Pathfinder compiler (nick-named "milprint_summer" or simply "MPS"), the development of the new improved and enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. While the original MPS version remains the default in this release of MonetDB/XQuery, a preview of the new ALG version is now available without recompilation. JDBC & XRPC: The Java code (JDBC driver & XRPC wrapper) has been moved from the clients and pathfinder source trees to a dedicated java source tree in CVS, which can simply be compiled using "ant". Apart from this, many bugs have been fixed, and many speed improvements have been made. See http://monetdb.cwi.nl/Development/Releases/Version4.22/ for more details. The new release consists of the following packages: MonetDB Common 1.22.0 MonetDB Clients 1.22.0 MonetDB4 Server 4.22.0 MonetDB XQuery 0.22.0 MonetDB5 Server 5.4.0 MonetDB SQL 2.22.0 monetdb-1.7-jdbc.jar There is of course also a new "Super Source Ball", a tarball containing the combined sources. All this is available on our SourceForge project page. See https://sourceforge.net/project/showfiles.php?group_id=56967 for the downloadable files. -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 8

Re: [MonetDB-users] Running multiple mserver5 processes, Merovingian
by McKennirey.Matthew 19 Feb '08

19 Feb '08

2 2

Re: [MonetDB-users] New MonetDB Release
by Rahul Chopra 19 Feb '08

19 Feb '08

Congratulations on the release. I ran a TPC-H test on it. The majority of queries now seem slower Example: Query 21: 119 sec (was 18 sec), while others are closer, example: Query 13: 21 sec (was 18 sec). Did you verify performance for this release? It could be something I misconfigured for the new release, so take this with a grain of salt. Thanks. > ----- Original Message ----- > From: "Stefan Manegold" <Stefan.Manegold(a)cwi.nl> > To: monetdb-announce(a)lists.sourceforge.net > Subject: [MonetDB-users] New MonetDB Release > Date: Tue, 19 Feb 2008 13:51:24 +0100 > > > > The MonetDB team at CWI is pleased to announce the new "Feb2008" release of > the MonetDB suite of programs. > > This release marks the roll forward of the binary storage layout to a single > column storage layout. This involved a major overhaul of the MonetDB Server > code. > > IMPORTANT NOTE: > The database format of this release is not compatible with that of any > previous release. Before installing this release, make a dump of your > database, and do a restore afterwards. > > Further changes include: > > SQL: The functional extensions to SQL are limited. The mostly concern > minor bugs and some of the optimizers have been improved to better exploit > re-occurring joins. The core cracker code has been cleaned up for better > inspection and sharing in the community. > > XQuery: Next to the original Pathfinder compiler (nick-named > "milprint_summer" or simply "MPS"), the development of the new improved and > enhanced Algebra-based Pathfinder compiler ("ALG") is progressing quickly. > While the original MPS version remains the default in this release of > MonetDB/XQuery, a preview of the new ALG version is now available without > recompilation. > > JDBC & XRPC: The Java code (JDBC driver & XRPC wrapper) has been moved from > the clients and pathfinder source trees to a dedicated java source tree in > CVS, which can simply be compiled using "ant". > > Apart from this, many bugs have been fixed, and many speed improvements have > been made. See > http://monetdb.cwi.nl/Development/Releases/Version4.22/ > for more details. > > The new release consists of the following packages: > > MonetDB Common 1.22.0 > MonetDB Clients 1.22.0 > MonetDB4 Server 4.22.0 > MonetDB XQuery 0.22.0 > MonetDB5 Server 5.4.0 > MonetDB SQL 2.22.0 > monetdb-1.7-jdbc.jar > > There is of course also a new "Super Source Ball", a tarball containing the > combined sources. > > All this is available on our SourceForge project page. See > https://sourceforge.net/project/showfiles.php?group_id=56967 for the > downloadable files. > > -- > | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Fishing Vacation Deals at Hyatt® Book Your Hyatt Fishing Vacation Now. Best Rates Online Guaranteed. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=a0e62a460a6115c26a11780… -- Powered by Outblaze

4 3

[MonetDB-users] compression & temp space
by Rahul Chopra 19 Feb '08

19 Feb '08

Hello, Does MonetDB use compression at all? I saw something on the site that made me think it might be used for low cardinality columns, but it was not clear, and I could not find any other instances, like for long char columns. Also, are there any plans to allow for specifying a separate device to use for temp space? (Or, can I create a link somewhere on the filesystem?) I just notice that sometimes MonetDB uses an awful lot of virtual memory, and I/O quickly becomes the bottleneck when it no longer fits in RAM. Thanks. = Profit From On Hold Messages 20 Years experience. Custom messages, free equipment, no contracts. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=49dd4ae6b19ee653ce7d12b… -- Powered by Outblaze

1 0

Re: [MonetDB-users] Analyze
by Rahul Chopra 19 Feb '08

19 Feb '08

Thank you. So, the database on its own will collect selectivity statistics so that if there are multiple atomic conditions on a column, it will choose to apply the more selective one first? Thanks > ----- Original Message ----- > From: "Martin Kersten" <Martin.Kersten(a)cwi.nl> > To: "Communication channel for MonetDB users" <monetdb-users(a)lists.sourceforge.net> > Subject: Re: [MonetDB-users] Analyze > Date: Tue, 19 Feb 2008 14:35:48 +0100 > > > Dear Rahul > > thank you for your interest in MonetDB. > There is no such thing as ANALYZE or UPDATE STATISTICS. > They are not part of the SQL 99 standard > and its implementation would not make sense. > The database performance continuous analysis to optimize > execution using actual properties of the tables involved. > > regards, Martin > Rahul Chopra wrote: > > Is there the equivalent of ANALYZE or UPDATE STATISTICS in > > MonetDB? I tried both, but could not get them to work. > > > > Thanks! > > > > = > > embedded system > > UML 2.0-Based Software Engineering Using Model-Driven Development. > > http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=b006f31a35aad8826624507… > > > > > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Microsoft > Defy all challenges. Microsoft(R) Visual Studio 2008. > http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users > = Sell Your Annuity If you are receiving payments from a structured settlement or annuity and need cash now, the Structured Settlement Alliance is here to help you get your cash now. http://a8-asy.a8ww.net/a8-ads/adftrclick?redirectid=fc012ccf36676d4eeec4d34… -- Powered by Outblaze

1 0