April 2009 - users-list

[MonetDB-users] Windows JDBC Issues
by gsd 24 Apr '09

24 Apr '09

Hello (again!), I have MonetDB running on a 32-bit Windows Vista box. When I start the server it says it's "MonetDB server v5.10.2, based on kernel v1.28.2". I'm using JDBC batch updates to populate about 100 tables with approximately 4 million records per table. I've a couple of issues: 1) My batch updates fail from time-to-time. On average this might be every 10,000 batch updates with each batch containing 100 insertions. I get an exception with the following message: "Error(s) occurred while executing the batch, see next SQLExceptions for details" If I close the connection then open another one I can continue quite happily. If I try to reuse the existing connection with a cleared batch I get the same error every time until I do close the connection. I think I may not be the only person with this issue: http://wiki.pentaho.com/display/EAI/MonetDB http://wiki.pentaho.com/display/EAI/MonetDB 2) After a very large amount of insertions (more like 100+ million records, across numerous tables) the insertion simply hangs (a commit usually takes about a second and I've left it 30+ minutes). After killing the client the server becomes impossible to connect to. Even after a reboot I get the following messages: SERVER: >!ERROR: Incompatible database version 000000, this server supports version 050000 !ERROR: Please move away c:\data\monetdbdata\MonetDB5\sql_logs\demo\sql\ and its corresponding dbfarm. !mvc_init: unable to create system tables CLIENT: !SQLException:SQLinit:Catalogue initialization failed JDBC CLIENT: Connection reset (Mserver still alive?) If, however, I delete/move the sql_logs directory then restart things then I can connect and my data is still there. The last time I did this one of the log files (log.97) was ~250mb in size. Any help, advice or comments would be much appreciated. I'm going to spend this afternoon trying out the bulk COPY INTO functionality. Thanks, Scott -- View this message in context: http://www.nabble.com/Windows-JDBC-Issues-tp23196456p23196456.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

Re: [MonetDB-users] Tweaking performance
by Garrett Smith 23 Apr '09

23 Apr '09

----- "Martin Kersten" <Martin.Kersten(a)cwi.nl> wrote: > Garrett Smith wrote: >> Does anyone have some guidance on approaches for optimizing database >> performance? >> >> I realize this is a pretty broad question :) >> >> More specifically, I'm running a simple query >> >> select max(date) from metric >> >> on a table with 21 million rows (or whatever you call them in >> Monet). >> It's taking quite a bit longer than I'd expect, given a typically >> indexed column. > Given the fact that there is no (b-tree) index, the underlying > engine has to perform a scan to find the maximum. > This entails a linear scan over 21M elements in the current > implementation. Thanks! After going back and reading the docs on MonetDB, I realized my question was pretty naive. The SQL interface triggered a Pavlovian response to think in terms of a relational database with b-tree indexes. It's pretty amazing how fast that linear scan is. Still, my application calls for something else. I took a look at FastBit, which is insanely adept at selects into huge data sets, but requires a relatively expensive index update to get that performance. As I have routine updates to deal with, that doesn't seem a good fit. We'll probably stick with MySQL+MyISAM for this particular component. Garrett

1 0

[MonetDB-users] COPY INTO with dates
by gsd 23 Apr '09

23 Apr '09

Hello, I'm looking into the COPY INTO bulk insert functionality but was wondering how I should format DATEs and TIMESTAMPs? Thanks, Scott -- View this message in context: http://www.nabble.com/COPY-INTO-with-dates-tp23194496p23194496.html Sent from the monetdb-users mailing list archive at Nabble.com.

2 1

Re: [MonetDB-users] Tweaking performance
by Martin Kersten 23 Apr '09

23 Apr '09

Garrett Smith wrote: > 123456789012345678901234567890123456789012345678901234567890123456789012 > > Does anyone have some guidance on approaches for optimizing database > performance? > > I realize this is a pretty broad question :) > > More specifically, I'm running a simple query > > select max(date) from metric > > on a table with 21 million rows (or whatever you call them in Monet). > It's taking quite a bit longer than I'd expect, given a typically > indexed column. Given the fact that there is no (b-tree) index, the underlying engine has to perform a scan to find the maximum. This entails a linear scan over 21M elements in the current implementation. > > On an "explain" I see that there's no difference between the plan with > or without the presence of an index on 'date'. Per the SQL index docs, I > see that indexes are more of a hint to Monet and they're often ignored. > > Cool. > > But what happens when I want to fix what seems to be a configuration > problem with my database/schema? > > Thanks! > > Garrett > > ------------------------------------------------------------------------------ > Stay on top of everything new and different, both inside and > around Java (TM) technology - register by April 22, and save > $200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco. > 300 plus technical and hands-on sessions. Register today. > Use priority code J9JMT32. http://p.sf.net/sfu/p > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

1 0

[MonetDB-users] Tweaking performance
by Garrett Smith 22 Apr '09

22 Apr '09

123456789012345678901234567890123456789012345678901234567890123456789012 Does anyone have some guidance on approaches for optimizing database performance? I realize this is a pretty broad question :) More specifically, I'm running a simple query select max(date) from metric on a table with 21 million rows (or whatever you call them in Monet). It's taking quite a bit longer than I'd expect, given a typically indexed column. On an "explain" I see that there's no difference between the plan with or without the presence of an index on 'date'. Per the SQL index docs, I see that indexes are more of a hint to Monet and they're often ignored. Cool. But what happens when I want to fix what seems to be a configuration problem with my database/schema? Thanks! Garrett

1 0

Re: [MonetDB-users] the x100 project
by Peter Boncz 20 Apr '09

20 Apr '09

Hi Tom, Thanks for your interest in X100. The project is very much alive in a recent CWI spin-off company called vectorwise. X100 is currently being developed into a fully functional database engine that will fly in datawarehousing and BI environments. Much more than that I cannot say about it now. More news will follow, certainly within a year. It is not yet open source (but in a future that is not off the cards), but if you need that now, you could try MonetDB. Peter > Hello, > I have been asked to evaluate MonetDB. We have interest in the x100 project, is this project still on going... is there a download for it? If > not, is there a plan to release it? > > thanks > tom

1 0

[MonetDB-users] monetdb-client ubuntu package missing binary ?
by Franck Routier 18 Apr '09

18 Apr '09

Hi, I have tried to install and test monetdb using ubuntu repository. But when it comes to using mclient, no binary seems to be installed. Indeed, looking at the package, there seems to be only docs in it. So, am I looking at the wrong place and missing some obvious point, or is the package broken ? (notice that the debian package seems to be similar) Thanks, Franck

2 4

[MonetDB-users] the x100 project
by tom edgar 16 Apr '09

16 Apr '09

Hello, I have been asked to evaluate MonetDB. We have interest in the x100 project, is this project still on going... is there a download for it? If not, is there a plan to release it? thanks tom

1 0

[MonetDB-users] loading big table
by dariuszs 16 Apr '09

16 Apr '09

Hi, I've got a table with 130 fields. When I load 130 million records into that table (about 50G) - it's not a problem, but when I try to load 160 million records (about 60G of data) into it this thing just sits for hours with no error messages. Can somebody help me? Thanks. Dariusz.

5 16

[MonetDB-users] MonetDB XQuery in Production
by Max Fedorov 14 Apr '09

14 Apr '09

The product looks very promising on paper, although not that many people are talking about it and I came across MonetDB by accident. Is anyone using MonetDB in production? Is the product stable enough? I do not feel like spending lots of time testing this XQuery server if nobody runs it in production. I loaded several million records and tested sample queries without issues for the most part, but then once the server just vanished (running latest 32bit version on WindowsXP). I restarted and everything is working again, but now I am very scared. Nothing like this ever happened when using MySQL, MS SQL Server or Oracle. Nothing ever just dies without a trace. Thank you. -Max

4 5