- users-list - monetdb.org

Perf. benchmark for monetdb
by nitin sharma 10 Jan '13

10 Jan '13

Hi there, I am evaluating monetdb and was wondering if there are performance bench mark numbers like the one published in C-store paper: http://people.csail.mit.edu/tdanford/6830papers/stonebraker-cstore.pdf I am looking for perf. number for 100 millions rows in fact table (500 GB) joined with couple of dimension tables. Does moentdb supports horizontal partitioning of data and joins across machine (distributed joins) ? Thanks!Nitin _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 3

Weird concurrency issue with JDBC getConnection
by Percy Wegmann 10 Jan '13

10 Jan '13

Dear Monet Team, I've been using the JDBC driver and have recently noticed that under load, calls to DriverManager.getConnection() will sometimes appear to hang. I was able to resolve this problem by serializing access to getConnection() using a synchronized block. I haven't had a chance to dig through the JDBC driver code to see what would account for this, but I figured I'd make you aware just in case someone else runs into this. Cheers, -- Percy Wegmann +1 512 637 8500 ext 148 _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 4

What is normally done to recover from 'crash'ed databases.
by Brandon Jackson 08 Jan '13

08 Jan '13

Hello, I have a toy database on my laptop that I've been using while testing development of a Pentaho Data Integration plugin for bulk loading. One day I shutdown my machine without stopping monetdb. When I turned on the computer and started monetdbd on the database everything seems to work fine. 'monetdb status' outputs the following: brandon-jacksons-macbook-pro:bin bjackson$ ./monetdb status name state uptime health demo crashed 0%, 0s crash 2013-01-05 00:29:20 brandon-jacksons-macbook-pro:bin bjackson$ ./monetdb start demo starting database 'demo'... done brandon-jacksons-macbook-pro:bin bjackson$ ./monetdb status name state uptime health demo running 5s 50%, 0s crash 2013-01-05 00:29:20 brandon-jacksons-macbook-pro:bin bjackson$ I cannot really tell if there is any real damage. What do people normally do to check the integrity of their databases and repair damage if it exists? Does a set of actions restore the health of the database? If I missed some documentation please feel free to refer me to some links. Thanks for the information. Brandon _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 3

Fwd: Re: c3p0 ConnectionTester socket read stuck after SQLException
by Hannes Mühleisen 08 Jan '13

08 Jan '13

Forwarding this to the list, please keep the discussion there... -------- Original Message -------- Subject: Re: c3p0 ConnectionTester socket read stuck after SQLException Date: Tue, 8 Jan 2013 08:17:35 -0800 (PST) From: Tapomay Dey <tapomay(a)yahoo.com> Reply-To: Tapomay Dey <tapomay(a)yahoo.com> To: Hannes.Muehleisen(a)cwi.nl <Hannes.Muehleisen(a)cwi.nl> My operations are as follows: 1. Create connections with auto-commit=false 2. Read queries from an external module. 3. Execute queries on the connection acquired in step 1 4. Update external module status as per result of queries. 5. If all is well, commit the connection. 6. If commit fails(optimistic locking is the culprit here), revert changes to the external module. All of these steps are implemented in a command chain pattern code. Thus I need to be able to pass around the connection. Thanks and Regards, Tapomay. ------------------------------------------------------------------------ *From:* Hannes Mühleisen <Hannes.Muehleisen(a)cwi.nl> *To:* Communication channel for MonetDB users <users-list(a)monetdb.org> *Sent:* Thursday, January 3, 2013 4:19 PM *Subject:* Re: c3p0 ConnectionTester socket read stuck after SQLException Hello Tapomay, after looking at your code, I was wondering why you chose the double management of connections in your Main/JdbcMonetdbTestClient classes. C3P0 provides a way to define "connection customizers" (com.mchange.v2.c3p0.ComboPooledDataSource.setConnectionCustomizerClassName(String)), which you can use to disable autocommit on new connections. This should simplify your example considerably. Nevertheless, I will try to provoke the exceptions you have mentioned and see how the JDBC driver handles them together with C3P0. Best, Hannes On 20/12/12 15:41, Tapomay Dey wrote: > As the source code has many unrelated dependencies I tried to extract > the relevant code. > Please find attached an eclipse project for the same. > However I haven't been able to reproduce the issue in this test project. > It should give you a fair idea of the sequence of calls made. Plz note > that a single insert query used in the live project is much much larger > (10K approx.) than the one in the test. > I reconfirmed that the issue is still there in the larger project. > > PS: I had sent this code long back and it got moderated just now(due to > some 40KB limit). Thus I have removed the dependencies > monetdb-jdbc-2.6.jar and c3p0-0.9.1.2.jar from lib. Plz add them to lib > folder and everything should work(after u set the correct DB names in > the code).(Anyways this code doesn't reproduce the issue I am facing so > I guess just a manual overview should suffice) > > Regards, > Tapomay. > > ------------------------------------------------------------------------ > *From:* Hannes Mühleisen <Hannes.Muehleisen(a)cwi.nl <mailto:Hannes.Muehleisen@cwi.nl>> > *To:* Communication channel for MonetDB users <users-list(a)monetdb.org <mailto:users-list@monetdb.org>> > *Sent:* Thursday, December 20, 2012 2:17 PM > *Subject:* Re: c3p0 ConnectionTester socket read stuck after SQLException > > Hello Tapomay, > > I will have a look later today, could you please also provide the source > code that you have used to provoke the issue? > > Thanks, > > Hannes > > On 20/12/12 08:26, Tapomay Dey wrote: > > Please find the thread dump below. > > Running MonetDB Database Server v1.6 (Oct2012-SP1). Compiled > > from MonetDB-11.13.5 src distribution. > > C3P0 version: 0.9.1.2. Also tried with 0.9.2. > > Occurs only after SQLException sometimes: INSERT INTO xyz : table xyz > > doesn't exist. > > Otherwise it works great. > > Also tried setting c3p0's preferredQuery for ConnectionTester to "select > > 1;". Problem persists. > > > > Name: pool-15-thread-2 > > State: RUNNABLE > > Total blocked: 0 Total waited: 1 > > > > Stack trace: > > java.net.SocketInputStream.socketRead0(Native Method) > > java.net.SocketInputStream.read(SocketInputStream.java:150) > > java.net.SocketInputStream.read(SocketInputStream.java:121) > > java.io.BufferedInputStream.fill(BufferedInputStream.java:235) > > java.io.BufferedInputStream.read1(BufferedInputStream.java:275) > > java.io.BufferedInputStream.read(BufferedInputStream.java:334) > > - locked java.io.BufferedInputStream@5cec5796 > > > nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream._read(MapiSocket.java:846) > > > nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(MapiSocket.java:895) > > > nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:965) > > sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) > > sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) > > sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) > > - locked java.io.InputStreamReader@206a6883 > > java.io.InputStreamReader.read(InputStreamReader.java:184) > > java.io.BufferedReader.fill(BufferedReader.java:154) > > java.io.BufferedReader.readLine(BufferedReader.java:317) > > - locked java.io.InputStreamReader@206a6883 > > java.io.BufferedReader.readLine(BufferedReader.java:382) > > > nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(BufferedMCLReader.java:119) > > > nl.cwi.monetdb.mcl.io.BufferedMCLReader.waitForPrompt(BufferedMCLReader.java:196) > > - locked nl.cwi.monetdb.mcl.io.BufferedMCLReader@29f38115 > > > nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2296) > > - locked nl.cwi.monetdb.mcl.net.MapiSocket@319f9f37 > > > nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2274) > > > nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508) > > nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349) > > nl.cwi.monetdb.jdbc.MonetStatement.executeQuery(MonetStatement.java:525) > > > com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73) > > > com.mchange.v2.c3p0.impl.DefaultConnectionTester.statusOnException(DefaultConnectionTester.java:143) > > > com.mchange.v2.c3p0.AbstractConnectionTester.statusOnException(AbstractConnectionTester.java:82) > > > com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:437) > > - locked com.mchange.v2.c3p0.impl.NewPooledConnection@3340302c > > > com.mchange.v2.c3p0.impl.NewProxyStatement.executeUpdate(NewProxyStatement.java:78) > > > com.sokrati.moneta.mclient.JdbcMonetdbClient.executeUpdate(JdbcMonetdbClient.java:154) > > > com.sokrati.moneta.multiplexer.MonetdbClientCallable.call(MonetdbClientCallable.java:46) > > > com.sokrati.moneta.multiplexer.MonetdbClientCallable.call(MonetdbClientCallable.java:1) > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > > java.util.concurrent.FutureTask.run(FutureTask.java:166) > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > java.lang.Thread.run(Thread.java:722) > > > > Regards, > > Tapomay. > > > > > > _______________________________________________ > > users-list mailing list > > users-list(a)monetdb.org <mailto:users-list@monetdb.org> <mailto:users-list@monetdb.org <mailto:users-list@monetdb.org>> > > http://mail.monetdb.org/mailman/listinfo/users-list > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org <mailto:users-list@monetdb.org> <mailto:users-list@monetdb.org <mailto:users-list@monetdb.org>> > http://mail.monetdb.org/mailman/listinfo/users-list > > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org <mailto:users-list@monetdb.org> > http://mail.monetdb.org/mailman/listinfo/users-list > _______________________________________________ users-list mailing list users-list(a)monetdb.org <mailto:users-list@monetdb.org> http://mail.monetdb.org/mailman/listinfo/users-list _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

the dbname option disappeared
by Stefan de Konink 04 Jan '13

04 Jan '13

I have justed checked out the Feb2013 release and noticed that the dbname option is gone, is there a substitute? Stefan _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

5 8

c3p0 ConnectionTester socket read stuck after SQLException
by Tapomay Dey 03 Jan '13

03 Jan '13

Please find the thread dump below. Running MonetDB Database Server v1.6 (Oct2012-SP1). Compiled from MonetDB-11.13.5 src distribution. C3P0 version: 0.9.1.2. Also tried with 0.9.2. Occurs only after SQLException sometimes: INSERT INTO xyz : table xyz doesn't exist. Otherwise it works great. Also tried setting c3p0's preferredQuery for ConnectionTester to "select 1;". Problem persists. Name: pool-15-thread-2 State: RUNNABLE Total blocked: 0 Total waited: 1 Stack trace: java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:150) java.net.SocketInputStream.read(SocketInputStream.java:121) java.io.BufferedInputStream.fill(BufferedInputStream.java:235) java.io.BufferedInputStream.read1(BufferedInputStream.java:275) java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked java.io.BufferedInputStream@5cec5796 nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream._read(MapiSocket.java:846) nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(MapiSocket.java:895) nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:965) sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) - locked java.io.InputStreamReader@206a6883 java.io.InputStreamReader.read(InputStreamReader.java:184) java.io.BufferedReader.fill(BufferedReader.java:154) java.io.BufferedReader.readLine(BufferedReader.java:317) - locked java.io.InputStreamReader@206a6883 java.io.BufferedReader.readLine(BufferedReader.java:382) nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(BufferedMCLReader.java:119) nl.cwi.monetdb.mcl.io.BufferedMCLReader.waitForPrompt(BufferedMCLReader.java:196) - locked nl.cwi.monetdb.mcl.io.BufferedMCLReader@29f38115 nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.executeQuery(MonetConnection.java:2296) - locked nl.cwi.monetdb.mcl.net.MapiSocket@319f9f37 nl.cwi.monetdb.jdbc.MonetConnection$ResponseList.processQuery(MonetConnection.java:2274) nl.cwi.monetdb.jdbc.MonetStatement.internalExecute(MonetStatement.java:508) nl.cwi.monetdb.jdbc.MonetStatement.execute(MonetStatement.java:349) nl.cwi.monetdb.jdbc.MonetStatement.executeQuery(MonetStatement.java:525) com.mchange.v2.c3p0.impl.DefaultConnectionTester.activeCheckConnection(DefaultConnectionTester.java:73) com.mchange.v2.c3p0.impl.DefaultConnectionTester.statusOnException(DefaultConnectionTester.java:143) com.mchange.v2.c3p0.AbstractConnectionTester.statusOnException(AbstractConnectionTester.java:82) com.mchange.v2.c3p0.impl.NewPooledConnection.handleThrowable(NewPooledConnection.java:437) - locked com.mchange.v2.c3p0.impl.NewPooledConnection@3340302c com.mchange.v2.c3p0.impl.NewProxyStatement.executeUpdate(NewProxyStatement.java:78) com.sokrati.moneta.mclient.JdbcMonetdbClient.executeUpdate(JdbcMonetdbClient.java:154) com.sokrati.moneta.multiplexer.MonetdbClientCallable.call(MonetdbClientCallable.java:46) com.sokrati.moneta.multiplexer.MonetdbClientCallable.call(MonetdbClientCallable.java:1) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) java.util.concurrent.FutureTask.run(FutureTask.java:166) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) java.lang.Thread.run(Thread.java:722) Regards, Tapomay. _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 6

Re: users-list Digest, Vol 5, Issue 1
by swapnil joshi 03 Jan '13

03 Jan '13

Hi nitin, # Serving database 'demo', using 12 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 23.998 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information !IOException:mal_mapi.listen: operation failed: bind to stream socket port 50000 failed: solution 1 = It's an just port conflict error the port had assign to monetdb is already used by any other application please change the port of monetdb or find your another application which using same port and sopt it's service, then it will work. Solution 2 = If you are an student and if you are using your own system / personal computer then just shut down the system and restart it and first start monetdb server. If any only if you are studying monetdb on your personal computer . (solution 2 is foolish solution :) ) On Thu, Jan 3, 2013 at 4:30 PM, <users-list-request(a)monetdb.org> wrote: > nitin -- Regards, Swapnil K. Joshi _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

Error while starting monetdb on windows server 2008 R2
by nitin sharma 03 Jan '13

03 Jan '13

Hi there, I am getting following error while running monetdb server >>>># MonetDB 5 server v11.13.7 "Oct2012-SP2" # Serving database 'demo', using 12 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 23.998 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information !IOException:mal_mapi.listen:operation failed: bind to stream socket port 50000 failed: No error > Any pointer what could be the issue? thx,Nitin _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1

failed to fork mserver: database 'lwg' started up, but failed to open up a communication channel
by Przemyslaw Wroblewski 28 Dec '12

28 Dec '12

hello, I'm trying to evaluate monetdb but I've failed at the beginning of the tutorial: http://www.monetdb.org/Documentation/monetdbd When I'm trying ot start up database I've got following: $ monetdb start lwg starting database 'lwg'... FAILED start: starting 'lwg' failed: database 'lwg' started up, but failed to open up a communication channel and in merovingian.log: 2012-12-27 16:43:26 MSG merovingian[29552]: startup of database under maintenance 'lwg' forced 2012-12-27 16:43:26 MSG merovingian[29552]: starting database 'lwg', up min/avg/max: 1s/1s/1s, crash average: 0.00 0.00 0.00 (1-1=0) 2012-12-27 16:43:27 MSG merovingian[29552]: sending process 9489 (database 'lwg') the TERM signal 2012-12-27 16:43:27 MSG lwg[9489]: arguments: /usr/bin/mserver5 --set gdk_dbfarm=/var/lib/monetdb --dbname=lwg --set merovingian_uri=mapi:monetdb://przemtest1:50000/lwg --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/lib/monetdb/lwg/.mapi.sock --set monet_vault_key=/var/lib/monetdb/lwg/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2012-12-27 16:43:27 MSG lwg[9489]: # MonetDB 5 server v11.13.7 "Oct2012-SP2" 2012-12-27 16:43:27 MSG lwg[9489]: # Serving database 'lwg', using 8 threads 2012-12-27 16:43:27 MSG lwg[9489]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked 2012-12-27 16:43:27 MSG lwg[9489]: # Found 23.591 GiB available main-memory. 2012-12-27 16:43:27 MSG lwg[9489]: # Copyright (c) 1993-July 2008 CWI. 2012-12-27 16:43:27 MSG lwg[9489]: # Copyright (c) August 2008-2012 MonetDB B.V., all rights reserved 2012-12-27 16:43:27 MSG lwg[9489]: # Visit http://www.monetdb.org/ for further information 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:batcalc.rank_grp[3]:object code for command sql.rank_grp missing 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:batcalc.rank_grp[1]:object code for command sql.rank missing 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:batcalc.dense_rank_grp[3]:object code for command sql.dense_rank_grp missing 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:batcalc.dense_rank_grp[1]:object code for command sql.dense_rank missing 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:aggr.median[7]:object code for command sql.middle missing 2012-12-27 16:43:27 MSG lwg[9489]: #WARNING: LoaderException:loadLibrary:Loading error could not locate library lsst (from within file 'lsst') 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:user.main[1240]:object code for command sql.prelude missing 2012-12-27 16:43:27 MSG lwg[9489]: !TypeException:user.main[1240]:'sql.prelude' undefined in: _1240:void := sql.prelude() 2012-12-27 16:43:27 MSG merovingian[29552]: database 'lwg' (9489) has exited with exit status 0 2012-12-27 16:43:27 MSG merovingian[29552]: database 'lwg' has shut down 2012-12-27 16:43:27 ERR control[29552]: (local): failed to fork mserver: database 'lwg' started up, but failed to open up a communication channel I'm on 64bit debian 6.0.4 in vserver container. It looks like exactly as http://bugs.monetdb.org/show_bug.cgi?id=3134 but I've latest version which should this bug already fixed. best regards Przemek _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1

Fwd: Enhance Performance
by Arjun Harish 27 Dec '12

27 Dec '12

I am bench-marking monetDB against some other databases. I have a system with intel i-7 processor, 32GB Ram and 2 TB Harddisk on a Ubuntu 12.10 system. I have a data work load of 700 GB in monetdb now. How can i enhance the performance and speed up the querying process. Any Suggestions would we welcome and please reply in detail. I am a newbie to Monetdb. Arjun Data Scientist -- Arjun Harish Nadh Data Scientist Flutura Business Solutions Pvt. Ltd. +919972736428 arjun.harish(a)fluturasolutions.com _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1