June 2010 - users-list

[MonetDB-users] Copy Into and .new files
by Alex Bo. 23 Jun '10

23 Jun '10

Hi, I am doing a lot of COPY INTOs, and I have noticed that sometimes while doing this MonetDB seems to hang while its process memory is slowly rising. I then noticed that while this is happening, files with a '.new' extension are added to the dbfarm folders. Now suddenly I reached a state in which no '.new' files are created at all, and the COPY INTOs work much faster because no 'hanging' occurs in which something is loaded into memory. What's the explanation for this? What can I do in order to always have COPY INTOs write directly to the final files without the '.new' intermediates? Thanks. -- View this message in context: http://old.nabble.com/Copy-Into-and-.new-files-tp28973215p28973215.html Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

Re: [MonetDB-users] Embedded server in Java with JDBC
by Luis Fernando Kauer 23 Jun '10

23 Jun '10

Hi, Thanks for the quick reply. When I say embedded I mean like in HSQLDB, which allows to embed the db server in the application instead of starting the db server in a separete process accepting connections from all clients. My requirements are that the end user should not have to deal with the database directly, like start and stop it, not even need to install/configure it separetely. But I would like to access the data through JDBC driver because the libraries I'm working with require it. I also would like a small footprint. If that's not possible, then I would like to be able to start and stop the server through the application, even if it runs in a separete process. The application allows the user to analyse and audit a large volume of data on different tables and MonetDB seems to be really fast for this. Each user project will have its own set of tables so it would be nice to be able to set the database files where the user chooses for each project. If that's not possible I'll have to stick with different schemas for each project. I'm still having some problems importing large tables (> 10 million rows divided in 10 files with 1 million rows each) when MonetDB is reporting the following error SQLException:sql:Failed to extend the BAT, perhaps disk full SQLException:importTable:failed to import table and on the console appears many times the follwing: #GDKmmap(111673344) fails, try to free up space [memory in use=56663400,virtual memory in use=1521352704] #GDKmmap(111673344) result [mem=56663160,vm=1127612416] #GDKmmap: recovery ok. Continuing.. Is there any configuration or a way to make MonetDB work with large tables without having more memory available? Thanks for your help! Regards, Luis Fernando

2 1

[MonetDB-users] Embedded server in Java with JDBC
by Luis Fernando Kauer 22 Jun '10

22 Jun '10

Hi, I would like to use the embedded server within a Java application connecting through JDBC. Is it possible? I would probably need to start the embedded server before trying to connect through JDBC, right? Or is there a connection string that makes the jdbc driver start and use an embedded server? That would be great! If not, what's the best way to start the embedded server in Java? Thanks for this great and fast dabatase. Regards, Luis Fernando

2 1

[MonetDB-users] Mapiclient password prompt
by jz＠uva 20 Jun '10

20 Jun '10

I use the Mclient in conjunction with XQuery. When I do a CVS checkout, I get the version of Mapiclient where the -P option has disappeared, yet it still asks me for a password. My question is: how do I disable the password prompt? It is a bit too much to enter the password for thousands of queries, although I can type quite quickly. ;-) junte

3 4

[MonetDB-users] Get average value for intervals
by Gustav Koller 18 Jun '10

18 Jun '10

Hello, I've some time series data (time + a few value columns), now I want to generate average values for each hour. Currently I'm using the following query: SELECT (time - (time % 3600)) AS timegroup, AVG(value0) AS avgvalue0 FROM dbbench WHERE time > x AND time <= y GROUP BY timegroup; I've a few GB of data and the query is running for several seconds. Does anyone have any recommendations on how to improve the query for faster execution time? Greetings Gustav

2 2

[MonetDB-users] Limit memory
by Miguel Ángel Águila Lorente 17 Jun '10

17 Jun '10

Hello, I'm trying to limit the memory that monetdb can use during its executions. I'm using mal language and I've found this operation in order to do it. command mem_maxsize(v:lng) :void address set_mem_maxsize comment "Set the maximum usable amount of physical swapspace in KB."; When I use this operation the connection is terminated. I let here some examples: (I'm working with a machine with 32 GB) mal>status.mem_maxsize(4294967296:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(4294967296:lng); ERROR = !Connection terminated mal>status.mem_maxsize(1073741824:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1073741824:lng); ERROR = !Connection terminated mal>status.mem_maxsize(1048576:lng); MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = status.mem_maxsize(1048576:lng); ERROR = !Connection terminated Looking forward to hearing from you. Thank you. Mike

4 6

Re: [MonetDB-users] Meaning of: Scenario not initialized 'sql'
by Yel 17 Jun '10

17 Jun '10

Hi, I know this topic has been discussed last year, but with the MonetDB-Feb2010-SuperBall-SP2 installed, I repeatedly got the error like MALException:setScenario:Scenario not initialized 'sql' almost every time when I try to start mclient with "mclient -l sql -d dbname", even when i try it later on. Moreover, only "mclient -l mal -d dbname" is OK, sql, xquery, mil all got the error above. Any idea ? BTW: 1. MonetDB is installed by ./monetdb-install.sh --enable-sql --enable-debug 2. mapi_port is set to 50003 in etc/monetdb5.conf; Thanks. ---- Best Regards Ye Liang Sjoerd Mullender-2 wrote: > Alex Bo. wrote: > > Hi, > > > > Sometimes I get !MALException:setScenario:Scenario not initialized 'sql' > > from the server when I try to start it in scenarios that usually work > > without any problems. I can't reproduce it consistently, but sometimes I get > > it. What does this exception mean? > > > > Thanks. > > This exception means that the server is not yet ready for accepting SQL > connections. The solution is to try again later. > > When the server starts up, it pretty quickly starts accepting > connections, but initially only for MAL. When the SQL engine is > initialized (which may take a while if there was any recovery to be > done), it starts accepting SQL connections as well. You get this > exception when you connect before the SQL engine is ready. > > -- > Sjoerd Mullender

4 4

Re: [MonetDB-users] MonetDB XML - slow execution of query, with test on attribute
by Patrick Schäfer 17 Jun '10

17 Jun '10

Hi Jan, thank you for your response. I thought that all documents within a collection are stored within one (or several) relational tables? I am not sure, if merging all documents into one big document is an option. The documents are supposed to be updateable ... does that give a good performance for one large document? Patrick > Message: 7 > Date: Tue, 15 Jun 2010 21:58:44 +0200 > From: Jan Rittinger <jan.rittinger(a)uni-tuebingen.de> > Subject: Re: [MonetDB-users] MonetDB XML - slow execution of query > with test on attribute > To: Communication channel for MonetDB users > <monetdb-users(a)lists.sourceforge.net> > Message-ID: <702CE627-774E-47C5-B3B7-34A5431F20CF(a)uni-tuebingen.de> > Content-Type: text/plain; charset="windows-1252" > > Hi Patrick, > > the evaluation plan of your query looks quite bad -- with respect to what could be possible... > > If you could use 'doc("tu")' (with a single big document containing all small documents), you would get the result in *no* time, as then the attribute index is used. Unfortunately I'm not aware of the difference in the attribute index with respect to collections... > > Regards, > Jan >

2 1

Re: [MonetDB-users] MonetDB XML - slow execution of query with test on attribute
by Patrick Schäfer 17 Jun '10

17 Jun '10

Hi Peter, thank you for your answer. using pf:collection gives me a "Speicherzugriffsfehler" (/Segmentation fault)/ when executing the query you proposed. Is that a bug in MonetDB XML? replacing "pf:collection" by "fn:collection" leads to Trans 55.436 msec Shred 19.083 msec Query 29241.029 msec Print 0.207 msec Timer 29582.455 msec So it is basically the same query time :( Patrick > Hi, > > How about: > > smiles> { > for $i in pf:collection("tu") > where $i//xq:molecule/@id = "1" > return > <smile>{$i}</smile> > } </smiles> > > is that any faster? > > Peter > > >

1 0

Re: [MonetDB-users] Meaning of: Scenario not initialized 'sql'
by yel 17 Jun '10

17 Jun '10

Hi, I know this topic has been discussed last year, but with the MonetDB-Feb2010-SuperBall-SP2 installed, I repeatedly got the error like MALException:setScenario:Scenario not initialized 'sql' almost every time when I try to start mclient with "mclient -l sql -d dbname", even when i try it later on. Moreover, only "mclient -l mal -d dbname" is OK, sql, xquery, mil all got the error above. Any idea ? BTW: 1. MonetDB is installed by ./monetdb-install.sh --enable-sql --enable-debug 2. mapi_port is set to 50003 in etc/monetdb5.conf; Thanks. ---- Best Regards Ye Liang > > Sjoerd Mullender-2 wrote: > Alex Bo. wrote: > > Hi, > > > > Sometimes I get !MALException:setScenario:Scenario not initialized 'sql' > > from the server when I try to start it in scenarios that usually work > > without any problems. I can't reproduce it consistently, but sometimes I get > > it. What does this exception mean? > > > > Thanks. > > This exception means that the server is not yet ready for accepting SQL > connections. The solution is to try again later. > > When the server starts up, it pretty quickly starts accepting > connections, but initially only for MAL. When the SQL engine is > initialized (which may take a while if there was any recovery to be > done), it starts accepting SQL connections as well. You get this > exception when you connect before the SQL engine is ready. > > -- > Sjoerd Mullender >

1 0