July 2009 - users-list

[MonetDB-users] put()
by Roy Walter 23 Jul '09

23 Jul '09

Hi What is the argument list for the fn:put() function? -- Roy

2 3

Re: [MonetDB-users] Database size
by Charles Samuels 22 Jul '09

22 Jul '09

On Wednesday 2009 July 22 12:25:23 pm Martin Kersten wrote: > Charles Samuels wrote: > > Hi, > > > > I have read previously that monetdb is intended to be used only when the > > data can fit in memory. I'm not sure how to interpret this statement - > > does this mean that the entire database must fit in memory, or does it > > mean that only the data being operated on when selected from the database > > must fit in memory? > > MonetDB does *not* limit your database to fit in main memory. > > However, it uses main memory techniques to address the elements in > tables loaded for processing. This means that the hot-set accessed > during query execution is limited when you run on 32-bit machines. Can you elaborate a little bit? Does hot-set mean that a select on the database can only span the chunk that can fit in a 32-bit process's address space? (This seems like a perfectly reasonable limitation, I'm just making sure). > > > We are wanting to store *huge* amounts of data, ultimately several TBs. > > The actual data being select at any given time will only be several > > hundred MBs, however. Is this a reasonable usage of MonetDB? > > The largest database we use internally to experiment with is a 6TB > astronomical database. It contains two large tables, one with 400M rows of > 500 columns and another of 20 B rows and 6 columns. Both tables are larger > then 1TB each. That's really great to hear :) > > This runs on a Linux dual quadcore with 64G Ram and lots of disk space. > Ofcourse, at some point you well notice IO behavior ;) For what kind of operations do you need that amount of memory? If you were to do a select on just a few thousand rows on that same mega-database, would it be happy with more "mundane" amounts of memory? > > > But most importantly, I have noticed that all the row ID indexes are > > "int" - why the 32 bit limitation? This just seems like a symptom of the > > above problem; that the entire database must fit in memory. > > You can use the system with both 32- and 64- bit oids. Ah, I previously had a bit of a brain failure. When reading Mapi.h, I noticed that "mapi_get_row_count" returned an int - obviously you wouldn't want select to return 2^32 rows, which is what this function counts the rows of. Thanks Martin (and also Lefteris), Charles -- Charles Samuels

2 1

[MonetDB-users] Database size
by Charles Samuels 22 Jul '09

22 Jul '09

Hi, I have read previously that monetdb is intended to be used only when the data can fit in memory. I'm not sure how to interpret this statement - does this mean that the entire database must fit in memory, or does it mean that only the data being operated on when selected from the database must fit in memory? We are wanting to store *huge* amounts of data, ultimately several TBs. The actual data being select at any given time will only be several hundred MBs, however. Is this a reasonable usage of MonetDB? But most importantly, I have noticed that all the row ID indexes are "int" - why the 32 bit limitation? This just seems like a symptom of the above problem; that the entire database must fit in memory. Thanks a lot for your advice, Charles -- Charles Samuels

4 3

Re: [MonetDB-users] Performance
by Roy Walter 22 Jul '09

22 Jul '09

OK so this is interesting. I don't have a dbfarm directory. I looked in monetdb.conf and noticed a couple of things. The MonetDB installation path value appears as: prefix=c:\documents and settings\sjoerd\my documents\src\stable\icc32\nt32 This means that neither the datadir nor the gdk_dbfarm entries can be properly processed. Might this explain why I [and others] have to start the XQuery Server from the command line and load the modules manually? So right now I can't find my data :) What file extensions does MonetDB use? -- Roy Lefteris wrote: > Hi, > > it looks that something got corrupted in the process. This is starting > to be very usual with windows installations. Anyway, you should delete > the contents of the dbfarm directory in the monetdb installation > directory. This will *delete* all data in your database (i guess you > dont have any important data still in cause you are testing). After > you delete dbfarm, start mserver, that will populate again dbfarm with > the default files. then, add again your data with the usual > pf:add-doc, then create the tijah indices and then run your queries:) > > If this does not work, we will have to try to "wake up" the pf/tijah > people to help us. > > Hope this will help, > > lefteris > > On Tue, Jul 21, 2009 at 10:35 AM, Roy Walter<garliestonhouse(a)yahoo.co.uk> wrote: > >> OK, I restarted and got an error message that pointed to a problem document >> in a collection. I deleted the offending document and then tried to generate >> the default index with tijah:create-ft-index(). This failed because, >> apparently, the DFLT_FT_INDEX already exists. >> >> So I thought that even though the index compilation appeared to have failed >> at the earlier error an index must have been created. >> >> I tried a tijah:query(). That failed because the DFLT_FT_INDEX does not >> exist. >> >> Hmm, so I tried tijah:delete-ft-index() and it too told me that >> DFLT_FT_INDEX does not exist. >> >> tijah:create-ft-index() still fails with: !ERROR tj_init_collection, pftijah >> collection already exists: DFLT_FT_INDEX >> >> How do I reset? >> >> -- Roy >> >> >> Lefteris wrote: >> >> This is not expected. >> >> Did you try to restart the server and retry? >> >> You might also have a corrupted dbfarm or the documents didn't shred >> correctly to begin with. Which version of monet are you using? how did >> you installed it? >> >> lefteris >> >> On Mon, Jul 20, 2009 at 8:52 PM, Roy Walter<garliestonhouse(a)yahoo.co.uk> >> wrote: >> >> >> Hi lefteris >> >> Well that seems to tick all the boxes. >> >> I tried the global index creation: >> >> tijah:create-ft-index() >> >> and it crashed the server with: >> >> !WARNING: readClient: unexpected end of file; discarding partial input >> >> Hmm... >> >> R. >> >> Lefteris wrote: >> >> >> Hi Roy, >> >> I suggest that you try the pf/tijah module for MonetDB/XQuery. >> >> http://dbappl.cs.utwente.nl/pftijah/ >> >> This will create specific indices for your queries to facilitate text >> search. >> >> Hope this helps for now. We will also investigate were the time is >> spent in your case (without pf/tijah) and come back to you. How many p >> elements your documents have? The problem might be that because monet >> does not build inverted indices on text by itself, it has to visit >> each p element and search with the help of the pcre library. Pf/tijah >> was build for that purpose and should help alot. >> >> Please feel free to contact us for further clarification and new >> findings from your tests:) >> >> cheers, >> >> lefteris >> >> On Mon, Jul 20, 2009 at 6:28 PM, Roy Walter<garliestonhouse(a)yahoo.co.uk> >> wrote: >> >> >> >> Running MonetDB/XQuery on a 2.6GHz 32-bit Windows XP box with 1GB of RAM. >> >> What is the best way to organise XML in MonetDB for rapid text searching? >> A >> run down of my recent experience might help. >> >> I created a collection of around 450 documents (153MB approx.). I ran the >> following query from the command line: >> >> collection("papers")//p[contains(., 'wind farm')] >> >> The query time is at best 19 seconds. That's bad. (It's worse than >> querying >> a Postgres database with documents stored in the XML field type.) >> >> So to get a reference point I loaded up the 114MB XMark document and ran >> this query: >> >> doc("standard")//text[contains(., "yoke")] >> >> The query time varies from 2 to 4 seconds. Better, but still not great. >> >> Now, adding more RAM (and moving to 64-bit) would speed things up I hope! >> But hardware aside: >> >> 1. Is it better to have big documents rather than big collections? >> >> 2. Is having small collections (<10 docs) of big documents also >> inefficient? >> >> Ideally I need to query collections comprising several thousand documents >> using 'text search' predicates. Are there other, better ways to run this >> type of query against a MonetDB XML database? Or should I really be using >> some other platform for this task? >> >> Thanks in advance for any pointers. >> >> -- Roy >> >> >> ------------------------------------------------------------------------------ >> Enter the BlackBerry Developer Challenge >> This is your chance to win up to $100,000 in prizes! For a limited time, >> vendors submitting new applications to BlackBerry App World(TM) will have >> the opportunity to enter the BlackBerry Developer Challenge. See full >> prize >> details at: http://p.sf.net/sfu/Challenge >> _______________________________________________ >> MonetDB-users mailing list >> MonetDB-users(a)lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/monetdb-users >> >> >> >> >> >> >> >> >> > >

3 9

[MonetDB-users] Linux memory overcommit settings
by Franck Routier 21 Jul '09

21 Jul '09

Hi, Does anyone know, when using monetdb, what would be the common wisdom regarding the memory-overcommit settings on a Linux platform, regarding the performance impacts ? Franck

1 0

[MonetDB-users] Wrong callHistory information ?
by Franck Routier 21 Jul '09

21 Jul '09

Hi, testing monetdb, I've come accross a query that takes several minutes on my dev machine (with only 2 GB of RAM). So it will swap for a while, and finally produce the result. I wanted to get the duration of the request, so I did 'set history=true;' The request takes a few minutes, and returns 464 tuples. But when I select from callHistory, here is the outcome : +---------+----------------------------+------------------+-------------+-------+------+--------+--------+---------+---------+ | id | ctime | arguments | xtime | rtime | foot | memory | tuples | inblock | oublock | +=========+============================+==================+============= +=======+======+========+========+=========+=========+ | 6776676 | 2009-07-21 13:25:27.000000 | user.s6_1(); | 22108050926 | 0 | 0 | 0 | 1 | 0 | 0 | +---------+----------------------------+------------------+-------------+-------+------+--------+--------+---------+---------+ Obviously, xtime is wrong (the query did not last for 6 hours). I guess rtime is also a bit quick :-) And the number of tuples is also wrong... Running the same query with \t in mclient gives: Timer 322316.803 msec 464 rows This seems more like truth. Well, I am conscious that with that information, it's hard to tell where the problem is.... So my question is rather: is the information in callHistory expected to be reliable, or more like an indication ? Franck

1 0

[MonetDB-users] Performance
by Roy Walter 20 Jul '09

20 Jul '09

Running MonetDB/XQuery on a 2.6GHz 32-bit Windows XP box with 1GB of RAM. What is the best way to organise XML in MonetDB for rapid text searching? A run down of my recent experience might help. I created a collection of around 450 documents (153MB approx.). I ran the following query from the command line: collection("papers")//p[contains(., 'wind farm')] The query time is at best 19 seconds. That's bad. (It's worse than querying a Postgres database with documents stored in the XML field type.) So to get a reference point I loaded up the 114MB XMark document and ran this query: doc("standard")//text[contains(., "yoke")] The query time varies from 2 to 4 seconds. Better, but still not great. Now, adding more RAM (and moving to 64-bit) would speed things up I hope! But hardware aside: 1. Is it better to have big documents rather than big collections? 2. Is having small collections (<10 docs) of big documents also inefficient? Ideally I need to query collections comprising several thousand documents using 'text search' predicates. Are there other, better ways to run this type of query against a MonetDB XML database? Or should I really be using some other platform for this task? Thanks in advance for any pointers. -- Roy

3 5

[MonetDB-users] XQuery performance
by Roy Walter 20 Jul '09

20 Jul '09

Hi I'm just trying out MonetDB and I have a performance question. I'm running the software on an 2.6GHz 32-bit XP box with 1GB RAM. I have created a small collection of 480 documents, approx. 153MB. Using the console client I run the following query: for $p in collection("papers")//p where contains($p, 'wind farm') return($p) Enabling the timer I see that the query time is 29675 msec. In other words, way too slow to be useful. Clearly there is something fundamentally wrong here. Any pointers? -- Roy Walter

1 0

[MonetDB-users] Has someone already made the jdbkc driver compile with recent jdk-1.5 or newer?
by Marc Weber 19 Jul '09

19 Jul '09

Hi, I tried compiling monetdb with jdbc support today using jdk-1.5.0.19. However the build failed because many abstract methods are not implemented. Has someone already done some work on this? Marc Weber

2 2

[MonetDB-users] server crash
by dariuszs 18 Jul '09

18 Jul '09

Hi, What does this mean 'too many open files'? !SABAOTHgetMyStatus: IOException:sabaoth.getStatus:Unable to open directory /vol1/MonetDB5/dbfarm: Too many open files Thanks. Dariusz.

3 2