Hello Nicolas, I need some more information about your problems to be able to help. First of all, some general questions: - which version of MonetDB/XQuery do you use? - which OS? - some hardware specifications On Jan 06, 2009, at 14:45 , Nicolas James wrote:
Hello Ying,
Thanks for all these responses, they have been very helpful to me (the real problem I had, and now resolved, was about the call to the functions of pathfinder).
I've coded a Java client for indexing my data (image descriptions and annotations in XML files) in MonetDB/XQuery, 46,557 XML documents / 181.1Mo and I use XRPC.
Is 181.1MB the size of one document (then the total size is > 8TB!) or the size of all 46557 documents? Do I understand it correctly that you have a Java client which send XRPC request messages to MonetDB/XQuery server to add documents? How does your Java client exactly work, i.e.: - does it add one document at a time, or all 46557 documents in one request? - is your Java client and MonetDB/XQuery server on the same machine? - do you use pf:add-doc() or fn:put() to add the documents?
My client works well but I don't understand some MonetDB behaviors. My first tests created collections like my dataset and his directories structure,
I don't understand what you mean here. As far as I know, it is not possible to create nested collections in MonetDB/XQuery, and it is not possible to create empty collections (without adding a document). So, how did you create the collections?
but I've some directories with over that 900 documents, and I think that MonetDB doesn't like that (i.e. 900 documents in one collections), because the indexation failed.
I don't know the exact max nr. of documents a collection may have, I think it is limited by if you are using a 32bits system or a 64bits system. Using mlcient, adding 46,557 documents of 11k each on a Fedora machine was no problem; adding > 900 documents of 200MB each is still processing (disk usage is still growing) when I'm writing this e-mail :) Did you add your documents using your Java client? Have you tried to add your documents without using XRPC, e.g., using mclient? What do you mean with "indexation failed"? Is there any error message?
Then, I try to split my dataset, I create collections of 128 documents, but my problem is that in MonetDB all my collections does not contain 128 doc, some 128 others a number between 5 and 128... It's a little strange because I don't think the reason is my client...
For this problem, I have similar questions, i.e.: - how large are the documents? - how are they added _exactly_? - Did you get any error message? Kind regards, Jennie
Have you ever heard about a similar problem in MonetDB?
Thanks previous responses,
best regards,
Nicolas James
Ying Zhang wrote:
Hello Nicolas,
First of all, thanks for trying MonetDB. I will try to answer your XRPC questions.
On Tue, Dec 16, 2008 at 04:35:33PM +0100, Nicolas James wrote:
Hello MonetDB,
I have some questions about the writing of a client for MonetDB in Java using the XRPC, and It could be very very helpful for my research if you could answer to some questions.
I'm a PhD student in the MAS laboratory, Ecole Centrale Paris, France, and I work on image automatic annotation. I think MonetDB is extremely powerful and I want to use it in my research, I would to store some images annotations under the form of XML file in MonetDB since my application (a Java application).
So, with your documentation I coded a light client using the JDBC driver for adding my documents in MonetDB, but this way don't satisfy me, the XRPC method is very attractive, but I've some problems.
1) how add a document using the XRPCWrapper (using the pf:add-doc, it's considered like an updating function, I've read the document, but I don't understand...).
XRPC has several components. The XRPC protocol is a way of communicating with MonetDB/XQuery to run XQuery queries. An XRPC server is integrated in MonetDB/XQuery to handle requests speaking the XRPC protocol.
Hence, to communicate with MonetDB/XQuery using XRPC, you only need to send requests according to the XRPC protocol. For Java, there are several API functions for creating request messages, sending and receiving messages, and handling response messages. The Java doc of these APIs could be found here: http://homepages.cwi.nl/~fabian/MonetDB/Java%20Package.html/index.html?nl/cw... The Java package of the APIs could be found under the directory where you have installed MonetDB: share/MonetDB/lib/xrpcapi.jar In the same directory there is an xrpcclient.jar package, which gives an example of how to use the APIs. It's source could found in: java/src/nl/cwi/monetdb/xquery/xrpc/client
You don't need the XRPCWrapper, in your case. XRPCWrapper is needed if you want to handle (simple) XRPC requests using another XQuery engine than MonetDB/XQuery.
2) Is-it possible to create collections using the XRPC (and add files in this collection) ?
I'm not sure if I understand you correctly.
With XRPC, you can execute the pf:add-doc() function (you need to put it into an user-defined-function). with pf:add-doc(), you could specify the name of the collection to which you want to add your document. If such collection doesn't exist, it is created. Otherwise, your document will be added to the existing collection.
3) and with the JDBC driver ? 4) using the JDBC driver, is it possible to specify the reference id for the file (for instance indexing the file "foobar.xml" and referenced "foobar.xml" in the database ?
What do you mean with 'indexing the file "foobar.xml"'? (Disclaimer: I don't know much about the JDBC driver)
Hope this helps a bit. If you have any problem with using the XRPC API, please don't hesitate to ask.
Regards,
Jennie
These answers could help me, hope you could have time to answer!
best regards,
-- Nicolas James PhD student MAS / ECP Phone: +33.1.41.13.17.37 Office: Lab, C 441A Laboratoire MAS - Ecole Centrale Paris Grande Voie des Vignes 92295 CHATENAY MALABRY Cedex France
-- Nicolas James PhD student MAS / ECP Phone: +33.1.41.13.17.37 Office: Lab, C 441A Laboratoire MAS - Ecole Centrale Paris Grande Voie des Vignes 92295 CHATENAY MALABRY Cedex France