On Sep 24, 2009, at 16:30 , Stefan Manegold wrote:

On Thu, Sep 24, 2009 at 12:12:38PM +0200, Axel Belinfante wrote:

...

First of all, I'm a monetdb/xquery/ruby novice.

I'm trying to do xqueries from the web in two ways: from a ruby-on-rails application, and from a cgi script.

For the cgi script I use (a perl wrapper around) mclient, which seems to work ok. The remainder of this mail is about the access from ruby-on-rails.

For the ruby-on-rails application, I typically have to do relatively small queries. The heart of the xqueries is in xquery module files (partly 'static', partly generated 'dynamically' from user data entered in web forms) and thus the queries to be done from ruby-on-rails are rather small: import the module, and invoke the right function with some parameters.

I started using xrpc.

However, I'm encountering issues that might be related to the fact that xrpc uses one backend, and mclient (by default) uses another one. (Being an xquery novice) I've also encountered some differences between the xquery support by both backends.

By default, mclient use the new "Algebra (ALG)" back-end; however, since this does not (yet) support XRPC, MonetDB/XQuery automatically falls back to the (old/legacy) "milprint_summer (MPS)" back-end. If required, you can make mclient use MPS instead of ALG by calling `mclient -lx --no-algebra`; for details, see http://monetdb.cwi.nl/Development/Releases/Version4.24/index.html#XQuery

Would you mind sharing which issues exactly you are encountering?

issues like: http://sourceforge.net/tracker/?func=detail&atid=482468&aid=2859986&group_id=56967 I don't know whether that issue is related to other issues I reported on the bug tracker, like the one that crashes monetdb.

...

I then started looking at the mapi interface, because that would allow the use of the same backend from both webapp an cgi scripts.

To my understanding there are a couple of different routes possible, and I'm not sure what would be the best way.

- use the native ruby code present in the MonetDB distribution this I tried, but it seems to not fully up-to-date/working in the Aug-2009 distribution at least, I had to hack MonetDBConnection.rb to make it work

The (new) native Ruby interface was mainly created with SQL in mind, and might indeed not (yet) be up to handling XQuery properly. Whould you mind sharing with us, which exact problems you encoutered, and which changes you applied?

I only used MonetDBConnection.rb I'm reporting from a diff between my hacked version of MonetDBConnection.rb and the distributed one (I did not keep notes while hacking) 1) in the receive method, if chunk_size==0 @socket.recv(chunk_size) would hang, I think, so I changed it as follows: def receive is_final, chunk_size = recv_decode_hdr - data = @socket.recv(chunk_size) + data = '' + data += @socket.recv(chunk_size) if chunk_size > 0 if is_final == false while is_final == false is_final, chunk_size = recv_decode_hdr - data += @socket.recv(chunk_size) + data += @socket.recv(chunk_size) if chunk_size > 0 end end 2) due to the monetdb crash that I reported on the bug tracker I would loose the connection to the server. Unfortunately, Socket.recv does not report a lost connection, but does return immediately, and then in recv_decode_hdr we forever loop with is_final==false and chunk_size==true therefore I changed the recv_decode_hdr method as below. there may be better ways to do this (I'm a ruby novice) (note: it seems that the exceptions defined in MonetDBExceptions.rb can not be used as they are now, because the initialize method of the exception classes expects an argument. this might require changes elsewhere too.) fb = @socket.recv(1) + if (fb.empty?) + @socket.close + @socket = nil + raise MonetDBSocketError.new("MonetDB connection closed by peer") + end sb = @socket.recv(1) + if (fb.empty?) + @socket.close + @socket = nil + raise MonetDBSocketError.new("MonetDB connection closed by peer") + end - raise MonetDBSocketError + raise MonetDBSocketError.new("unexpected nil socket for MonetDB") 3) I added two methods to send an xquery message and to decode a response. maybe this better fits somewhere else in the code. + # Formats a query <i>string</i> so that it can be parsed by the server + def format_query(q) + if @lang.downcase == 'xquery' + return "s" + q + "\n" + else + raise LanguageNotSupported, @lang + end + end + + def decode_result(r) + r.gsub(/\n=/,"\n").sub(/^=/,"") if ! r.nil? + end + I use this as follows, where req is a single string that contains an xquery. it is not robust. require 'MonetDBConnection' def self.post_query(host, port, req) @connection = MonetDBConnection.new(user = 'monetdb', passwd = 'monetdb', lang = 'xquery', host = '127.0.0.1', port = '51012') @connection.connect('demo', 'plain') @connection.set_reply_size data = @connection.receive @connection.send(@connection.format_query(req)) data = @connection.receive res = @connection.decode_result(data) @connection.disconnect() res end I have similar code to send a single formatted query over xrpc, and code to format a query for mapi use, and for xrpc use, with the same interface - this allows me to easily switch between mapi and xrpc as underlying transport. My current code is still specific to my use in one minor detail; it could easily be generalised, I think. However, error checking in my code is pretty minimal (absent?) right now.

This might help us to improve the Ruby such that you and other users might benefit from it ...

...

- use the swig mapi wrapper (via http://rubyforge.org/projects/monetdb-ror) I did not try that yet

This was never "officially" supported and has been replaced by the new native Ruby interface. I strongly recommend to stick to the native one. In fact, I doubt, that this swig mapi wrapper provides any better XQuery support than the new native one ...

thanks for making this explicit. I could imagine that simply using swig would avoid the additional effort of having to maintain yet one more (in this case a native Ruby) interface.

...

- somehow wrap mclient (using something like fork/exec from ruby) did not try that yet, but I have been using mclient to try queries

Probably possible, but not as "elegant" as using the native Ruby interface --- as indecated above, the latter might require your detailed feedback and/or help to get it working properly.

...

- use libmapi to build my own translation daemon/program, that is kind of like mclient but e.g. without the prompts etc, that receives/sends xqueries/responses as strings on a socket (or so), and talks mapi to the database. I did not try this; I did compile and run the sample mapi program, which could be used as a start, I guess

I'm kind of looking for comments, and in particular I'm curious about the status of the native ruby code included in the distribution - I would prefer to use that, but for some of the basic things I looked at the mapi C library seems to do things in a more involved way than the ruby code, which makes me wonder.

AFAIK, we have not looked into Ruby + XQuery, yet, hence, your detailed experiences and feedback are more than welcome.

...

On the other hand, it seems that for xquery support the library mainly has to pass strings around - not all the complicated things that seem to be necessary for, for example, the sql support.

Thanks, Axel.

Stefan