Maurice, finally, here's my reply --- at least as far as I Think I can answers your question right now... On Fri, Sep 29, 2006 at 10:02:51AM +0200, Maurice van Keulen wrote:

Hi Stefan,

Thanks for resolving some of my bugs. I'm trying to develop an XQuery module for probabilistic XML. This involves highly recursive functions and XML fragments being constructed and then queried upon again in the next function. I'm impressed that I got so far already (322 lines of really complex XQuery code). Every time I do stumble onto a bug, I try to isolate it and report it nicely. Currently, the bug that handers my progress most is "1562868: Complex XQuery fails with error in "batbat_lng_add_inplace". It is hard to already give the correct answer to the test you created. I suggest to simply approve the result when the error message does not occur anymore. And, if you are ready to pick one of my bugs I reported, please have a look at this one if you will.

Well, Jan spent quite some time on this one (see the bug report; Thanks, Jan!) and it appears to be a tircky one --- I'll consider applying Jan's "fast and ugly" solution for the time being, and *try* to find a better one as soon as time permits...

I also have a question. Eventually, my functions need to become part of a module for Probabilistic XML. Besides XQuery functions, I also have a few fundamental functions that are much better dealt with in MIL. I have, for example, a function product that, given a sequence of numbers, multiplies all these numbers:

declare function product($seq as xs:decimal*) as xs:decimal { if (count($seq) gt 0) then exactly-one($seq[1]) * product($seq[position() gt 1]) else 1.0 };

This recursive definition of the function is for sure much less efficient than something I could write in MIL.

My question is: how can I develop a module with additional functions that are implemented in MIL and usable in XQuery?

Well, obviously(?), there are to ways to go. The first is to indeed add a new built-in function and provide a MIL implementation as we do for all other built-in functions --- I guess, this is not what you intended, right? The second one is to provide an XQuery module. In fact, MonetDB/XQuery allows both "plain" XQuery module definitions and MIL module definitions --- as far as I know, the best (only? for now?) way to create a MIL modules (template) is to have the pathfinder compiler translate an XQuery module to MIL (basically, Peter added the support for MIL modules to have pre-compiled modules, saving the XQuery to MIL translation...). Here's my approach to your example: ======== $ cat /tmp/mvk.xq module namespace mvk = "http://wwwhome.cs.utwente.nl/~keulen/"; declare function mvk:product($seq as xs:decimal*) as xs:decimal { if (count($seq) gt 0) then exactly-one($seq[1]) * mvk:product($seq[position() gt 1]) else 1.0 }; -------- $ pf /tmp/mvk.xq > /tmp/mvk.mil -------- $ echo 'import module namespace mvkXQ = "http://wwwhome.cs.utwente.nl/~keulen/" at "/tmp/mvk.xq"; mvkXQ:product((1,2,3,4))' | MapiClient -lx <?xml version="1.0" encoding="utf-8"?> <XQueryResult>24.000000</XQueryResult> -------- $ echo 'import module namespace mvkMIL = "http://wwwhome.cs.utwente.nl/~keulen/" at "/tmp/mvk.mil"; mvkMIL:product((1,2,3,4))' | MapiClient -lx <?xml version="1.0" encoding="utf-8"?> <XQueryResult>24.000000</XQueryResult> ======== well, if you look at the attached mvk.mil, you'll see that it looks quite complicated, and using it as a template to replace the recursive product implementation by a more efficient version will be quite tricky. Hence, here's another approach: While there is (AFAIK) not product aggregation in XQUery (yet?), there is one in MIL; and there is a sum aggregation in both. Hence, simply take a fn:sum in XQuery have it translated to MIL, which results in usage of {sum}() in MIL, and replace the latter by {prod}(): ======== $ cat /tmp/stm.xq module namespace stm = "http://www.cwi.nl/~manegold/"; declare function stm:product($seq as xs:decimal*) as xs:decimal { sum($seq) }; -------- $ pf /tmp/stm.xq > /tmp/stm.mil -------- $ egrep '{sum}|{prod}' /tmp/stm.mil var iter_aggr := {sum}(item.leftfetchjoin(dec_values), iter_grp, reverse(loop000)).tmark(0@0); -------- $ perl -i -p -e 's|{sum}|{prod}|' /tmp/stm.mil -------- $ egrep '{sum}|{prod}' /tmp/stm.mil var iter_aggr := {prod}(item.leftfetchjoin(dec_values), iter_grp, reverse(loop000)).tmark(0@0); -------- $ echo 'import module namespace stmXQ = "http://www.cwi.nl/~manegold/" at "/tmp/stm.xq"; stmXQ:product((1,2,3,4))' | MapiClient -lx <?xml version="1.0" encoding="utf-8"?> <XQueryResult>10.000000</XQueryResult> -------- $ echo 'import module namespace stmMIL = "http://www.cwi.nl/~manegold/" at "/tmp/stm.mil"; stmMIL:product((1,2,3,4))' | MapiClient -lx <?xml version="1.0" encoding="utf-8"?> <XQueryResult>24.000000</XQueryResult> ======== Seem to work well ;-) As said, I don't know a better way to produce a MonetDB/XQuery conforming MIL module by hand, but this way, you can create a template and modify it even further than I did here. There's one(?) caveat, though: Modules created this way are not necessairyly protable from one version of MonetDB/XQuery to another one, as the requires/expected signature of the respective MIL procedure and/or format/content of its arguments might change... Hope this helps you further, anyway... More suggestions are of course welcome (that's why I moved this discussion to this mailing list ;-)) Stefan

Thanks again for your effort! Maurice.

-- ---------------------------------------------------------------------- Dr.Ir. M. van Keulen - Assistant Professor, Data Management Technology Univ. of Twente, Dept of EEMCS, POBox 217, 7500 AE Enschede, Netherlands Email: m.vankeulen@utwente.nl, Phone: +31 534893688, Fax: +31 534892927 Room: INF3039, WWW: http://www.cs.utwente.nl/~keulen

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |