Re: [Monetdb-developers] [Monetdb-pf-checkins] pathfinder/runtime pathfinder.mx, , 1.421, 1.422 pf_support.mx, , 1.301, 1.302
On Tue, Jun 17, 2008 at 05:07:29PM +0000, Peter Boncz wrote:
Update of /cvsroot/monetdb/pathfinder/runtime In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv2937
Modified Files: pathfinder.mx pf_support.mx Log Message: - BUG 1995879: only use algebra for normal queries
cachable queries and the XRPC server still use mps
Peter, just for clarification and the records: am I right that this means that now all XQuery modules are compiled with MPS, but when a (non-XRPC?) query uses an XQuery module, this XQuery itself is compiled with ALG? Or are such queries compiled with MPS, too? Thanks in advance! Stefan
U pathfinder.mx Index: pathfinder.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pathfinder.mx,v retrieving revision 1.421 retrieving revision 1.422 diff -u -d -r1.421 -r1.422 --- pathfinder.mx 16 Jun 2008 23:38:41 -0000 1.421 +++ pathfinder.mx 17 Jun 2008 17:07:26 -0000 1.422 @@ -4246,7 +4246,7 @@ * - execute MIL buffer (parse & execute) * * int - * xquery_compile_exec(xquery_client *ctx, char* xquery, int is_url, + * xquery_compile_exec(xquery_client *ctx, int options, char* xquery, int is_url, * char** prologue, char** query, char** epilogue, char* nsurl) * - translate xquery to MIL and execute * @@ -4345,6 +4345,7 @@ extern char* PFmaxstack; static int xquery_compile_exec(xquery_client *ctx, + int options, char* url, int is_url, char** prologue, @@ -4356,22 +4357,9 @@ char *xquery = url; char *err = NULL;
- int options = 0; /* Setting the StandOff flag based on runtime settings */ if ((GDKgetenv("standoff") != NULL) && (strcmp(GDKgetenv("standoff"),"enabled") == 0)) options |= COMPILE_OPTION_STANDOFF; - /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ -#if MILPRINT_SUMMER_IS_DEFAULT - if (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) - options |= COMPILE_OPTION_ALGEBRA; -#else /* ALGEBRA_IS_DEFAULT */ - if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) - options |= COMPILE_OPTION_ALGEBRA; -#endif
MT_set_lock(pf_compiler_lock, "xquery_compile_exec"); if (is_url) { @@ -4910,7 +4898,7 @@ strcpy(mod->url, url); mod->nsurl = mod->url + url_len;
- ret = xquery_compile_exec(ctx, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); + ret = xquery_compile_exec(ctx, 0, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); if (!ret) { xquery_module_free(mod); return NULL; @@ -5745,9 +5733,22 @@ char *sec1 = NULL; char *sec2 = NULL; char *sec3 = NULL; + int options = 0; /* compile and execute the query (minus module imports) */ err = xquery_nondescriptive_error; - if (xquery_compile_exec(ctx, query, 0, &sec1, &sec2, &sec3, NULL)) + /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ +#if MILPRINT_SUMMER_IS_DEFAULT + if (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) + options = COMPILE_OPTION_ALGEBRA; +#else /* ALGEBRA_IS_DEFAULT */ + if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) + options = COMPILE_OPTION_ALGEBRA; +#endif + if (xquery_compile_exec(ctx, options, query, 0, &sec1, &sec2, &sec3, NULL)) err = NULL; if (sec1) free(sec1); if (sec2) free(sec2);
U pf_support.mx Index: pf_support.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pf_support.mx,v retrieving revision 1.301 retrieving revision 1.302 diff -u -d -r1.301 -r1.302 --- pf_support.mx 10 Jun 2008 09:22:17 -0000 1.301 +++ pf_support.mx 17 Jun 2008 17:07:27 -0000 1.302 @@ -8373,7 +8373,7 @@ BUNins(*res, &docpre, &docid[j], FALSE);
/* skip all nodes of this document */ - docpre = docpre + size[docpre]; + docpre = docpre + size[docpre] + 1; for (delta = (n-i) >> 4; delta > 40; delta >>= 4) while (i+delta < n && * (oid *) BUNtail(inputi, i+delta) <= docpre) i += delta;
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
Stefan, The pre-parser still uses MPS. It loads and caches modules. When it sees a query that is just a function call with simle atomic parameters, it uses the fast-path (i.e. execute a precompiled MIL tree). XRPC always enters on the fast-path. If the query is of any other form, it is fully recompiled and then executed with whatever compiler is active (ALG or MPS). This means that indeed imported modules may be compiled twice (that was already the case). First, the preparser compiles them with MPS and caches them. Second, when re-use it is not possible, the complete query is compiled again. Porting XRPC to algebra is on Jennie's plate. However, Jennie is not such a fast eater and it is a big meal. Therefore, this workaround should go into the HEAD also. Peter -----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: woensdag 18 juni 2008 10:11 To: monetdb-developers@lists.sourceforge.net; Peter Boncz Subject: Re: [Monetdb-pf-checkins] pathfinder/runtime pathfinder.mx, ,1.421, 1.422 pf_support.mx, , 1.301, 1.302 On Tue, Jun 17, 2008 at 05:07:29PM +0000, Peter Boncz wrote:
Update of /cvsroot/monetdb/pathfinder/runtime In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv2937
Modified Files: pathfinder.mx pf_support.mx Log Message: - BUG 1995879: only use algebra for normal queries
cachable queries and the XRPC server still use mps
Peter, just for clarification and the records: am I right that this means that now all XQuery modules are compiled with MPS, but when a (non-XRPC?) query uses an XQuery module, this XQuery itself is compiled with ALG? Or are such queries compiled with MPS, too? Thanks in advance! Stefan
U pathfinder.mx Index: pathfinder.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pathfinder.mx,v retrieving revision 1.421 retrieving revision 1.422 diff -u -d -r1.421 -r1.422 --- pathfinder.mx 16 Jun 2008 23:38:41 -0000 1.421 +++ pathfinder.mx 17 Jun 2008 17:07:26 -0000 1.422 @@ -4246,7 +4246,7 @@ * - execute MIL buffer (parse & execute) * * int - * xquery_compile_exec(xquery_client *ctx, char* xquery, int is_url, + * xquery_compile_exec(xquery_client *ctx, int options, char* xquery, int
is_url,
* char** prologue, char** query, char** epilogue, char* nsurl) * - translate xquery to MIL and execute * @@ -4345,6 +4345,7 @@ extern char* PFmaxstack; static int xquery_compile_exec(xquery_client *ctx, + int options, char* url, int is_url, char** prologue, @@ -4356,22 +4357,9 @@ char *xquery = url; char *err = NULL;
- int options = 0; /* Setting the StandOff flag based on runtime settings */ if ((GDKgetenv("standoff") != NULL) && (strcmp(GDKgetenv("standoff"),"enabled") == 0)) options |= COMPILE_OPTION_STANDOFF; - /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ -#if MILPRINT_SUMMER_IS_DEFAULT - if (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) - options |= COMPILE_OPTION_ALGEBRA; -#else /* ALGEBRA_IS_DEFAULT */ - if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) - options |= COMPILE_OPTION_ALGEBRA; -#endif
MT_set_lock(pf_compiler_lock, "xquery_compile_exec"); if (is_url) { @@ -4910,7 +4898,7 @@ strcpy(mod->url, url); mod->nsurl = mod->url + url_len;
- ret = xquery_compile_exec(ctx, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); + ret = xquery_compile_exec(ctx, 0, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); if (!ret) { xquery_module_free(mod); return NULL; @@ -5745,9 +5733,22 @@ char *sec1 = NULL; char *sec2 = NULL; char *sec3 = NULL; + int options = 0; /* compile and execute the query (minus module imports) */ err = xquery_nondescriptive_error; - if (xquery_compile_exec(ctx, query, 0, &sec1, &sec2, &sec3, NULL)) + /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ +#if MILPRINT_SUMMER_IS_DEFAULT + if (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) + options = COMPILE_OPTION_ALGEBRA; +#else /* ALGEBRA_IS_DEFAULT */ + if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) + options = COMPILE_OPTION_ALGEBRA; +#endif + if (xquery_compile_exec(ctx, options, query, 0, &sec1, &sec2, &sec3, NULL)) err = NULL; if (sec1) free(sec1); if (sec2) free(sec2);
U pf_support.mx Index: pf_support.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pf_support.mx,v retrieving revision 1.301 retrieving revision 1.302 diff -u -d -r1.301 -r1.302 --- pf_support.mx 10 Jun 2008 09:22:17 -0000 1.301 +++ pf_support.mx 17 Jun 2008 17:07:27 -0000 1.302 @@ -8373,7 +8373,7 @@ BUNins(*res, &docpre, &docid[j], FALSE);
/* skip all nodes of this document */ - docpre = docpre + size[docpre]; + docpre = docpre + size[docpre] + 1; for (delta = (n-i) >> 4; delta > 40; delta >>= 4) while (i+delta < n && * (oid *) BUNtail(inputi, i+delta) <= docpre) i += delta;
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Wed, Jun 18, 2008 at 10:25:41AM +0200, Peter Boncz wrote:
Stefan,
The pre-parser still uses MPS. It loads and caches modules. When it sees a query that is just a function call with simle atomic parameters, it uses the fast-path (i.e. execute a precompiled MIL tree). XRPC always enters on the fast-path.
If the query is of any other form, it is fully recompiled and then executed with whatever compiler is active (ALG or MPS). This means that indeed imported modules may be compiled twice (that was already the case). First, the preparser compiles them with MPS and caches them. Second, when re-use it is not possible, the complete query is compiled again.
I see. Thanks!
Porting XRPC to algebra is on Jennie's plate. However, Jennie is not such a fast eater and it is a big meal. Therefore, this workaround should go into the HEAD also.
Well, it is already there, since you checked it in there, first ;-) Stefan
Peter
-----Original Message----- From: Stefan Manegold [mailto:Stefan.Manegold@cwi.nl] Sent: woensdag 18 juni 2008 10:11 To: monetdb-developers@lists.sourceforge.net; Peter Boncz Subject: Re: [Monetdb-pf-checkins] pathfinder/runtime pathfinder.mx, ,1.421, 1.422 pf_support.mx, , 1.301, 1.302
On Tue, Jun 17, 2008 at 05:07:29PM +0000, Peter Boncz wrote:
Update of /cvsroot/monetdb/pathfinder/runtime In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv2937
Modified Files: pathfinder.mx pf_support.mx Log Message: - BUG 1995879: only use algebra for normal queries
cachable queries and the XRPC server still use mps
Peter,
just for clarification and the records: am I right that this means that now all XQuery modules are compiled with MPS, but when a (non-XRPC?) query uses an XQuery module, this XQuery itself is compiled with ALG? Or are such queries compiled with MPS, too?
Thanks in advance!
Stefan
U pathfinder.mx Index: pathfinder.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pathfinder.mx,v retrieving revision 1.421 retrieving revision 1.422 diff -u -d -r1.421 -r1.422 --- pathfinder.mx 16 Jun 2008 23:38:41 -0000 1.421 +++ pathfinder.mx 17 Jun 2008 17:07:26 -0000 1.422 @@ -4246,7 +4246,7 @@ * - execute MIL buffer (parse & execute) * * int - * xquery_compile_exec(xquery_client *ctx, char* xquery, int is_url, + * xquery_compile_exec(xquery_client *ctx, int options, char* xquery, int
is_url,
* char** prologue, char** query, char** epilogue, char* nsurl) * - translate xquery to MIL and execute * @@ -4345,6 +4345,7 @@ extern char* PFmaxstack; static int xquery_compile_exec(xquery_client *ctx, + int options, char* url, int is_url, char** prologue, @@ -4356,22 +4357,9 @@ char *xquery = url; char *err = NULL;
- int options = 0; /* Setting the StandOff flag based on runtime settings */ if ((GDKgetenv("standoff") != NULL) && (strcmp(GDKgetenv("standoff"),"enabled") == 0)) options |= COMPILE_OPTION_STANDOFF; - /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ -#if MILPRINT_SUMMER_IS_DEFAULT - if (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) - options |= COMPILE_OPTION_ALGEBRA; -#else /* ALGEBRA_IS_DEFAULT */ - if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ - (((ctx->mode & XQ_ALGEBRA) != 0) || \ - (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) - options |= COMPILE_OPTION_ALGEBRA; -#endif
MT_set_lock(pf_compiler_lock, "xquery_compile_exec"); if (is_url) { @@ -4910,7 +4898,7 @@ strcpy(mod->url, url); mod->nsurl = mod->url + url_len;
- ret = xquery_compile_exec(ctx, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); + ret = xquery_compile_exec(ctx, 0, url, 1, &mod->prologue, NULL, &mod->epilogue, mod->nsurl); if (!ret) { xquery_module_free(mod); return NULL; @@ -5745,9 +5733,22 @@ char *sec1 = NULL; char *sec2 = NULL; char *sec3 = NULL; + int options = 0; /* compile and execute the query (minus module imports) */ err = xquery_nondescriptive_error; - if (xquery_compile_exec(ctx, query, 0, &sec1, &sec2, &sec3, NULL)) + /* Setting the Algebra flag; server setting overules compile-time default; client choice overrules server setting */ +#if MILPRINT_SUMMER_IS_DEFAULT + if (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (GDKgetenv("xquery_backend") != NULL) && (strcmp(GDKgetenv("xquery_backend"),"algebra") == 0))) + options = COMPILE_OPTION_ALGEBRA; +#else /* ALGEBRA_IS_DEFAULT */ + if (((ctx->mode & XQ_MILPRINT_SUMMER) == 0) && \ + (((ctx->mode & XQ_ALGEBRA) != 0) || \ + (GDKgetenv("xquery_backend") == NULL) || (strcmp(GDKgetenv("xquery_backend"),"milprint_summer") != 0))) + options = COMPILE_OPTION_ALGEBRA; +#endif + if (xquery_compile_exec(ctx, options, query, 0, &sec1, &sec2, &sec3, NULL)) err = NULL; if (sec1) free(sec1); if (sec2) free(sec2);
U pf_support.mx Index: pf_support.mx =================================================================== RCS file: /cvsroot/monetdb/pathfinder/runtime/pf_support.mx,v retrieving revision 1.301 retrieving revision 1.302 diff -u -d -r1.301 -r1.302 --- pf_support.mx 10 Jun 2008 09:22:17 -0000 1.301 +++ pf_support.mx 17 Jun 2008 17:07:27 -0000 1.302 @@ -8373,7 +8373,7 @@ BUNins(*res, &docpre, &docid[j], FALSE);
/* skip all nodes of this document */ - docpre = docpre + size[docpre]; + docpre = docpre + size[docpre] + 1; for (delta = (n-i) >> 4; delta > 40; delta >>= 4) while (i+delta < n && * (oid *) BUNtail(inputi, i+delta) <= docpre) i += delta;
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-pf-checkins mailing list Monetdb-pf-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
participants (2)
-
Peter Boncz -
Stefan Manegold