- developers-list - monetdb.org

[Monetdb-developers] Compiling buildtools.
by Romulo Goncalves 20 Oct '06

20 Oct '06

I update my buildtools,bootstraap, configure, make and I got this error: /ufs/goncalve/scratch/MonetDB/buildtools/build/_MX/Mx -x C /ufs/goncalve/MonetDB/buildtools/mel/object.mx /ufs/goncalve/MonetDB/buildtools/mel/object.mx: ./object.cc - created if g++ -DHAVE_CONFIG_H -I. -I/ufs/goncalve/MonetDB/buildtools/mel -I. -g -MT object.o -MD -MP -MF ".deps/object.Tpo" -c -o object.o object.cc; \ then mv -f ".deps/object.Tpo" ".deps/object.Po"; else rm -f ".deps/object.Tpo"; exit 1; fi g++: Internal error: Segmentation fault (program cc1plus) Please submit a full bug report. See <URL:http://bugzilla.redhat.com/bugzilla> for instructions. make[2]: *** [object.o] Error 1 make[2]: Leaving directory `/net/amelia/export/scratch1/goncalve/MonetDB/buildtools/build/_MEL' make[1]: *** [all] Error 2 make[1]: Leaving directory `/net/amelia/export/scratch1/goncalve/MonetDB/buildtools/build/_MEL' make: *** [all] Error 2 [goncalve@amelia scripts]$ Any idea? Regards, Romulo

2 1

[Monetdb-developers] Planning the next MonetDB release
by Stefan Manegold 15 Oct '06

15 Oct '06

Dear all, with the recent deadlines passed and the next(?) one (SIGMOD) five weeks ahead, it might be a good time to assess the current status of developments around MonetDB to make more concrete plans for the next release. I guess, we all agree that the next release should be *the* MonetDB/XQuery Update release, i.e., providing an implementation of the W3C XQuery Update Facility (http://www.w3.org/TR/xqupdate/) for MonetDB/XQuery. Hence, the major focus is obviously on finishing this implementation. See details below under "XQuery Updates". Concerning MQX/PF, there are three more subjects/areas where we need to see to which extend they should be officially announced with the release (and hence, need to be "finished" before the release): - algebra path (TUM: JanR, Jens) - recursions (TUM: JanR, Jens) - XRpc (CWI: Jennie, Peter) Further, the 64-bit Windows port of MQX (i.e., pathfinder) is still open. Concerning MonetDB in general, we need to see to which extend we want/need to improve the compilation process before the release: - clean-up configure scripts - clean-up usage of *-config.h include files - "meta scripts" to simplify the cvs checkout/update and build process Additionally, Niels & Fabian probably also have a todo list for MonetDB/SQL. Finally, all parts require some thorough testing --- especially all newly added features, in particular the XQuery update & transaction features and the new multi-document support --- simply using it (like with the recently added (multi-doc) benchmark tests) is a good start... Also, there are 50(!) open bug reports, and for many (most?) bugs that have been filed or closed since 01.01.2006, no test script has been provided, yet, to monitor their status. Lots of work to do... As said, this is to trigger an assessment of the current state and a planning for the release, hence, please feel free to comment and add your points. I suggest to discuss this in (or after) "MADAM" tomorrow and in the PF skype conference on Tuesday. Stefan XQuery Updates ============== As far as I'm aware, the major missing features are transaction support, mainly logging & recovery. Undoubtedly, this is a CWI task. Apart from that, we need to finish/extend the implementation of document management within MonetDB/XQuery --- both in MIL & XQuery. My vision is that we provide comparable (if not equal) functionality in both MIL & XQuery, providing functions beyond the XQuery standard where necessary. The envisaged functionally should cover the following: - access a single document by name (or URI) XQ!: fn:doc($uri as xs:string?) as document-node()? Done. - access all documents in a collection XQ!: fn:collection($arg as xs:string?) as node()* ToDo! - shred a single document into a new "singleton" collection: MIL: shred_doc(str URI [,str doc-name [,str col-name]] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?doc-name:URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. XQ!: fn:doc($uri as xs:string?) as document-node()? (implies doc-name = URI, col-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. - shred a set of documents into a single new collection: MIL: shred_doc(BAT[void,str] URI [,BAT[void,str] doc-name [,str col-name]] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?first-doc-name:first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ!: loop-lifted fn:doc($uri as xs:string?) as document-node()? (implies doc-name = URI, col-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) Done. XQ?: fn:doc($uri as xs:string*) as document-node()? (implies doc-name = URI, col-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo? - shred a set of documents into a set of new collections: MIL: shred_doc(BAT[void,str] URI [,BAT[void,str] doc-name [,BAT[void,str] col-name]] [,BAT[void,lng] freeSlotsPerPage]) : void (defaults: doc-name = URI, col-name = doc-name?doc-name:URI, freeSlotsPerPage = 0 (i.e., read-only)) (constraint: freeSlotsPerPage must be identical for all documents within a collection) Done. XQ?: loop-lifted fn:doc($uri as xs:string*) as document-node()? (implies doc-name = URI, col-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo? - shred a single document and add it to an existing collection: MIL: add_doc(str URI, str col-name [,str doc-name] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: fn:add_doc($uri as xs:string, $coll as xs:string) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - shred a set of documents and add it to a single existing collection: MIL: add_doc(BAT[void,str] URI, str col-name [,BAT[void,str] doc-name] [,lng freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: loop-lifted fn:add_doc($uri as xs:string, $coll as xs:string) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! XQ?: fn:add_doc($uri as xs:string*, $coll as xs:string) as document-node()? (implies doc-name = first-URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - shred a set of documents and add it to a set of existing collections: MIL: add_doc(BAT[void,str] URI, BAT[void,str] col-name [,BAT[void,str] doc-name] [,BAT[void,lng] freeSlotsPerPage]) : void (defaults: doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) (constraint: freeSlotsPerPage must be identical for all documents within a collection) ToDo! XQ?: fn:add_doc($uri as xs:string*, $coll as xs:string*) as document-node()? (implies doc-name = URI, freeSlotsPerPage = 0 (i.e., read-only)) ToDo! - delete a single document by name (and the whole collection, if this was the last document in that collection) MIL: delete_doc(str doc-name) : void Done. XQ?: fn:del_doc($uri as xs:string) as ??? ToDo! - dito for a set of documents? - delete a collection MIL: delete_col(str col-name) : void ToDo! XQ?: fn:del_col($col as xs:string) as ??? ToDo! - dito for a set of collections? - delete all cached/persistent documents MIL: delete_all_docs(bit cachedOnly) : void Done. XQ?: fn:del_all_docs($cachedOnly as xs:boolean) as ??? ToDo! - view all explicitly shredded and/or cached documents and collections (at least their names, maybe more info) in the database/cache, a la "xmlcache_print()" & "xmldb_print()" - view/list all documents (names) in a collection - manage cache rules Anything more? -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] shredding loop-lifted fn:doc() into one collection
by Stefan Manegold 15 Oct '06

15 Oct '06

Peter, our fix of Friday night to shred all documents of a loop-lifted fn:doc() call into one single collection (as opposed to creating one collection per document) --- not checked in, yet; see attached "pathfinder.mx.LLdoc.PATCH." for details --- seems to work fine (at least correct; speed/performance needs to be analyzed in detail) for file-URLs, i.e., documents that get indeed cached persistently). However, for non-file URLs (i.e., document that do get cached and hence stay trnasient), it did/does not work, basically as the docuemnts are shredded into the "transient" collection, and are (hence?) not assigned unique document IDs --- the document do not end up in the doc_* BATs. A HACK(?) to use the (unified) temporary document-/collection-ID for all documents (see attached "pathfinder.mx.LLdoc.FIX.") seems to work --- however, I have no idea, whether this is what was indended... More "live" on Monday. Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] Problems with SQL module...
by Romulo Goncalves 12 Oct '06

12 Oct '06

Dear all, I founded some problems in SQL module. Maybe they are not problems just a misunderstood. To see the problems in your computer just start a mserver5, load sql module, and sql.start(); Then start a mclient -lsql. First Problem: Execute this two queries in the mclient: CREATE FUNCTION zero(id int) RETURNS int BEGIN IF id > 0 THEN RETURN 1; ELSE RETURN 0; END IF; END; CREATE FUNCTION TEST() RETURNS BOOLEAN BEGIN DECLARE x int; SET x = zero(1); IF x > 0 THEN RETURN TRUE; ELSE RETURN FALSE; END IF; END; UNTIL now you should not find any problems. Lets see the first problem: In the mclient execute: SELECT test(); I got this error: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 1098918240 (LWP 14589)] 0x00002aaaaaaba2c1 in idcmp (n=0x2aaab485227b "destroy", m=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/mal/mal_type.mx:251 251 if (* n == * m) (gdb) up #1 0x00002aaab483ab94 in isUpdated (mb=0xf59548, pc=9) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_support.mx:885 885 if (idcmp("destroy", getFunctionId(q)) = = 0) (gdb) up #2 0x00002aaab483f287 in OPTcommonTermsImplementation (mb=0xf59548, stk=0x0, pci=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_commonTerms.mx:17 6 176 if (hasSameSignature(p, q) && (gdb) up #3 0x00002aaab483f718 in OPTcommonTerms (mb=0xf59548, stk=0x0, p=0x0) at /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimizer/opt_commonTerms.mx:24 9 Line number 249 out of range; /ufs/goncalve/MonetDB/MonetDB5/monet5/src/optimize r/opt_commonTerms.mx has 220 lines. (gdb) up #4 0x00002aaab4efb73a in addQueryToCache (c=0x503a10) at /ufs/goncalve/MonetDB/SQL/sql/src/backends/monet5/sql_optimizer.mx:434 434 OPTcommonTerms(mb,0,0); /* common terms */ (gdb) up #5 0x00002aaab4efa84e in backend_create_func (be=0x10e1218, f=0x10e80b8) at /ufs/goncalve/MonetDB/SQL/sql/src/backends/monet5/sql_gencode.mx:1938 1938 addQueryToCache(c); If you want all the details, run your mserver with gdb (I know that you know the procedures for debugging). ;) Second problem: If I restart the mclient and restart the mserver: The mclient can see the functions in the table sys.functions, but the client started in the server by sql.start() does see the functions created. Third problem: Just execute the following query: select * from functions where name like 'hash'; Why they are so many rows with the same information for the function hash? Fourth problem: Just create the following function: CREATE FUNCTION niels(id_val int) RETURNS TABLE (id int) BEGIN DECLARE TABLE roro(id int); RETURN TABLE (SELECT id from functions where id = id_val order by id); END; I will try to correct this bugs, but until there I will appreciate some help. This problems are delaying the translation of Skyserver functions and also the my implementation of Dependency Control table. Best Regards, Romulo Goncalves

3 2

[Monetdb-developers] re-organizing PF/MXQ tests
by Stefan Manegold 12 Oct '06

12 Oct '06

Dear all, before adding another ~80 tests (all 4 flavors of the XBench benchmark suite) to the current pathfinder test set of 1384 tests, I'd like to propose a re-organization/re-structuring of the pathfinder test set. With all tests enabled, testing takes almost 30 minutes --- with an optimized 64-bit binary with 32-bit OIDs on my Athlon64 X2 3800+ (2GHz) with 2GB RAM. A quick analysis clusters the tests in 5 categories: 1) compiler ("pf" only) tests 2) tests for features provided by the MXQ runtime 3) tests for additional/optional modules (PFtijah) 4) tests derived from bug reports, use cases, etc. 5) benchmark tests Currently, the respective tests reside in the follwoing directory trees: 1) compiler// 2) runtime// 3) modules// 4) tests// 5) tests// This allows to conveniently run each category individually (e.g., `Mtest -r compiler/`), except from the "tests/" tree, which contains two categories: 4 & 5. I propose to replace 5) tests// by 5) benchmarks// i.e., add "benchmarks/" and move all benchmark tests from tests// to benchmarks//, basically: mv tests/{MBench,X007,XBench,XMach-1,XMark,XPath,XPathMark} benchmarks/ Additionally, I'd suggest to also move the tests/XRpc/xmark_queries to benchmarks/XMark/XRpc Open question: should we then also move tests/Music to benchmarks/Music ? Votes? Well, as you might imagine, I'v everything ready here for checkin ;-) Hence, unless there are complains or different suggestions by Saturday (Oct 14 2006) morning, I'll apply above changes during the weekend. Kind regards, Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[Monetdb-developers] [Monetdb-sql-checkins] Problems with SQL installation.
by Stefan Manegold 10 Oct '06

10 Oct '06

Romulo, this should have been either filed as a bug report or sent to Monetdb-developers(a)lists.sourceforge.net. On Tue, Oct 10, 2006 at 05:19:32PM +0200, Romulo Goncalves wrote: > Dear SQL developers, > > I checked out the buildtools and I installed it. > > > I checked out MonetDB4 v4.13.1 and I install with this configure: > configure --prefix=/ufs/goncalve/scratch/MonetDB/monetdb4 --enable-debug > --enable-bits=64 At this point: is /ufs/goncalve/scratch/MonetDB/monetdb4/bin in your PATH? > I checked out MonetDB5 version v4.99.19 and I installed it with the > following configure: > configure --prefix=/ufs/goncalve/scratch/MonetDB/monetdb5 --enable-debug > --enable-bits=64 > > > Then I checked out SQL module and I installed it with the following > configure: At this point: is /ufs/goncalve/scratch/MonetDB/monetdb5/bin in your PATH? > configure --prefix=/ufs/goncalve/scratch/MonetDB/monetdb5/ > --enable-debug --enable-bits=64 > > > For all the installations the directories were removed and the > bootstraap was executed. > > The error that I got in the installation of SQL is: > > ve/scratch/MonetDB/monetdb4/lib/libstream.so -lpthread -m64 -Wl,-soname > -Wl,lib _sql.so.0 -o .libs/lib_sql.so.0.0.0 > /usr/bin/ld: cannot find -loptimizer > collect2: ld returned 1 exit status > make[7]: *** [lib_sql.la] Error 1 > make[7]: Leaving directory > `/net/amelia/export/scratch1/goncalve/SQL/sql/build/s rc/backends/monet5' > > > This problem started after the following update: > > [Monetdb-sql-checkins] sql/src/backends/monet5 Makefile.ag, 1.30, 1.31 > > Update of /cvsroot/monetdb/sql/src/backends/monet5 > In directory > sc8-pr-cvs7.sourceforge.net:/tmp/cvs-serv6378/src/backends/monet5 > > Modified Files: > Makefile.ag > Log Message: > Added some libraries that are used by the module. did you re-compiler Monet5 & SQL from scratch (bootstrap; configure; make; make install) after this update? > Could you tell me where I am wrong? > > Best regards, > Romulo Goncalves Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

3 4

[Monetdb-developers] Linking against a library
by Wouter Scherphof 09 Oct '06

09 Oct '06

Hi, My first module (called "geom"), created from "template", compiles (on Windows XP with the "native tools") and the hello() function works. Now I want to use some functions of a certain library (the GEOS<http://geos.refractions.net>C API). I'm fairly new to C, so probably I'm doing something obvious wrong. What I have is a header file "geos_c.h<http://geos.refractions.net/ro/doxygen_docs/html/geos__c_8h-source.html>" and a DLL "geos_c_fw.dll" (I chose to use a prebuilt DLL for now, since I didn't manage to build the GEOS library as well :-(.) I put those in the NT directory. In my geom.mx, I added: *#include* <geos_c.h>* **int* testGeos( ){* * initGEOS(*NULL*,*NULL*);* * finishGEOS();* ** return* GDK_SUCCEED;* *} Now, when I run "nmake NEED_MX=1", I get: (...) cl -GF -W3 -wd4273 -wd4102 -MD -nologo -Zi -Ox -I. -I.\.. "-IC:\Pthreads\include" -I.\..\..\src "-IC:\monetdb\MonetDB\NT\include" "-IC:\monetd b\MonetDB\NT\include\common" "-IC:\monetdb\MonetDB\NT\include\gdk" "-IC:\monetdb\MonetDB\NT\include\monet" "-IC:\monetdb\MonetDB\NT\include\plain" "-I C:\monetdb\MonetDB\NT\include\mapi" -DNDEBUG -DLIBGEOM -DCOMPILE_DL_GEOM -Fo"geom.obj" -c "geom.c" geom.c cl -GF -W3 -wd4273 -wd4102 -MD -nologo -Zi -Ox -I. -I.\.. "-IC:\Pthreads\include" -I.\..\..\src "-IC:\monetdb\MonetDB\NT\include" "-IC:\monetd b\MonetDB\NT\include\common" "-IC:\monetdb\MonetDB\NT\include\gdk" "-IC:\monetdb\MonetDB\NT\include\monet" "-IC:\monetdb\MonetDB\NT\include\plain" "-I C:\monetdb\MonetDB\NT\include\mapi" -DNDEBUG -LD -Felib_geom.dll geom.glue.obj geom.obj /link "/LIBPATH:C:\monetdb\MonetDB\NT\lib" "/LIBPATH:C:\monet db\MonetDB\NT\lib\MonetDB" libbat.lib libmonet.lib Creating library lib_geom.lib and object lib_geom.exp geom.obj : error LNK2019: unresolved external symbol _finishGEOS referenced in function _testGeos geom.obj : error LNK2019: unresolved external symbol _initGEOS referenced in function _testGeos lib_geom.dll : fatal error LNK1120: 2 unresolved externals So the compilation of geom.c succeeds, but the linking of lib_geom.dll fails. Apperently, the geos_c_fw.dll is not found. I tried putting it in one of the /LIBPATH, but that doesn't help. How can I make this work? Best regards, Wouter Scherphof. Utrecht Professional School, The Netherlands.

2 1

[Monetdb-developers] Crash leaves me unable to start mserver
by Joseph.Baird＠cox.com 09 Oct '06

09 Oct '06

After loading several million records via odbc, the server is suddenly unresponsive but can be restarted. When using jdbc, the server crashes, again after several million records, but cannot be restarted. Performance and usability are impressive with a million records or so but I cannot seem to load more than that without consistent problems. Is there any way to make the server - and ideally the database - usable again? Thank you in advance, Joe Baird

2 1

Re: [Monetdb-developers] Monetdb-developers Digest, Vol 5, Issue 6
by p.a.boncz 06 Oct '06

06 Oct '06

Hi all, > In PROC rpc_client() (runtime/xrpc.mx), I had changed > ws_addcoll(ws, ... > to > ws_opencoll(ws_id(ws), ... it should we ws_opencoll(wsid, ...) You have that, because in the query context we always have both variable 'ws' and 'wsid'. ws_id(ws) should be called once, it asks for a unique oid. The combination of (id,int(ws)) is squeezed into a lng, and that is wsid. As for doc_tbl(), what would work is to change its parameter to wsid; one can always get the ws with ws := ws_bat(wsid), this is done now in numerous places in pf_support.mx. Your conclusions (1), (2) and (3) are all correct. But, Jan's suggestion to store the wsid inside the ws can also work. However, I deem it very awkward to introduce a new BAT in the ws just to retain a number. Other suggestions are to rename the ws-bat to some unique key, and then we can use that name as transaction key (=wsid). We would change the wsid type then from lng to str. Another idea is to set the head-seqbase of the ws-bat to an oid. I think the ws is accessed with fetch() only, which disregards any seqbase. Note that internally in pathfinder.mx there are a number of functions that just pass around an artificial wsid, and in fact a ws does not exist at all (this is the MIL document management interface e.g. shred_doc), so a number of (internal) functions in pathfinder.mx will have to stay using a wsid in their signature. They do not need a ws-bat. But the exported functions ws_opencoll and ws_opendoc could indeed just use a ws as they did before. I will not have time for this until next week, so if anyone feels like adventurous, he may try. Peter -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 | ------------------------------ Message: 2 Date: Fri, 6 Oct 2006 12:43:21 +0200 From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> Subject: Re: [Monetdb-developers] The new "ws-API", Algebra & XRpc (& PFtijah) To: Monetdb-developers(a)lists.sourceforge.net Message-ID: <20061006104321.GA2821(a)corona.ins.cwi.nl> Content-Type: text/plain; charset=us-ascii On Thu, Oct 05, 2006 at 11:03:09AM +0200, Stefan Manegold wrote: > Dear Peter, dear fellow PF & MXQ developers, [...] > In cases of XRpc & Algebra, there are still places were wsid := ws_id(ws) is > IMHO called in the wrong location: > > XRpc: > In PROC rpc_client() (runtime/xrpc.mx), I had changed > ws_addcoll(ws, ... > to > ws_opencoll(ws_id(ws), ... > While this seems to worg for now with read-only documents (i.e., in the > absense of updates and concurrency), it is IMHO incorrect in general. > Rather, rpc_client should receive wsid as argument instead of ws; > if necessary, ws can then be derived via ws := ws_bat(wsid). I just changed the signatures of doLoopLiftedRPC() & rpc_client() to receive lng wsid instead of BAT ws, BAT ws is the locally derived via ws := ws_bat(wsid); All XRpc tests seem to work fine. Stefan [...] -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 | ------------------------------ Message: 3 Date: Fri, 06 Oct 2006 14:39:19 +0200 From: Jan Rittinger <rittinge(a)in.tum.de> Subject: Re: [Monetdb-developers] The new "ws-API", Algebra & XRpc (& PFtijah) To: Monetdb-developers(a)lists.sourceforge.net Message-ID: <45264E77.3060201(a)in.tum.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed If this wsid is a concept that is strictly necessary for the update support (and nobody except me thinks that this bit shifting might be not the nicest solution) we could also think about getting rid of the ws and replacing it with the wsid. I just do not like the fact that there are two variables underway that have some side effects on each other. So a fourth proposal (d) would be to replace the occurrences of ws by wsid. This would mean that *all* document accesses would have to include another indirection (even all path steps). On 10/05/2006 11:03 AM, Stefan Manegold wrote with possible deletions: > Dear Peter, dear fellow PF & MXQ developers, > > I'm not sure, whether I do understand the new "ws-API" correctly, and would > be grateful if someone of you could enlighten me. > > First, here's how far I get so far (please correct me if/where I'm wrong!), > followed by concrete open questions. > > For transaction support, Peter introduced a new wsid that uniquely > identifies (a reference to?) a ws, which is required for conflict detection > (as far as I understand). > > A wsid is to be generated from/for a ws by calling wsid := ws_id(ws); > however, this generated a new unique id each time is called (two calls on > the same ws yield two different ids, right?), hence, wsid := ws_id(ws) > should immediately follow ws := ws_create() . > > For the inverse, there is ws := ws_bat(wsid) that return the ws identified > by wsid; obviously, this function yield the same result each time it's > called with the same wsid. > > Since some functionality required to to the wsid instead of "only" the ws, > Peter had changes the signatures/API of some runtime PROCs. As far as I see > right now, these are mainly (only?) > ws_doc(ws, ... -> ws_opendoc(wsid, ... > ws_addcoll(ws, ... -> ws_opencoll(wsid, ... > ws_destroy(int(ws)) -> ws_destroy(wsid) > > I finished these changes by simply running the folloing one-liners: > > find * -type f | xargs grep -l 'var ws := ws_create();' | xargs perl -i -p -e 's|(var ws := ws_create;)|$1 var wsid := ws_id(ws);|' > find * -type f | xargs grep -l 'ws_doc(ws,' | xargs perl -i -p -e 's|ws_doc$ws,|ws_opendoc(wsid,|' > find * -type f | xargs grep -l 'ws_destroy(int(ws))' | xargs perl -i -p -e 's|ws_destroy\(int\(ws$\)|ws_destroy(wsid)|' > find * -type f | xargs grep -l 'ws_addcoll(ws,' | xargs perl -i -p -e 's|ws_addcoll\(ws,|ws_opencoll(wsid,|' > find * -type f | xargs grep -l 'ws_addcoll' | xargs perl -i -p -e 's|ws_addcoll|ws_opencoll|' > > > While this seemed to have been enough to fix all .milS tests and (most of) > the milprint_summer functionallity, there are two open issues left: > the Algebra version, XRpc & PFtijah. > > To be honest, I haven't looked at PFtijah, yet; some help by the respective > experst would be appreciated. > > > In cases of XRpc & Algebra, there are still places were wsid := ws_id(ws) is > IMHO called in the wrong location: > > XRpc: > In PROC rpc_client() (runtime/xrpc.mx), I had changed > ws_addcoll(ws, ... > to > ws_opencoll(ws_id(ws), ... > While this seems to worg for now with read-only documents (i.e., in the > absense of updates and concurrency), it is IMHO incorrect in general. > Rather, rpc_client should receive wsid as argument instead of ws; > if necessary, ws can then be derived via ws := ws_bat(wsid). > > Algebra: > in PROC doc_tbl() (runtime/pf_support.mx), I had changed > var r := ws_doc(ws, item); > into > var wsid := ws_id(ws); > var r := ws_opendoc(wsid, item); > Same story as above. > However, doc_tbl return ws also in its result BAT-of-BATs, as far as I > understand mainly for "canonical API" reasons. > Obviously, it is not possible to replace BAT ws by lng wsid, here... > > > Leaving the latter problem aside for the moment, the following could (have) > work(ed) as a general rule, and we should check and change the codebase > accordingly: > > 1) wsid := ws_id(ws) must only be called immediately after ws := ws_create() > > 2) all functions/PROCs that recursively (i.e., including all transitively > called functions/PROCs) require wsid instead of or inaddition to ws should > be modified in that the receive wsid instead of ws as an argument; > ws can then locally be derived via ws := ws_bat(wsid) if/where necessary. > > 3) all functions/PROCs that recursively (i.e., including all transitively > called functions/PROCs) "never" (at least not yet) require wsid, but are > fine with ws can stay unchanged, i.e., getting (only) ws as argument. > Obviously, wherever these functions need to maintain a signature/API > aligned with those that fall under point 2) above, we should also change > these functions as described above with 2). > > > As indicated above, this does not work for doc_tbl and its companion PROCs > in runtime/pf_support.mx that make-up the interface between the Algebra > version of the compiler and the runtime. > I assume the respective canonical API could be changed in that the > respective PROC get wsid instead of (or in addition to?) ws as arguments. > However, we cannot easily change the API to return wsid (lng) instead of ws > (BAT) in the result BAT-of-BATs. Here, I see three solutions --- the last > one actually comes from JanR: > > a) Is ws/wsid indeed required in the result? > If not, we could simply discard it. > > b) Wrap wsid temprarly into a ("fake") [void,lng] BAT containing only one > (nil,wsid) BUN. > (Ugly!) > > c) (JanR) Put the wsid inside ws --- basically as a ("fake") BAT [void,lng]. > Kind of the "encapsulated" solution --- not only for the algebra-runtime > interface! > Consequently, we should have ws_create() call ws_id(ws) internally and > stored the result in ws (this could save the seemingly "redundant" > ws := ws_create(); wsid := ws_id(ws); sequence), and all (most, except at > least ws_destroy()) signatures/API could be kept / re-unified to pass > (only) ws (now including wsid) instead of wsid as argument. > > There seems to be one problem, so, if I understand Peter's comment > correctly: > " > - ws-IDs are now lng-s (combination of *unique* ID and bat-ID) such IDs > are meaningful also after the query is done (and ws-bat deallocated). > ^1^1^1^1^1^1^1^1^1^1^1^1^1^1 ^2^2^2^2^2^2^2^2^2^2^2 > this is needed for trans mgmt. > All meta-bats witha ws-id field now hold lng instead of int. > " > > In other words, (^2) wsid needs to be available also if ws is gone. > I'm not sure, though, whether is has to be available in the global wsid > variable (i.e., inside/during the query), or (only) in some meta-bats > outside (and hence after) the query (-context) (as suggested by ^1). > > > Peter, > > could you please enlighten me/us, here? > > > In case "only" (^1) is required, the "encapsulated" solution seems indeed > to be suitable for all code. > Otherwise, is would at least be a local solution for the algebra-runtime > interface (in case (a) & (b0 are no option), but bears the burden of > maintaining consistency between the global wsid variable and the wsid stored > inside the ws. > > > I hope, this email and any reactions to it help to clearify the current > situation, rather than to blur it even more ... > > Stefan > > -- Jan Rittinger Database Systems Technische Universit?t M?nchen (Germany) http://www-db.in.tum.de/~rittinge/ ------------------------------ Message: 4 Date: Fri, 6 Oct 2006 15:08:00 +0200 From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> Subject: Re: [Monetdb-developers] The new "ws-API", Algebra & XRpc (& PFtijah) To: Jan Rittinger <rittinge(a)in.tum.de> Cc: Monetdb-developers(a)lists.sourceforge.net Message-ID: <20061006130800.GB5598(a)corona.ins.cwi.nl> Content-Type: text/plain; charset=us-ascii On Fri, Oct 06, 2006 at 02:39:19PM +0200, Jan Rittinger wrote: > If this wsid is a concept that is strictly necessary for the update > support (and nobody except me thinks that this bit shifting might be not > the nicest solution) we could also think about getting rid of the ws and > replacing it with the wsid. I just do not like the fact that there are > two variables underway that have some side effects on each other. To be honest, I'm still in the process of analyzing trying to understand, what all the flavors, pros, cons, and ideas behind having/using wsid are; hence, I cannot tell, whether it is "strictly necessary" or not. As far as I understand so far, the basic idea behind the "bit shifting" is merely to store both the id of (and hence reference to) BAT ws as well as a unique identifier --- that is still valid after a query/transaction (and hence cannot simply be the ws BAT id, which might be re-used for a subsequent query/transaction) --- in one single atomic value. Further, I agree, that having two related variables that need to be kept in sync and treated accordingly is not very handy for maintenance --- my guess is, that the current situation resulted from lack of time and fear to implement (something like) your (d) proposal; which I agree is "nicer" & "cleaner", but requires quite a lot of possibly error-prone code changes --- nevertheless, I might give it a try during the weekend ... Stefan > So a fourth proposal (d) would be to replace the occurrences of ws by > wsid. This would mean that *all* document accesses would have to include > another indirection (even all path steps). -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 | ------------------------------ Message: 5 Date: Fri, 6 Oct 2006 15:42:02 +0200 From: Stefan Manegold <Stefan.Manegold(a)cwi.nl> Subject: Re: [Monetdb-developers] The new "ws-API", Algebra & XRpc (& PFtijah) To: Monetdb-developers(a)lists.sourceforge.net Cc: Jan Rittinger <rittinge(a)in.tum.de> Message-ID: <20061006134202.GA6958(a)corona.ins.cwi.nl> Content-Type: text/plain; charset=us-ascii On Fri, Oct 06, 2006 at 03:08:00PM +0200, Stefan Manegold wrote: > On Fri, Oct 06, 2006 at 02:39:19PM +0200, Jan Rittinger wrote: > > If this wsid is a concept that is strictly necessary for the update > > support (and nobody except me thinks that this bit shifting might be not > > the nicest solution) we could also think about getting rid of the ws and > > replacing it with the wsid. I just do not like the fact that there are > > two variables underway that have some side effects on each other. > > To be honest, I'm still in the process of analyzing trying to understand, > what all the flavors, pros, cons, and ideas behind having/using wsid are; > hence, I cannot tell, whether it is "strictly necessary" or not. > > As far as I understand so far, the basic idea behind the "bit shifting" is > merely to store both the id of (and hence reference to) BAT ws as well as a > unique identifier --- that is still valid after a query/transaction (and hence > cannot simply be the ws BAT id, which might be re-used for a subsequent > query/transaction) --- in one single atomic value. > > Further, I agree, that having two related variables that need to be kept in > sync and treated accordingly is not very handy for maintenance --- my guess > is, that the current situation resulted from lack of time and fear to > implement (something like) your (d) proposal; which I agree is "nicer" & > "cleaner", but requires quite a lot of possibly error-prone code changes --- > nevertheless, I might give it a try during the weekend ... > > Stefan > > > So a fourth proposal (d) would be to replace the occurrences of ws by > > wsid. This would mean that *all* document accesses would have to include > > another indirection (even all path steps). Well, I just realize that this actually only helps to save us from having function interfaces that expect/use either ws or wsid or both --- it does not help to get rid of the global ws variable: to exist throught the whole query, the ws BAT must either be a global variable or be persistent --- the latter is not possible, since ws is a BAT-of-BATs, and those cannot be persistent... ... I just recall, that we have something like "session" BATs --- I'll check whether that's an option (any help/hint is of course welcome ...) Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 | ------------------------------ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV ------------------------------ _______________________________________________ Monetdb-developers mailing list Monetdb-developers(a)lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers End of Monetdb-developers Digest, Vol 5, Issue 6 ************************************************

2 3

[Monetdb-developers] The new "ws-API", Algebra & XRpc (& PFtijah)
by Stefan Manegold 06 Oct '06

06 Oct '06

Dear Peter, dear fellow PF & MXQ developers, I'm not sure, whether I do understand the new "ws-API" correctly, and would be grateful if someone of you could enlighten me. First, here's how far I get so far (please correct me if/where I'm wrong!), followed by concrete open questions. For transaction support, Peter introduced a new wsid that uniquely identifies (a reference to?) a ws, which is required for conflict detection (as far as I understand). A wsid is to be generated from/for a ws by calling wsid := ws_id(ws); however, this generated a new unique id each time is called (two calls on the same ws yield two different ids, right?), hence, wsid := ws_id(ws) should immediately follow ws := ws_create() . For the inverse, there is ws := ws_bat(wsid) that return the ws identified by wsid; obviously, this function yield the same result each time it's called with the same wsid. Since some functionality required to to the wsid instead of "only" the ws, Peter had changes the signatures/API of some runtime PROCs. As far as I see right now, these are mainly (only?) ws_doc(ws, ... -> ws_opendoc(wsid, ... ws_addcoll(ws, ... -> ws_opencoll(wsid, ... ws_destroy(int(ws)) -> ws_destroy(wsid) I finished these changes by simply running the folloing one-liners: find * -type f | xargs grep -l 'var ws := ws_create();' | xargs perl -i -p -e 's|(var ws := ws_create;)|$1 var wsid := ws_id(ws);|' find * -type f | xargs grep -l 'ws_doc(ws,' | xargs perl -i -p -e 's|ws_doc$ws,|ws_opendoc(wsid,|' find * -type f | xargs grep -l 'ws_destroy(int(ws))' | xargs perl -i -p -e 's|ws_destroy\(int\(ws$\)|ws_destroy(wsid)|' find * -type f | xargs grep -l 'ws_addcoll(ws,' | xargs perl -i -p -e 's|ws_addcoll\(ws,|ws_opencoll(wsid,|' find * -type f | xargs grep -l 'ws_addcoll' | xargs perl -i -p -e 's|ws_addcoll|ws_opencoll|' While this seemed to have been enough to fix all .milS tests and (most of) the milprint_summer functionallity, there are two open issues left: the Algebra version, XRpc & PFtijah. To be honest, I haven't looked at PFtijah, yet; some help by the respective experst would be appreciated. In cases of XRpc & Algebra, there are still places were wsid := ws_id(ws) is IMHO called in the wrong location: XRpc: In PROC rpc_client() (runtime/xrpc.mx), I had changed ws_addcoll(ws, ... to ws_opencoll(ws_id(ws), ... While this seems to worg for now with read-only documents (i.e., in the absense of updates and concurrency), it is IMHO incorrect in general. Rather, rpc_client should receive wsid as argument instead of ws; if necessary, ws can then be derived via ws := ws_bat(wsid). Algebra: in PROC doc_tbl() (runtime/pf_support.mx), I had changed var r := ws_doc(ws, item); into var wsid := ws_id(ws); var r := ws_opendoc(wsid, item); Same story as above. However, doc_tbl return ws also in its result BAT-of-BATs, as far as I understand mainly for "canonical API" reasons. Obviously, it is not possible to replace BAT ws by lng wsid, here... Leaving the latter problem aside for the moment, the following could (have) work(ed) as a general rule, and we should check and change the codebase accordingly: 1) wsid := ws_id(ws) must only be called immediately after ws := ws_create() 2) all functions/PROCs that recursively (i.e., including all transitively called functions/PROCs) require wsid instead of or inaddition to ws should be modified in that the receive wsid instead of ws as an argument; ws can then locally be derived via ws := ws_bat(wsid) if/where necessary. 3) all functions/PROCs that recursively (i.e., including all transitively called functions/PROCs) "never" (at least not yet) require wsid, but are fine with ws can stay unchanged, i.e., getting (only) ws as argument. Obviously, wherever these functions need to maintain a signature/API aligned with those that fall under point 2) above, we should also change these functions as described above with 2). As indicated above, this does not work for doc_tbl and its companion PROCs in runtime/pf_support.mx that make-up the interface between the Algebra version of the compiler and the runtime. I assume the respective canonical API could be changed in that the respective PROC get wsid instead of (or in addition to?) ws as arguments. However, we cannot easily change the API to return wsid (lng) instead of ws (BAT) in the result BAT-of-BATs. Here, I see three solutions --- the last one actually comes from JanR: a) Is ws/wsid indeed required in the result? If not, we could simply discard it. b) Wrap wsid temprarly into a ("fake") [void,lng] BAT containing only one (nil,wsid) BUN. (Ugly!) c) (JanR) Put the wsid inside ws --- basically as a ("fake") BAT [void,lng]. Kind of the "encapsulated" solution --- not only for the algebra-runtime interface! Consequently, we should have ws_create() call ws_id(ws) internally and stored the result in ws (this could save the seemingly "redundant" ws := ws_create(); wsid := ws_id(ws); sequence), and all (most, except at least ws_destroy()) signatures/API could be kept / re-unified to pass (only) ws (now including wsid) instead of wsid as argument. There seems to be one problem, so, if I understand Peter's comment correctly: " - ws-IDs are now lng-s (combination of *unique* ID and bat-ID) such IDs are meaningful also after the query is done (and ws-bat deallocated). ^1^1^1^1^1^1^1^1^1^1^1^1^1^1 ^2^2^2^2^2^2^2^2^2^2^2 this is needed for trans mgmt. All meta-bats witha ws-id field now hold lng instead of int. " In other words, (^2) wsid needs to be available also if ws is gone. I'm not sure, though, whether is has to be available in the global wsid variable (i.e., inside/during the query), or (only) in some meta-bats outside (and hence after) the query (-context) (as suggested by ^1). Peter, could you please enlighten me/us, here? In case "only" (^1) is required, the "encapsulated" solution seems indeed to be suitable for all code. Otherwise, is would at least be a local solution for the algebra-runtime interface (in case (a) & (b0 are no option), but bears the burden of maintaining consistency between the global wsid variable and the wsid stored inside the ws. I hope, this email and any reactions to it help to clearify the current situation, rather than to blur it even more ... Stefan -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

2 4