[Monetdb-developers] Re: X100
Hi Joel, About X100. In my opinion this is a very cool project that actually constitutes an entirely new query engine. It is more scalable than the current Monet4 engine, and also than the Monet5 engine. There is no main memory limitation anymore. Let's think about it as Monet6 (though that is not sure). The system is quite functional, but especially regarding APIs, database updates and also binary/source distribution of the code, a lot has to be done. Currently, we are concentrating on obtaining scientific results. Some functioanlities (like database updates) still have to be developed. I think we are more than a year off from a releasable system. Also, it has not been decided yet whether that will be an open-source release. Maybe X100 has some commercial future.. Then more general. I just read your posts on this mailing list and I do not have a full grasp of your plans in order to properly advise you. Some general tips: - try to keep all bulk operations in MIL, with interfaces to the lisp engine that move only little data in and out. - use unary bats preferable. i/e/ those that have a void head type - try to seperate data and metadata. Thus, the schema you use, and also the results, are best kept outside MonetDB. Maybe store them in flat files XML or something. It makes the architecture easier to maintain. Also, your application does not depend that strongly on MonetDB repositories. As for special purpose indexing, often it is possible to represent them as a set of bats. you will have to do some coding in a new monet extension module (a .mx file as found in the src/modules/plain/ directory) to write some primitives that access those bats (that are your index) efficiently. just my 2 cts. Peter
Peter, I would actually like reimplement the MIL primitives in Lisp. I cannot implement trading systems in MIL and writing a Lisp kernel on top of the GDK would let me run trading systems in the engine, without the overhead of pushing data around through ODBC, MAPI, etc. I don't want to be limited by main memory so I guess I'm up against implementing X100 but on a far smaller scale :-). There's still a lot to be reused from the GDK since it implements the very basic primitives (join, aggregate, select, etc.). From what I'm reading X100 seems to use fragmented BATs to avoid memory limitations which makes me think that I could do the same. My basic goal is to store market data and process it efficiently. Most of the storing is appending to BATs where the [void,timestamp] bat is sorted by timestamp (inverted?). Most of the processing is running trading strategies over one or many BATs at the same time (e.g. price, volume of MSFT and IBM), using a window of X BUNs and sliding that window forward one BUN at a time. Finally, I would like to do similarity matching of the different time series. That's icing on the cake and I can figure out how to do it later. There's a lot of similarity between what MonetDB is doing and what I need to do, I'm just working with Lisp. What do you think? Thanks, Joel On Jul 30, 2005, at 8:39 PM, Peter Boncz wrote:
Then more general. I just read your posts on this mailing list and I do not have a full grasp of your plans in order to properly advise you.
participants (2)
-
Joel Reymont
-
Peter Boncz