On 27-05-2010 07:26:16 -0700, Hering Cheng wrote:
Thank you, Stefan, for the advice.
I apologize if I gave the impression that I am using MonetDB as a mere data store. I do realize that one should leverage analytical capabilities of the DBMS engine as much as possible before getting the results out. Part of my system will indeed make extensive use of MonetDB's amazing query capability.
Unfortunately, another part of my application deals with what is known as "complex event processing" (CEP), the most fundamental building block is the use of fixed or sliding time windows. An example would be to calculate averages of a column with a 5-minute window. Needless to say, the data are time series. The trick is to meld CEP logic with MonetDB. Ideally, I would enhance MonetDB with this capability, but I deem it to be beyond my own ability. Thus my question about how to feed data from MonetDB into my logic in bulk.
The problem very much is that the speed you see now roughly what you're going to get at max. The upcoming Jun2010 release will give you some improvements once you connect to the database from the same machine using a C-program based on Mapilib or just around mclient, but in fact, our protocol of sending over the data is all but optimised for speed in transmission of large volumes. It's very hard to get by this, and you will quickly enter deep levels down in the engine to reach BATs and try to serialise them if it's all about speed. But perhaps it is a better approach to get your CEP queries translated into something that is helped by user defined functions either in MAL or maybe even C if you really have only a few operations that you need to be performed very fast.