On Thu, May 27, 2010 at 04:47:54PM +0200, Peter Boncz wrote:
Hi Fabian, Hering,
Would it not be possible to COPY INTO a file? Performance of that will most likely be higher than going through Mapi, though it carries the disadvantages that the file will be available on the server computer only and it is still converting to ascii (and your app will then parse it back, most likely).
Indeed that is possible; cf., "3.1.6.5 Copy into File" at http://monetdb.cwi.nl/SQL/Documentation/Data-Manipulation.html#Data-Manipula...
Otherwise, as Fabian hints, if you want insane speed, then yes you go into shady kernel regions.. you could select into a temptable, and then obtain the BAT filenames. If you are just interested in fixed-size numeric types (32- or 64-bytes signed integers or 32- or 64-byte doubles) than this is simple as the .tail file just contains one column of that data, as a raw one-dimensional array. Strings are less obvious.
It might be a bit scary to read directly from the data files in MonetDB's dbfarm while they are under the controle of MonetDB (less scary if there are no concurrent updates on the data while the external program is reading the files), but it is indeed possible (surely on operating-/file-systems that allow multiple processes to open the same file concurrently ;-)). As a side note, while not (yet?) available in open source MonetDB, you also might find our research papers on Stream Processing with MonetDB interesting: http://repository.cwi.nl/search/fullrecord.php?publnr=14291 http://repository.cwi.nl/search/fullrecord.php?publnr=14315 http://repository.cwi.nl/search/fullrecord.php?publnr=12229 As said, this is still very much on-going research and far from "production ready". Stefan
Peter
-----Original Message----- From: Fabian Groffen [mailto:Fabian.Groffen@cwi.nl] Sent: Thursday, May 27, 2010 4:41 PM To: monetdb-users@lists.sourceforge.net Subject: Re: [MonetDB-users] Bulk Data Extraction & Embedded mserver5
Thank you, Stefan, for the advice.
I apologize if I gave the impression that I am using MonetDB as a mere data store. I do realize that one should leverage analytical capabilities of
DBMS engine as much as possible before getting the results out. Part of my system will indeed make extensive use of MonetDB's amazing query capability.
Unfortunately, another part of my application deals with what is known as "complex event processing" (CEP), the most fundamental building block is
On 27-05-2010 07:26:16 -0700, Hering Cheng wrote: the the
use of fixed or sliding time windows. An example would be to calculate averages of a column with a 5-minute window. Needless to say, the data are time series. The trick is to meld CEP logic with MonetDB. Ideally, I would enhance MonetDB with this capability, but I deem it to be beyond my own ability. Thus my question about how to feed data from MonetDB into my logic in bulk.
The problem very much is that the speed you see now roughly what you're going to get at max. The upcoming Jun2010 release will give you some improvements once you connect to the database from the same machine using a C-program based on Mapilib or just around mclient, but in fact, our protocol of sending over the data is all but optimised for speed in transmission of large volumes.
It's very hard to get by this, and you will quickly enter deep levels down in the engine to reach BATs and try to serialise them if it's all about speed. But perhaps it is a better approach to get your CEP queries translated into something that is helped by user defined functions either in MAL or maybe even C if you really have only a few operations that you need to be performed very fast.
--------------------------------------------------------------------------- ---
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4199 |