Re: [MonetDB-users] speed comparison off inserts and deletes, and db size question

Quartz wrote:

--- Fabian wrote:

...

...

We have need for a 300 million recors DB of about 500 GB. We insert about 1000 records per seconds. or about 100 millions record per day, for 3 days.

wow Can you tell us which database on what OS on which hardware this runs?

Hehehe. We have needs for it... It doesn't exist yet. We have a dual amd opteron, linux 32 bits for now, 120 GB partition in a striped raid, 4 gigs of ram. Mysql 4.1.10 with InnoDB. The secret is in the extended insert syntax of mysql. Magnitude faster than jdbc batches.

That is the 'raw' mode I was talking about we also have. It has around the same orders of magnitude of speed improvement in comparison to JDBC batching (even with PreparedStatements).

...

...

7-Do you think MonetDB is at all a good choice for our high-thoughput highly volatile data?

I think you would seriously need horizontal partitioning which will be available in the next generation MonetDB kernel, to get around some problems both memory and delete wise.

If it cannot be supported transparently, I won't pursue it. We already have another prototype with multi-table partitionning. It is a bit slower on queries though.

The idea of the partitions in the next generation of MonetDB is to support transparent access to a pseudo BAT, which consists of multiple BATs. I think there is still some work to do on this.