Thomas Backlund wrote:
Hi,
Hi Thomas. Thanks for your interest in MonetDB.
I stumbled over MonetDB a few days ago and it really cought my interest. But it is hard to find information about it, so I hope You could answer a few questions.
I need a database engine that could handle many rows, about 350*10^9 rows of about 1000TB data.
This is really large, do you really mean 1 PetaByte? For MonetDB the first interesting part is that it will automatically split the table into 350 tables of 1000.000.000 rows, each would then be be 3 TB.. which means you are using a special column type? I think we need a snippet of the relation declaration to understand your schema We are currently working on automatic horizontal fragmentation with distribution to indeed handle them as single tables. To understand if MonetDB is the solution to your problem, it is also relevant to know what kind of queries you intend to run. We know there are users moving MonetDB into this direction (sizewise) and we would certainly be interested in learning the problems encountered. Given, of course, you have the time to help us on this track and you know of any database system that can already solve the problem out of the box. regards, Martin Kersten
Well I dont really need it right now, but I want to be prepared.
My option up til now has been to use MySQL and to do some kind of sharding using alot of databases. But that complicates stuff alot since I still want to do selects in this gigantic table and that will be a bit of a problem doing selects if the data is split up in 8000 different MySQL databases.
Do You think MonetDB could handle this in one single table? Even if MonetDB could handle this I still will need a hard disk of 1000TB, do You know of any technique merging many computers together into one virtiualized hard drive that MonetDB would accept?
Thank You very much!
Best regards Thomas Backlund, Sweden.