[MonetDB-users] time series data scaling
In a large scale time series row-store DB (Oracle), I'm accustomed to partitioning fact tables on time. This has has a couple of benefits: - since queries always include predicates on the time field, partition elimination can help a lot - when the time based partitions map one-to-one with tablespaces, removing old data is extremely fast. (when you've reached the maximum data retention period, you need to remove data as fast as you can load it). Are there any features in monetdb that would facilitate these needs (or ameliorate them)? TIA, -david
On 12/19/11 11:20 PM, David Pennell wrote:
In a large scale time series row-store DB (Oracle), I'm accustomed to partitioning fact tables on time. This has has a couple of benefits: Dear David,
If the table is sorted on the time dimension then MonetDB will already automagically use horizontal column partitioning (at no cost) to speed up queries in a multicore setting.
- since queries always include predicates on the time field, partition elimination can help a lot - when the time based partitions map one-to-one with tablespaces, removing old data is extremely fast. (when you've reached the maximum data retention period, you need to remove data as fast as you can load it).
That feature will most likely be released in Q1. It can be mimicked easily using base tables glued together with a view definition.
Are there any features in monetdb that would facilitate these needs (or ameliorate them)?
regards, Martin Kersten
TIA, -david
------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
On Mon, Dec 19, 2011 at 4:55 PM, Martin Kersten
In a large scale time series row-store DB (Oracle), I'm accustomed to
On 12/19/11 11:20 PM, David Pennell wrote: partitioning fact tables on time. This has has a couple of benefits: Dear David,
If the table is sorted on the time dimension then MonetDB will already automagically use horizontal column partitioning (at no cost) to speed up queries in a multicore setting.
I assume you are talking about mitosis. From what I've been able to read (lots of publications, scanned comments in most of code base), this appears to be something that happens at query time. I've assumed that you simply map BAT's to ranges of the column file but this is certainly a WAG. Where is the best place to look for more details?
- since queries always include predicates on the time field, partition elimination can help a lot - when the time based partitions map one-to-one with tablespaces, removing old data is extremely fast. (when you've reached the maximum data retention period, you need to remove data as fast as you can load it).
That feature will most likely be released in Q1. It can be mimicked easily using base tables glued together with a view definition.
Which feature - partition elimination or the ability to rapidly delete entire partions of data? Is there anything written? I'm happy to look at code, if you can point me to the right area.
Are there any features in monetdb that would facilitate these needs (or ameliorate them)?
regards, Martin Kersten
TIA, -david
------------------------------------------------------------------------------
Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
------------------------------------------------------------------------------ Write once. Port to many. Get the SDK and tools to simplify cross-platform app development. Create new or port existing apps to sell to consumers worldwide. Explore the Intel AppUpSM program developer opportunity. appdeveloper.intel.com/join http://p.sf.net/sfu/intel-appdev _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
participants (2)
-
David Pennell -
Martin Kersten