[Monetdb-developers] Calculate disk space for table?
Hi - how can I tell how much disk storage space on of my tables is using? I tried one rough way but not sure if it is accurate. Basically I did a linux du command and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data? There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way. Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods? And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making? Thank you!
Hi - how can I tell how much disk storage space on of my tables is using?
I tried one rough way but not sure if it is accurate. Basically I did a linux du command
Rt Ibmer wrote: this is the correct way until a fullfledged DBA tool is available or the necessary functions are packaged in a SQL module.
and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data?
bytes or blocks
There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way.
compression is not applied beyond ordinary byte alignments
Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods?
none
And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making?
MonetDB is targeted as a self organizing system. You can not even control the creation of indices and some recent additions (not in the to be released version) make common subexpression elimination automatic over long query sequences and at any level of profitability.
Thank you!
------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers
On Sun, Jun 08, 2008 at 11:38:12PM +0200, Martin Kersten wrote:
Rt Ibmer wrote:
Hi - how can I tell how much disk storage space on of my tables is using?
I tried one rough way but not sure if it is accurate. Basically I did a linux du command this is the correct way until a fullfledged DBA tool is available or the necessary functions are packaged in a SQL module. and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data?
bytes or blocks
MB = Megabytes = 1024 bytes. but: don't forget about the logs in .../sql-logs (next to .../dbfarm)/ MonetDB/SQL uses write-ahead-logging, i.e., until the logs are flushed (due to a server restart or after some time (AFAIK ca. 5 min right now) changes are kept in the WAL (here: sql-logs), before they are physically applied to the DBfarm. Stefan
There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way.
compression is not applied beyond ordinary byte alignments
Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods?
none
And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making?
MonetDB is targeted as a self organizing system. You can not even control the creation of indices and some recent additions (not in the to be released version) make common subexpression elimination automatic over long query sequences and at any level of profitability.
Thank you!
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
On Mon, Jun 09, 2008 at 12:34:58AM +0200, Stefan Manegold wrote:
On Sun, Jun 08, 2008 at 11:38:12PM +0200, Martin Kersten wrote:
Rt Ibmer wrote:
Hi - how can I tell how much disk storage space on of my tables is using?
I tried one rough way but not sure if it is accurate. Basically I did a linux du command this is the correct way until a fullfledged DBA tool is available or the necessary functions are packaged in a SQL module. and looked at the farmdb dir root, then imported about 2M rows, then looked a tthe du number again. But it only went up about 1.5MB for 2M rows of data?
bytes or blocks
MB = Megabytes = 1024 bytes.
... MB = megabytes = 1024 kilobytes = 1024 * 1024 bytes ... S.
but: don't forget about the logs in .../sql-logs (next to .../dbfarm)/ MonetDB/SQL uses write-ahead-logging, i.e., until the logs are flushed (due to a server restart or after some time (AFAIK ca. 5 min right now) changes are kept in the WAL (here: sql-logs), before they are physically applied to the DBfarm.
Stefan
There are about 150 columns but only about 30-40 columns for any given row are populated. Certainly this would be fantastic if the compression is really this good! Seems possible, but I wasn't sure if this technique was accurate and if there was a better way.
compression is not applied beyond ordinary byte alignments
Also on a related note - is there any detailed information on what types of compression and encoding MonetDB uses. For instance does it use Run Length Encoding or deltas or other such methods?
none
And are there query plans / optimizers of any sort that you can feed MonetDB and it will optimize the layout or create internal subtables to help speed the types of queries we will be making?
MonetDB is targeted as a self organizing system. You can not even control the creation of indices and some recent additions (not in the to be released version) make common subexpression elimination automatic over long query sequences and at any level of profitability.
Thank you!
-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |
participants (3)
-
Martin Kersten -
Rt Ibmer -
Stefan Manegold