The effect of the more compact storage only marginally affects direct performance (for the better). The big win is that it reduces RAM consumption when query processing. This means that where MonetDB performance suffers with 64-bits oids due to swapping when you use datasets that just do not fit your memory, the reduced storage of 32-bits oids may avoid avoid this problem. You can also say that you need less RAM to make MonetDB fast. Memory consumption may be reduced to 70% or even 50% in (rare) extreme cases. However, it will not work with columns with many tuples (2*1024*1024*1024 being the hard limit) or huge string columns (where all string data of a single column exceeds 32GB). -----Original Message----- From: dariuszs [mailto:dariuszs@svnp.com] Sent: Monday, June 22, 2009 7:52 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage Hi, That makes sense, now does the performance suffer with 64bit object ids, when you have to go thru more storage? Dariusz. Peter Boncz wrote:
since the last release, there is a storage/memory consumption optimization in place when using the configure option --enable-oid32, which causes tables to be restricted to 2Gtuples, but leads to smaller data storage for surrogate keys (oids). As a second effect, this configure setting also enables a more compact string heap organization (restricting the amount of string data in a single column to 32GB).
windows is configured by default with --enable-oid32
If you can live with these restrictions, you can also compile/get linux MonetDB binaries with --enable-oid32
-----Original Message----- From: Ying Zhang [mailto:Y.Zhang@cwi.nl] Sent: Monday, June 22, 2009 7:01 PM To: Communication channel for MonetDB users Subject: Re: [MonetDB-users] disk usage
Hello Dariusz,
The first thing came to my mind is the bits system. I guess your windows is 32bits, while your linux is 64 bits. Since, on 32bits systems, 'long', 'long int', oids and pointers are 4 bytes; while on they are 8 bytes. Thus, you data will most probably take more storage on a 64bits system, than on a 32bits system. How many more it is, depends on the type of your columns.
Regards,
Jennie
On Mon, Jun 22, 2009 at 12:35:52PM -0400, dariuszs wrote:
Hi, I've loaded 60 GB of data file into a table. On Windows (NTFS) the size of dbfarm changed to 80GB but on Linux (ext4 or reiserfs) dbfarm changed to 180GB. Why is that? Thanks. Dariusz.
-----
Are you an open source citizen? Join us for the Open Source Bridge
conference!
Portland, OR, June 17-19. Two days of sessions, one day of unconference:
$250.
Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
--- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.
org _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00
---------------------------------------------------------------------------- --
Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today!
http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg
_______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users
---------------------------------------------------------------------------- -- Are you an open source citizen? Join us for the Open Source Bridge conference! Portland, OR, June 17-19. Two days of sessions, one day of unconference: $250. Need another reason to go? 24-hour hacker lounge. Register today! http://ad.doubleclick.net/clk;215844324;13503038;v?http://opensourcebridge.o rg _______________________________________________ MonetDB-users mailing list MonetDB-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-users No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.339 / Virus Database: 270.12.71/2178 - Release Date: 06/21/09 20:02:00