Re: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

newer
Re: [Monetdb-developers]...

older
Re: [Monetdb-developers]...

Stefan Manegold

19 Feb 2009 19 Feb '09

9:49 p.m.

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...

Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 - make large allocs fall back again on mmap

Peter, (how) can we make the limit (more) dependend on / adaptive to the actual memory size? Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ? I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ... Jan R. has seen similar effects with XMark on his Mac ... Thanks, Stefan

...

now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); - h->storage = STORE_MMAP; + h->newstorage = STORE_MMAP; HEAPload(h, nme, ext, FALSE); } GDKfree(of);

------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

Show replies by date

Stefan Manegold

19 Feb 19 Feb

10:01 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

On Thu, Feb 19, 2009 at 10:49:24PM +0100, Stefan Manegold wrote:

...

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 - make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

... on a 64GB Linux machine, I even see a 10x degradation: 30s -> 300s ...

...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...
now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); - h->storage = STORE_MMAP; + h->newstorage = STORE_MMAP; HEAPload(h, nme, ext, FALSE); } GDKfree(of);

------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

Peter Boncz

10:52 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

Hi, The mmap method is very costly on Windows as it does not use a virtual file with holes, but really creates it first. This may also be the case on macos. However, the pert degradation on Linux is unexpected. Maybe newer kernels have introduced different behavior. I think the principle of falling back on mmap is sound, however the threshold needs to be increased. Maybe at a fixed perc (20%?) of available ram. Peter On Feb 19, 2009, at 10:49 PM, "Stefan Manegold" wrote:

...

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 - make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...
now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); - h->storage = STORE_MMAP; + h->newstorage = STORE_MMAP; HEAPload(h, nme, ext, FALSE); } GDKfree(of);

--- --- --- --------------------------------------------------------------------- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/ adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 :07:00

Peter Boncz

10:53 p.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

You said 5gb on 8gb works better on swap memory.. wow. Maybe go for 100% of ram then for the threshold Sent from my iPhone On Feb 19, 2009, at 10:49 PM, "Stefan Manegold" wrote:

...

On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 - make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...
now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); - h->storage = STORE_MMAP; + h->newstorage = STORE_MMAP; HEAPload(h, nme, ext, FALSE); } GDKfree(of);

--- --- --- --------------------------------------------------------------------- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/ adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 :07:00

Stefan Manegold

20 Feb 20 Feb

7:51 a.m.

New subject: [Monetdb-developers] [Monetdb-checkins] MonetDB/src/gdk gdk_heap.mx, Feb2009, 1.107, 1.107.2.1

On Thu, Feb 19, 2009 at 02:53:44PM -0800, Peter Boncz wrote:

...

You said 5gb on 8gb works better on swap memory.. wow. Maybe go for 100% of ram then for the threshold

Ok. Thanks. --- Just need to find our where & how ... Stefan

...

Sent from my iPhone

On Feb 19, 2009, at 10:49 PM, "Stefan Manegold" wrote:

...
On Tue, Feb 10, 2009 at 09:13:21AM +0000, Stefan Manegold wrote:

...
Update of /cvsroot/monetdb/MonetDB/src/gdk In directory 23jxhf1.ch3.sourceforge.com:/tmp/cvs-serv18180/MonetDB/ src/gdk

Modified Files: Tag: Feb2009 gdk_heap.mx Log Message:

[on behalf of Peter] re-doing

2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108 - make large allocs fall back again on mmap

Peter,

(how) can we make the limit (more) dependend on / adaptive to the actual memory size?

Shouldn't this have an effect only with blocks > gdk_vm_minsize == 128 GB ?

I have sample scenarios (on Linux) where, e.g., queries produce intermediate results of ~ 5GB --- on my 64-bit 8GB machine those queries run in 40 secs before this change (i.e., with h->storage = STORE_MMAP;), but take 200 secs (i.e., 5 times as long) after this change (i.e., with h->newstorage = STORE_MMAP;) ...

Jan R. has seen similar effects with XMark on his Mac ...

Thanks,

Stefan

...
now only the actually intended changes and as bug fix, i.e., in the Feb2009 release branch

Index: gdk_heap.mx =================================================================== RCS file: /cvsroot/monetdb/MonetDB/src/gdk/gdk_heap.mx,v retrieving revision 1.107 retrieving revision 1.107.2.1 diff -u -d -r1.107 -r1.107.2.1 --- gdk_heap.mx 22 Jan 2009 14:47:53 -0000 1.107 +++ gdk_heap.mx 10 Feb 2009 09:13:19 -0000 1.107.2.1 @@ -115,7 +115,7 @@ fp = GDKfilelocate(nme, "wb", ext); if (fp != NULL) { fclose(fp); - h->storage = STORE_MMAP; + h->newstorage = STORE_MMAP; HEAPload(h, nme, ext, FALSE); } GDKfree(of);

--- --- --- --------------------------------------------------------------------- Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/ adobe-com _______________________________________________ Monetdb-checkins mailing list Monetdb-checkins@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-checkins

-- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.0.237 / Virus Database: 270.10.24/1954 - Release Date: 02/17/09 07 :07:00

------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ Monetdb-developers mailing list Monetdb-developers@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/monetdb-developers

5738

Age (days ago)

5739

Last active (days ago)

List overview

Download

4 comments

2 participants

participants (2)

Peter Boncz
Stefan Manegold