Re: MonetDB: Feb2013 - Reduce CATNAP in vmtrim
Please revert. The reason why there is a loop that sleeps 50 ms per iteration is to get a responsive exit. With your change you have to wait up to half a second before the server exits. That is too long. If you want to reduce the frequency with which the vmtrim thread does actual work, you need to change the number of times the loop is run, not the duration of the sleep. On 2013-01-08 01:21, Martin Kersten wrote:
Changeset: 4ca22f5177b6 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4ca22f5177b6 Modified Files: gdk/gdk_utils.c Branch: Feb2013 Log Message:
Reduce CATNAP in vmtrim The overload on memory is already protected in the dataflow scheduler and optimizer pipeline, which means we probably can relax the time to issue the BBPtrim().
diffs (22 lines):
diff --git a/gdk/gdk_utils.c b/gdk/gdk_utils.c --- a/gdk/gdk_utils.c +++ b/gdk/gdk_utils.c @@ -1034,6 +1034,7 @@ GDKmunmap(void *addr, size_t size) * Their value is turned into a blanc space. */
+#define CATNAP 500 int GDKrecovery = 0;
static MT_Id GDKvmtrim_id; @@ -1053,8 +1054,8 @@ GDKvmtrim(void *limit) size_t cursize;
/* sleep using catnaps so we can exit in a timely fashion */ - for (t = highload ? 500 : 5000; t > 0; t -= 50) { - MT_sleep_ms(50); + for (t = highload ? CATNAP : 10 * CATNAP; t > 0; t -= CATNAP) { + MT_sleep_ms(CATNAP); if (GDKexiting()) return; } _______________________________________________ checkin-list mailing list checkin-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/checkin-list
-- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
On Tue, Jan 08, 2013 at 10:12:44AM +0100, Sjoerd Mullender wrote:
Please revert.
The reason why there is a loop that sleeps 50 ms per iteration is to get a responsive exit. With your change you have to wait up to half a second before the server exits. That is too long.
"nasty" side effect of this checkin is also that Mtest now takes more than twice as long as before this very checkin, mainly because each tiny test is now bound to take at least 500 ms (half a second), rather than 50ms before ... Stefan
If you want to reduce the frequency with which the vmtrim thread does actual work, you need to change the number of times the loop is run, not the duration of the sleep.
On 2013-01-08 01:21, Martin Kersten wrote:
Changeset: 4ca22f5177b6 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4ca22f5177b6 Modified Files: gdk/gdk_utils.c Branch: Feb2013 Log Message:
Reduce CATNAP in vmtrim The overload on memory is already protected in the dataflow scheduler and optimizer pipeline, which means we probably can relax the time to issue the BBPtrim().
diffs (22 lines):
diff --git a/gdk/gdk_utils.c b/gdk/gdk_utils.c --- a/gdk/gdk_utils.c +++ b/gdk/gdk_utils.c @@ -1034,6 +1034,7 @@ GDKmunmap(void *addr, size_t size) * Their value is turned into a blanc space. */
+#define CATNAP 500 int GDKrecovery = 0;
static MT_Id GDKvmtrim_id; @@ -1053,8 +1054,8 @@ GDKvmtrim(void *limit) size_t cursize;
/* sleep using catnaps so we can exit in a timely fashion */ - for (t = highload ? 500 : 5000; t > 0; t -= 50) { - MT_sleep_ms(50); + for (t = highload ? CATNAP : 10 * CATNAP; t > 0; t -= CATNAP) { + MT_sleep_ms(CATNAP); if (GDKexiting()) return; } _______________________________________________ checkin-list mailing list checkin-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/checkin-list
-- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
-- | Stefan.Manegold @ CWI.nl | DB Architectures (INS1) | | http://CWI.nl/~manegold/ | Science Park 123 (L321) | | Tel.: +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) | _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
On 1/8/13 6:47 PM, Stefan Manegold wrote:
On Tue, Jan 08, 2013 at 10:12:44AM +0100, Sjoerd Mullender wrote:
Please revert.
The reason why there is a loop that sleeps 50 ms per iteration is to get a responsive exit. With your change you have to wait up to half a second before the server exits. That is too long.
Yes, it is unclear what is the best level of the NAPs would be. Perhaps there are more modern/recent approaches for spinlocking in the kernel. Or we have to deal with another way to shut down the kernel.
"nasty" side effect of this checkin is also that Mtest now takes more than twice as long as before this very checkin, mainly because each tiny test is now bound to take at least 500 ms (half a second), rather than 50ms before ...
Stefan
If you want to reduce the frequency with which the vmtrim thread does actual work, you need to change the number of times the loop is run, not the duration of the sleep.
On 2013-01-08 01:21, Martin Kersten wrote:
Changeset: 4ca22f5177b6 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4ca22f5177b6 Modified Files: gdk/gdk_utils.c Branch: Feb2013 Log Message:
Reduce CATNAP in vmtrim The overload on memory is already protected in the dataflow scheduler and optimizer pipeline, which means we probably can relax the time to issue the BBPtrim().
diffs (22 lines):
diff --git a/gdk/gdk_utils.c b/gdk/gdk_utils.c --- a/gdk/gdk_utils.c +++ b/gdk/gdk_utils.c @@ -1034,6 +1034,7 @@ GDKmunmap(void *addr, size_t size) * Their value is turned into a blanc space. */
+#define CATNAP 500 int GDKrecovery = 0;
static MT_Id GDKvmtrim_id; @@ -1053,8 +1054,8 @@ GDKvmtrim(void *limit) size_t cursize;
/* sleep using catnaps so we can exit in a timely fashion */ - for (t = highload ? 500 : 5000; t > 0; t -= 50) { - MT_sleep_ms(50); + for (t = highload ? CATNAP : 10 * CATNAP; t > 0; t -= CATNAP) { + MT_sleep_ms(CATNAP); if (GDKexiting()) return; } _______________________________________________ checkin-list mailing list checkin-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/checkin-list
-- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
On 2013-01-08 18:56, Martin Kersten wrote:
On 1/8/13 6:47 PM, Stefan Manegold wrote:
On Tue, Jan 08, 2013 at 10:12:44AM +0100, Sjoerd Mullender wrote:
Please revert.
The reason why there is a loop that sleeps 50 ms per iteration is to get a responsive exit. With your change you have to wait up to half a second before the server exits. That is too long.
Yes, it is unclear what is the best level of the NAPs would be. Perhaps there are more modern/recent approaches for spinlocking in the kernel. Or we have to deal with another way to shut down the kernel.
Seriously, one select every 50 ms isn't going to be noticeable.
"nasty" side effect of this checkin is also that Mtest now takes more than twice as long as before this very checkin, mainly because each tiny test is now bound to take at least 500 ms (half a second), rather than 50ms before ...
Stefan
If you want to reduce the frequency with which the vmtrim thread does actual work, you need to change the number of times the loop is run, not the duration of the sleep.
On 2013-01-08 01:21, Martin Kersten wrote:
Changeset: 4ca22f5177b6 for MonetDB URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4ca22f5177b6 Modified Files: gdk/gdk_utils.c Branch: Feb2013 Log Message:
Reduce CATNAP in vmtrim The overload on memory is already protected in the dataflow scheduler and optimizer pipeline, which means we probably can relax the time to issue the BBPtrim().
diffs (22 lines):
diff --git a/gdk/gdk_utils.c b/gdk/gdk_utils.c --- a/gdk/gdk_utils.c +++ b/gdk/gdk_utils.c @@ -1034,6 +1034,7 @@ GDKmunmap(void *addr, size_t size) * Their value is turned into a blanc space. */
+#define CATNAP 500 int GDKrecovery = 0;
static MT_Id GDKvmtrim_id; @@ -1053,8 +1054,8 @@ GDKvmtrim(void *limit) size_t cursize;
/* sleep using catnaps so we can exit in a timely fashion */ - for (t = highload ? 500 : 5000; t > 0; t -= 50) { - MT_sleep_ms(50); + for (t = highload ? CATNAP : 10 * CATNAP; t > 0; t -= CATNAP) { + MT_sleep_ms(CATNAP); if (GDKexiting()) return; } _______________________________________________ checkin-list mailing list checkin-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/checkin-list
-- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
_______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
-- Sjoerd Mullender _______________________________________________ developers-list mailing list developers-list@monetdb.org http://mail.monetdb.org/mailman/listinfo/developers-list
participants (3)
-
Martin Kersten -
Sjoerd Mullender -
Stefan Manegold