Another huge issue is that deleting records from a table still won't shrink the size of the table. I like to keep some of my raw data tables small so once a day I delete some already processed records out of them, dump the table, and then reload it.
To manage the rest of the database I have a series of scripts that check to see if the database is using a stupid amount of memory (80%+ of physical memory on the machine) or to see if the database is deadlocked (this happens several times a day). If either of these situations exist then I SIGHUP the merovingian and mserver processes... when the database comes back up and gets primed then it will be using a reasonable amount of memory for about 8 hours or so.
This is all acceptable at the moment because the database is ridiculously fast.
73,
Matthew W. Jones (KI4ZIB)
http://matburt.net
On Mon, May 17, 2010 at 9:54 AM, Stefan de Konink
<stefan@konink.de> wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Op 17-05-10 15:39, Matthew Jones schreef:
> You're still going to have to be very careful with MonetDB, depending on
> the volume of incoming data you'll need to restart the database at least
> a couple of times a day.
One new developer already managed to get 90GB of used memory on a table
having only 22MB, consisting of integers and timestamps. I did
congratulate this user with this achievement, but want to reproduce it.
Interestingly, after a restart (ok a kill) everything went back to
acceptable levels: 1191MB (virtual), 289MB (resources), 11MB (shared) it
is interesting to see how such things can expand.
Stefan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEAREKAAYFAkvxSoEACgkQYH1+F2Rqwn3LdgCdEXJdCs1uGH+1CVFTAIdTt6xs
PQcAoI7AYnwTn5dFf8MB4GrCdqAyUU8M
=pB2H
-----END PGP SIGNATURE-----