[MonetDB-users] ODBC Stops inserting
Hello everyone! We're using the odbc driver to insert records every half second or so. These records are for analytics information about visitors to various websites that we host. (We found MonetDB to be very fast for fetching information like average request time, number of visitors, general analytics information like that.) We've run into a few issues though. The biggest concern is that we stop seeing new records entered after a while. The log shows 2011-12-13 17:00:00 MSG merovingian[86]: proxying client 10.60.5.13:37359 for database 'web_requests' to mapi:monetdb:///var/lib/monetdb/web_requests/web_requests/.mapi.sock?database=web_requests 2011-12-13 17:00:00 MSG merovingian[86]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying over and over for each machine that would be inserting these records. But no new records are ever added. Another concern is that in our installation, we don't see a MonetDB.conf file. I was hoping that we could specify how much RAM we'd like to allow MonetDB to consume. This is to ensure that our memory manager doesn't kill MonetDB, since that's the primary role of this server. Is this a file that we should create? I'm still fairly new to MonetDB and as a result, am still trying to find my way with it. The last concern I have is the log file size. Is there a way (once we've ironed out the before mentioned issues) to instruct MonetDB to only log things of a certain importance or higher? For example, to only log ERR rather than ERR and MSG messages? Our log files will become quite large and unmanageable if we don't disable or implement some sort of log rotation system. The version of MonetDB that we are running is MonetDB 5 server v11.5.7 "Aug2011-SP2". We are running under Ubuntu 10.04 64Bit. Thanks, Joseph Brower
Just answering the bit I have an answer for immediately. On 13-12-2011 10:06:00 -0700, Joseph Brower wrote:
Another concern is that in our installation, we don't see a MonetDB.conf file. I was hoping that we could specify how much RAM we'd like to allow MonetDB to consume. This is to ensure that our memory manager doesn't kill MonetDB, since that's the primary role of this server. Is this a file that we should create? I'm still fairly new to MonetDB and as a result, am still trying to find my way with it.
Short answer, you can't. The config files are gone too, infavour of settings managed by monetdbd.
The last concern I have is the log file size. Is there a way (once we've ironed out the before mentioned issues) to instruct MonetDB to only log things of a certain importance or higher? For example, to only log ERR rather than ERR and MSG messages? Our log files will become quite large and unmanageable if we don't disable or implement some sort of log rotation system.
There is no notion of a log-level, and MSG and ERR messages can't be separated either. You could locally patch, but I don't think it's worth it. Rotating logs is fairly easy. monetdbd accepts SIGHUP to reopen its logfile, so it's easy to integrate with logrotate.
Thanks. On 12/13/2011 10:38 AM, Fabian Groffen wrote:
Just answering the bit I have an answer for immediately.
On 13-12-2011 10:06:00 -0700, Joseph Brower wrote:
Another concern is that in our installation, we don't see a MonetDB.conf file. I was hoping that we could specify how much RAM we'd like to allow MonetDB to consume. This is to ensure that our memory manager doesn't kill MonetDB, since that's the primary role of this server. Is this a file that we should create? I'm still fairly new to MonetDB and as a result, am still trying to find my way with it. Short answer, you can't. The config files are gone too, infavour of settings managed by monetdbd. Is there any documentation about how to properly manage settings via monetdbd? The last concern I have is the log file size. Is there a way (once we've ironed out the before mentioned issues) to instruct MonetDB to only log things of a certain importance or higher? For example, to only log ERR rather than ERR and MSG messages? Our log files will become quite large and unmanageable if we don't disable or implement some sort of log rotation system. There is no notion of a log-level, and MSG and ERR messages can't be separated either. You could locally patch, but I don't think it's worth it. Rotating logs is fairly easy. monetdbd accepts SIGHUP to reopen its logfile, so it's easy to integrate with logrotate.
That's what I figured. :-D I'll make sure we implement the log rotation. I've done some more research. It appears to be an issue with how our other system is inserting information. We'll be continuing to diagnose this. Thanks, Joseph Brower
On 13-12-2011 10:42:40 -0700, Joseph Brower wrote:
On 13-12-2011 10:06:00 -0700, Joseph Brower wrote:
Another concern is that in our installation, we don't see a MonetDB.conf file. I was hoping that we could specify how much RAM we'd like to allow MonetDB to consume. This is to ensure that our memory manager doesn't kill MonetDB, since that's the primary role of this server. Is this a file that we should create? I'm still fairly new to MonetDB and as a result, am still trying to find my way with it. Short answer, you can't. The config files are gone too, infavour of settings managed by monetdbd. Is there any documentation about how to properly manage settings via monetdbd?
See the manpages for monetdbd(1) and monetdb(1): http://www.monetdb.org/Documentation/monetdbd-man-page http://www.monetdb.org/Documentation/monetdb-man-page
There is no notion of a log-level, and MSG and ERR messages can't be separated either. You could locally patch, but I don't think it's worth it. Rotating logs is fairly easy. monetdbd accepts SIGHUP to reopen its logfile, so it's easy to integrate with logrotate.
That's what I figured. :-D I'll make sure we implement the log rotation.
I've done some more research. It appears to be an issue with how our other system is inserting information. We'll be continuing to diagnose this.
You should have not so rapidly growing logs, if you can get clients to use a persistent connection (not making a new connection for every query).
participants (2)
-
Fabian Groffen -
Joseph Brower