Some time monetdbd can't start existing dbfarm
Hi, I created a dbfarm (/tmp/exm ), then created database (EXM), load data, do queries, etc, everything works fine for a week. Then I found mclient stopped working (can't connect to monetdb). The log file shows 'too many files opened' issue. I guess it might be some connection leak issue. I tried 'monetdbd stop /tmp/exm', but it's not responding. It looks I can't stop the dbfarm now. I then reboot the system and after reboot tried 'monetdbd start /tmp/exm' but I got an error saying ' cannot find file in /tmp/exm'. I verified the /tmp/exm folder did exist and there're still sub folder ( ./EXM) and log file still under /tmp/exm. The only option left seems to be creating a new dbfarm and recreating the whole database again. We're moving to use MonetDB in production and if this is a real bug then it would be too risky for production. The question is, is this a real issue? Or it's just some user error that I made? How do I make sure we won't run into this situation again in production mode. Many thanks, HQ
Hi Haiqing,
Please provide the output of your merovingian.log for more information.
Regards,
Brian Hood
On Feb 10, 2016 10:42 PM, "Haiqing Ying"
Hi,
I created a dbfarm (/tmp/exm ), then created database (EXM), load data, do queries, etc, everything works fine for a week. Then I found mclient stopped working (can’t connect to monetdb). The log file shows ‘too many files opened’ issue. I guess it might be some connection leak issue. I tried ‘monetdbd stop /tmp/exm’, but it’s not responding. It looks I can’t stop the dbfarm now. I then reboot the system and after reboot tried ‘monetdbd start /tmp/exm’ but I got an error saying ‘ cannot find file in /tmp/exm’. I verified the /tmp/exm folder did exist and there’re still sub folder ( ./EXM) and log file still under /tmp/exm. The only option left seems to be creating a new dbfarm and recreating the whole database again. We’re moving to use MonetDB in production and if this is a real bug then it would be too risky for production. The question is, is this a real issue? Or it’s just some user error that I made? How do I make sure we won’t run into this situation again in production mode.
Many thanks, HQ
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
Hi,
using /tmp for a dbfarm is very risky and not recommended.
I am not aware of the latest updates for /tmp in linux but last time I
checked /tmp is cleared when the system reboots, and also it does not
guarantees the safety of any file between runs. It is a linux thing.
I would suggest to run monetdb with a dbfarm somewhere different than /tmp.
I googled a bit about /tmp and I found this:
(http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/tmp.html)
This directory contains mostly files that are required temporarily. Many
programs use this to create lock files and for temporary storage of data.
Do not remove files from this directory unless you know exactly what you
are doing! Many of these files are important for currently running programs
and deleting them may result in a system crash. Usually it won't contain
more than a few KB anyway. On most systems, this directory is cleared out
at boot or at shutdown by the local system. The basis for this was
historical precedent and common practice. However, it was not made a
requirement because system administration is not within the scope of the
FSSTND. For this reason people and programs must not assume that any files
or directories in /tmp are preserved between invocations of the program.
The reasoning behind this is for compliance with IEEE standard P1003.2
(POSIX, part 2)
Hence, it is very likely that linux deleted something that was needed. Be
safe and don't run dbfarm under /tmp.
Lefteris
------------------------------
P http://www.tldp.org/LDP/Linux-Filesystem-Hierarchy/html/srv.html
On Thu, Feb 11, 2016 at 6:56 PM, Brian Hood
Hi Haiqing,
Please provide the output of your merovingian.log for more information.
Regards,
Brian Hood On Feb 10, 2016 10:42 PM, "Haiqing Ying"
wrote: Hi,
I created a dbfarm (/tmp/exm ), then created database (EXM), load data, do queries, etc, everything works fine for a week. Then I found mclient stopped working (can’t connect to monetdb). The log file shows ‘too many files opened’ issue. I guess it might be some connection leak issue. I tried ‘monetdbd stop /tmp/exm’, but it’s not responding. It looks I can’t stop the dbfarm now. I then reboot the system and after reboot tried ‘monetdbd start /tmp/exm’ but I got an error saying ‘ cannot find file in /tmp/exm’. I verified the /tmp/exm folder did exist and there’re still sub folder ( ./EXM) and log file still under /tmp/exm. The only option left seems to be creating a new dbfarm and recreating the whole database again. We’re moving to use MonetDB in production and if this is a real bug then it would be too risky for production. The question is, is this a real issue? Or it’s just some user error that I made? How do I make sure we won’t run into this situation again in production mode.
Many thanks, HQ
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
participants (3)
-
Brian Hood
-
Haiqing Ying
-
Lefteris