Re: Re: Some time monetdbd can't start existing dbfarm
Hi Brian, Thanks for looking into this issue. Here is some more details in meroveringian.log - The DB was up for about a week. There're a lot of lines of 'proxying client ...', until the line 'client error: cannot start database 'EXM': no .vaultkey found...' 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33861 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33862 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33863 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM 2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 08:36:21 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) And then, a lot of lines like below: 2016-02-07 14:44:18 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-07 14:44:23 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) Eventually, there're lot of errors like this: 2016-02-08 01:13:58 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1) 2016-02-08 01:14:03 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create EXM`?) 2016-02-08 01:14:11 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:18 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file s 2016-02-08 01:14:26 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file S Then neither mclient nor monetdbd stop cmd works any more. This is what I have in /tmp/exm folder after reboot: [centos@ip-10-72-0-98 exm]$ pwd /tmp/exm [centos@ip-10-72-0-98 exm]$ ls -al total 8088 drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 . drwxrwxrwt. 11 root root 4096 Feb 11 13:36 .. drwx------. 4 centos centos 77 Feb 9 17:33 EXM -rw-------. 1 centos centos 0 Feb 9 17:33 .merovingian_lock -rw-------. 1 centos centos 8272151 Feb 9 17:33 merovingian.log [centos@ip-10-72-0-98 exm]$ ls -al EXM total 8 drwx------. 4 centos centos 77 Feb 9 17:33 . drwxr-xr-x. 3 centos centos 62 Feb 9 17:33 .. drwx------. 66 centos centos 4096 Feb 2 19:48 bat -rw-------. 1 centos centos 0 Feb 7 08:36 .gdk_lock srwx------. 1 centos centos 0 Jan 27 14:50 .mapi.sock drwx------. 3 centos centos 16 Jan 27 14:50 sql_logs -rw-------. 1 centos centos 22 Feb 9 17:33 .uplog If needed, I can send you the whole merovingian.log file (which is 8GB in size). Thanks, HQ
participants (1)
-
Haiqing Ying