Hi Brian, 

Thanks for looking into this issue. 

Here is some more details in meroveringian.log – The DB was up for about a week.  There’re a lot of lines of ‘proxying client …’, until the line ‘client error: cannot start database ‘EXM’: no .vaultkey found…'

2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33861 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM

2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying

2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33862 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM

2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying

2016-02-07 08:08:13 MSG merovingian[12262]: proxying client localhost:33863 for database 'EXM' to mapi:monetdb:///tmp/exm/EXM/.mapi.sock?database=EXM

2016-02-07 08:08:13 MSG merovingian[12262]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying

2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1)

2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create

 EXM`?)

2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1)

2016-02-07 08:36:20 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create

 EXM`?)

2016-02-07 08:36:20 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1)

2016-02-07 08:36:21 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create

 EXM`?)

2016-02-07 14:44:18 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1)

2016-02-07 14:44:23 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create

 EXM`?)

Eventually, there’re lot of errors like this:

2016-02-08 01:13:58 MSG merovingian[12262]: starting database 'EXM', up min/avg/max: 0s/0s/0s, crash average: 1.00 0.10 0.03 (1-0=1)

2016-02-08 01:14:03 ERR merovingian[12262]: client error: cannot start database 'EXM': no .vaultkey found (did you create the database with `monetdb create

 EXM`?)

2016-02-08 01:14:11 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file

s

2016-02-08 01:14:18 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file

s

2016-02-08 01:14:26 ERR merovingian[12262]: client error: could not retrieve uplog information: could not open file /tmp/exm/EXM/.uplog: Too many open file

S

Then neither mclient nor monetdbd stop cmd works any more. 

This is what I have in /tmp/exm folder after reboot:

[centos@ip-10-72-0-98 exm]$ pwd

/tmp/exm

[centos@ip-10-72-0-98 exm]$ ls -al

total 8088

drwxr-xr-x.  3 centos centos      62 Feb  9 17:33 .

drwxrwxrwt. 11 root   root      4096 Feb 11 13:36 ..

drwx------.  4 centos centos      77 Feb  9 17:33 EXM

-rw-------.  1 centos centos       0 Feb  9 17:33 .merovingian_lock

-rw-------.  1 centos centos 8272151 Feb  9 17:33 merovingian.log

[centos@ip-10-72-0-98 exm]$ ls -al EXM

total 8

drwx------.  4 centos centos   77 Feb  9 17:33 .

drwxr-xr-x.  3 centos centos   62 Feb  9 17:33 ..

drwx------. 66 centos centos 4096 Feb  2 19:48 bat

-rw-------.  1 centos centos    0 Feb  7 08:36 .gdk_lock

srwx------.  1 centos centos    0 Jan 27 14:50 .mapi.sock

drwx------.  3 centos centos   16 Jan 27 14:50 sql_logs

-rw-------.  1 centos centos   22 Feb  9 17:33 .uplog

If needed, I can send you the whole merovingian.log file (which is 8GB in size). 

Thanks,

HQ