Re: Memory management, 8 years later

Hi Roberto, ----- Original Message -----

From: "Roberto Cornacchia" To: "Communication channel for MonetDB users" Sent: Thursday, June 28, 2018 11:39:09 AM Subject: Memory management, 8 years later

Hi all,

8 yeas after this interesting thread about using ulimit to limit the data segment size ( https://www.monetdb.org/pipermail/users-list/2010-March/004007.html ), a few things have changed and I find myself a bit lost again about memory management.

At some point (can't remember when) mserver5 stopped respecting ulimit. No matter how I change the limits, it always sees all available physical ram. Does any other mechanism exist?

I think this is the correct behaviour of ulimit. The mserver5 process has nothing to do with it, ulimit controls how much resources a shell can use, it does not "fake" the amount of physical memory of the machine. And the behaviour of a process that hits the ulimit setting likely behaves differently than a process on a machine that runs out of real memory. And you also have to take into account that you can have ulimit settings for users, processes and shells.

The issue I'm facing all the time, and for which I find very little documentation, is: - multiple dbfarms, possibly tens of mserver5 instances, running on the same (virtual) machine, plus other applications - Not necessarily, but most times each dmfarm is served by a Docker container - CPU and memory resources need to be share fairly (ideally, controlling priorities) - No mserver5 should ever be killed because of memory issues

Now, especially when using Docker, there are means to control the maximum memory size allowed to each process. However, the resolution in case of overusage is to kill the process. That is of course not a solution.

What I'm looking for is: can I set a cap to resources *at process level*, in a way that mserver5 will just *see* those, make the best out of them, and never try to use more? (yes, one way would be a VM, but far from ideal for me)

I think you are right that the only available solution is vm's. Otherwise monetdb would have to enforce the memory usage per process itself. But i am not sure if this information is available internally. And even if it is, how would you handle for example a situation where one of the dbfarms runs out of memory during a query that is split over multiple dbfarms? Arjen

Does anyone have experience with / tips about multi-dbfarm environments?

Thanks! Roberto

_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list