In Linux, a process is in place called the OOM Killer that kills processes when memory starts getting too low.
While this is an interesting idea, it can kill your application before it kills something that might be a lower priority.
Of course, the best solution is to add more memory to a server. But if it’s not immediately possible, you can make some changes to memory management and the OOM Killer to make sure your application has the highest priority for memory usage.
In the /proc/[process id]/oom_adj file, is the rating of a process. By default, every process has a 0 (zero) rating. As time goes by, some processes gain priority and this file will drop. A quick look at one of my servers show a lot of zeros, a couple of -4’s, a bunch of -15’s, and a few -17’s. Based on the configuration of OOM Killer, any process with a zero rating will be killed before the -4, -15, or -17, which will be the last one killed.
In order to ensure the application has the highest priority, make a list of the processes (not process IDs) that are a lower priority such as monitoring or backup agents, and whip up a script that runs regularly. The script retrieves the PID of the listed process and sets the oom_adj value to 100. This ensures this lower priority process is killed before a higher, more important application is touched. I use 100 however it can be anything greater than zero.
#!/bin/bash
for i in $(oompriority)
do
OOMPID=$(ps -e | awk '/${i}/{print $1}')
if [[ ${OOMPID} -gt 0 ]]
then
echo 100 > /proc/${OOMPID}/oom_adj
fi
done