PATCH - fs: limit maximum concurrent coredumps

A moderately large number of large core files being dumped simultaneously
can impose severe latency penalties on other processes due to IO load.
For example, a common configuration for PHP web-servers includes apache's
prefork MPM, mod-php and a PHP opcode cache utilizing shared memory. In
certain failure modes, all requests serviced by PHP result in a segfault.
Enabling coredumps might lead to 10-20 coredumps per second, all attempting
to write a 150-200MB core file. This leads to the whole system becoming
entirely unresponsive for many minutes.
The ability to limit concurrent coredumps allows dumping core to be safely
enabled in these situations without affecting responsiveness of the system
as a whole.
I have several servers running with this patch applied (actually backported
to v2.6.26) and it has allowed me to deal successfully with the situation
described above.
In do-coredump I have pulled dump-count back out to the top-level scope
and core-dump-count is now incremented in the normal path.
Added sysctl parameter for tuning core-max-concurrency.
checkpatch complains about "extern ... core-max-concurrency" but this seems
to be an acceptable exception based on the preponderance of other extern
variables in kernel/sysctl.c
Signed-off-by: Edward Allcutt
diff ==============================================================
+core-max-concurrency:
+
+Controls the maximum number of coredumps that can be in progress
+concurrently. Coredumping processes that would cause this value
+to be exceeded are noted via the kernel log and their cores are
+skipped. 0 is a special value, indicating that unlimited processes
+may be captured in parallel. This value defaults to 0.
+
+==============================================================
core-pattern:
core-pattern is used to specify a core dumpfile pattern name.
diff int core-uses-pid;
+unsigned int core-max-concurrency;
char core-pattern[CORENAME-MAX-SIZE] = "core";
unsigned int core-pipe-limit;
int suid-dumpable = 0;
@@ -1844,6 +1845,7 @@ void do-coredump(long signr, int exit-code, struct pt-regs *regs)
int retval = 0;
int flag = 0;
int ispipe;
+ int dump-count = 0;
static atomic-t core-dump-count = ATOMIC-INIT(0);
struct coredump-params cprm = {
.signr = signr,
@@ -1865,6 +1867,14 @@ void do-coredump(long signr, int exit-code, struct pt-regs *regs)
if (!+ dump-count = atomic-inc-return(&core-dump-count);
+ if (core-max-concurrency && (core-max-concurrency < dump-count)) {
+ printk(KERN-WARNING "Pid %d(%s) over core-max-concurrencyn",
+ task-tgid-vnr(current), current-+ printk(KERN-WARNING "Skipping core dumpn");
+ goto fail;
+ }
+
cred = prepare-creds();
if (!cred)
goto fail;
@@ -1900,7 +1910,6 @@ void do-coredump(long signr, int exit-code, struct pt-regs *regs)
unlock-kernel();
if (ispipe) {
- int dump-count;
char **helper-argv;
if (cprm.limit == 1) {
@@ -1926,19 +1935,18 @@ void do-coredump(long signr, int exit-code, struct pt-regs *regs)
}
cprm.limit = RLIM-INFINITY;
- dump-count = atomic-inc-return(&core-dump-count);
if (core-pipe-limit && (core-pipe-limit < dump-count)) {
printk(KERN-WARNING "Pid %d(%s) over core-pipe-limitn",
task-tgid-vnr(current), current- printk(KERN-WARNING "Skipping core dumpn");
- goto fail-dropcount;
+ goto fail-unlock;
}
helper-argv = argv-split(GFP-KERNEL, corename+1, NULL);
if (!helper-argv) {
printk(KERN-WARNING "%s failed to allocate memoryn",
retval = call-usermodehelper-fns(helper-argv[0], helper-argv,
@@ -1994,14 +2002,13 @@ void do-coredump(long signr, int exit-code, struct pt-regs *regs)
close-fail:
if (cprm.file)
filp-close(cprm.file, NULL);
-fail-dropcount:
- if (ispipe)
- atomic-dec(&core-dump-count);
fail-unlock:
coredump-finish(mm);
revert-creds(old-cred);
fail-creds:
put-cred(cred);
fail:
+ if (dump-count)
+ atomic-dec(&core-dump-count);
return;
}
diff More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

A core dump is just an instance of a process suddenly reading lots of its
address space and doing lots of filesystem writes, producing the kinds of
thrashing that any such instance might entail. It really seems like the
real solution to this kind of problem will be in some more general kind of
throttling of processes (or whatever manner of collections thereof) when
they got hog-wild on page-ins or filesystem writes, or whatever else. I'm
not trying to get into the details of what that would be. But I have to
cite this hack as the off-topic kludge that it really is. That said, I do
certainly sympathize with the desire for a quick hack that addresses the
scenario you experience.
For the case you described, it seems to me that constraining concurrency
per se would be better than punting core dumps when too concurrent. That
is, you should not skip the dump when you hit the limit. Rather, you
should block in do-coredump() until the next dump already in progress
finishes. (It should be possible to use TASK-KILLABLE so that those dumps
in waiting can be aborted with a follow-on SIGKILL. But Oleg will have to
check on the signals details being right for that.)
That won't make your crashers each complete quickly, but it will prevent
the thrashing. Instead of some crashers suddenly not producing dumps at
all, they'll just all queue up waiting to finish crashing but not using any
CPU or IO resources. That way you don't lose any core dumps unless you
want to start SIGKILL'ing things (which oom-kill might do if need be),
you just don't die in flames trying to do nothing but dump cores.
Thanks,
Roland
More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, 21 Jun 2010 18:23:03 -0700 (PDT) Roland McGrath wrote:
yup.
yup.
Might be able to use semaphores for this. Use sema-init(),
down-killable() and up().
Modifying the max concurrency value would require a loop of up()s and
down()s, probably all surrounded by a mutex-lock. Which is a bit ugly,
and should be done in kernel/semaphore.c I guess.
A global knob is a bit old-school. Perhaps it should be a per-memcg
knob or something.
otoh, one could perhaps toss all these tasks into a blkio-cgroup and
solve this problem with the block IO controller. After all, that's
what it's for.
More majordomo info at http://vger.kernel.org/majordomo-info.html

To be honest, My customers have very similar problem for long time. If
HPC MPI job or apache prefork MPM processces die, The system load to
make large concurrent core dump I/O and consume all I/O band width.
Eventually, all other service stall long time and HA cluster software
shutdown such node forcely.
My case used two technique, 1) limit concurrent coredump (but not skip
as sugessted Roland) 2) reduce I/O priority automatically in do-coredump().
I don't know my way was correct way. but I believe we need something case.
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 06/21/2010 06:23 PM, Roland McGrath wrote:
It is, although with one possibly important difference: the process has
had an involuntary state transition, which may mean that its priority
settings that it had as a "live" process are no longer applicable. It
would certainly seem appropriate to give the administrator the option of
altering the priority parameters of coredumping processes.
-hpa
More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, 21 Jun 2010, Roland McGrath wrote:
But priority settings don't apply any more for core dumping process, do
they?
More majordomo info at http://vger.kernel.org/majordomo-info.html

On Mon, 21 Jun 2010 18:41:16 -0700
Hmm, in my desktop, it seems coredump in a group is charged against
root cgroup. (not against the group it belongs to.)
This seems strange.....I've chased why...for 2 hours. I noticed
==
[root@bluextal kamezawa]# cat /proc/sys/kernel/core-pattern
|/usr/libexec/abrt-hook-ccpp /var/cache/abrt %p %s %u %c
==
This is fedora-12.
Then, for recent distros, doing "coredump" with some limited resource may
be a job of abrt program. It can make use of I/O cgroup + direct I/O.
If a kernel help is necesary, this helper function should work in
the caller's cgroup, maybe.
Regards,
-Kame
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Why wouldn't they?
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 06/21, Edward Allcutt wrote:
OK, but please note that the patch is not right,
-
Suppose that core-dump-count SIGQUIT to the process. With this patch SIGQUIT suddenly starts to kill
the single thread, this must not happen.
If you change the patch to sleep until core-dump-count < core-max-concurrency,
then, again, we should kill other threads first.
Oleg.
More majordomo info at http://vger.kernel.org/majordomo-info.html

On 06/23, Oleg Nesterov wrote:
OOPS, sorry, I was not exactly right too.
sub-threads.
However, this doesn't kill other CLONE-VM tasks. Perhaps this is fine,
but I am not sure.
Yes, this is true. If we are going to sleep, we shouldn't allow other
threads to run.
Oleg.
More majordomo info at http://vger.kernel.org/majordomo-info.html