[BUG][PATCH] 2.4.* mlockall(MCL_FUTURE) is broken -- child inherits

[Dave Anderson]

In 2.4.*, mlockall(MCL_FUTURE) is erroneously inherited by child processes
across fork() and exec():

 1. across a fork(), the inherited memory is not locked, but any new memory
    allocations by the child will be VM_LOCKED.
 2. across a subsequent exec(), *all* of the exec'd program's memory except
    for its stack pages will be VM_LOCKED.

The problem is:

 1.  if MCL_FUTURE, mm->def_flags gets set to VM_LOCKED in do_mlockall().
 2.  mm->def_flags is not cleared during subsequent forks and execs.
 3   mm->def_flags, with the leftover VM_LOCKED flag set, is subsequently
     utilized in calc_vm_flags() when called by do_brk() to extend the
     address space of a forked process, and by do_mmap_pgoff() when
     building the non-stack address space of an exec'd process.

The proposed patch puts the fix in mm_init(), which seems to be the most
appropriate place since it's called by copy_mm(), and by mm_alloc() on behalf
of exec_mmap():

# diff -u linux/kernel/fork.c linux-2.4.17/kernel/fork.c
--- linux/kernel/fork.c Tue Jan  8 15:11:13 2002
+++ linux-2.4.17/kernel/fork.c  Tue Jan  8 15:12:26 2002
@@ -219,6 +219,7 @@
        init_rwsem(&mm->mmap_sem);
        mm->page_table_lock = SPIN_LOCK_UNLOCKED;
        mm->pgd = pgd_alloc(mm);
+       mm->def_flags = 0;
        if (mm->pgd)
                return mm;
        free_mm(mm);

Note that it worked OK in 2.2 because mm->def_flags was explicitly cleared in
mm_alloc(), which was called by both copy_mm() and exec_mmap().  But things
were shuffled around a bit in 2.4, and it must have gotten lost in the
translation...

Dave Anderson

==============================================================================
  David Anderson                             [e-mail]
  Mission Critical Linux, Inc.               http://www.mclinux.com
  100 Foot of John St.                       Work: 978-606-0225
  Lowell, MA 01852                           Fax: [phone]
==============================================================================