Re: Deadlock on the mm->mmap_sem

[David Howells]

Looking through the do_page_fault(), I noticed there's a race in expand stack
because expand_stack() expects the caller to have the mm-sem write-locked.

I've attached a patch that might fix it appropriately. Alternatively, it may
be worth applying Andrea's 00_silent-stack-overflow-10 patch which fixes this
and something else too.

David

diff -uNr linux-2.4.10-pre12/include/linux/mm.h linux-rwsem/include/linux/mm.h
--- linux-2.4.10-pre12/include/linux/mm.h	Wed Sep 19 10:39:23 2001
+++ linux-rwsem/include/linux/mm.h	Wed Sep 19 10:40:48 2001
@@ -586,11 +586,11 @@
 	 * before relocating the vma range ourself.
 	 */
 	address &= PAGE_MASK;
+	spin_lock(&vma->vm_mm->page_table_lock);
 	grow = (vma->vm_start - address) >> PAGE_SHIFT;
 	if (vma->vm_end - address > current->rlim[RLIMIT_STACK].rlim_cur ||
 	    ((vma->vm_mm->total_vm + grow) << PAGE_SHIFT) > current->rlim[RLIMIT_AS].rlim_cur)
-		return -ENOMEM;
-	spin_lock(&vma->vm_mm->page_table_lock);
+		goto nomem;
 	vma->vm_start = address;
 	vma->vm_pgoff -= grow;
 	vma->vm_mm->total_vm += grow;
@@ -598,6 +598,9 @@
 		vma->vm_mm->locked_vm += grow;
 	spin_unlock(&vma->vm_mm->page_table_lock);
 	return 0;
+ nomem:
+	spin_unlock(&vma->vm_mm->page_table_lock);
+	return -ENOMEM;
 }
 
 /* Look up the first VMA which satisfies  addr < vm_end,  NULL if none. */