Re: 4G SGI quad Xeon - memory-related slowdowns

[Ingo Molnar]

On 15 Jan 2001, Linus Torvalds wrote:

> The performance problem is _probably_ due to the kernel having to
> double-buffer the IO requests, coupled with bad MTRR settings (ie
> memory above the 4GB range is probably marked as non-cacheable or
> something, which means that you'll get really bad performance).

the highmem related double-buffering alone on such a category of system is
miniscule, compared to other costs of IO, and considering the expected
bandwidth (20-30 MB/sec).

the MTRR part could be a problem.

> Not using the high memory will avoid the double-buffering, and will
> also avoid using memory that isn't cached. If I'm right.

> The hang still indicates that something is wrong in PAE-land, though.

it's working just fine on all 4GB+ systems tested (including 32GB
systems), Intel, Dell, Compaq boxes. So if it's a unique PAE bug, then it
must be some boundary condition.

Paul, here is the memory map of my 8GB system:

BIOS-provided physical RAM map:
 BIOS-e820: 000000000009d400 @ 0000000000000000 (usable)
 BIOS-e820: 0000000000002c00 @ 000000000009d400 (reserved)
 BIOS-e820: 0000000000020000 @ 00000000000e0000 (reserved)
 BIOS-e820: 0000000003ef8000 @ 0000000000100000 (usable)
 BIOS-e820: 0000000000007c00 @ 0000000003ff8000 (ACPI data)
 BIOS-e820: 0000000000000400 @ 0000000003fffc00 (ACPI NVS)
 BIOS-e820: 00000000ec000000 @ 0000000004000000 (usable)
 BIOS-e820: 0000000001400000 @ 00000000fec00000 (reserved)
 BIOS-e820: 00000000f0000000 @ 0000000100000000 (usable)

and here are the MTRR settings:

[root@m mingo]# cat /proc/mtrr
reg00: base=0xf0000000 (3840MB), size= 256MB: uncachable, count=1
reg01: base=0x00000000 (   0MB), size=4096MB: write-back, count=1
reg02: base=0x100000000 (4096MB), size=2048MB: write-back, count=1
reg03: base=0x180000000 (6144MB), size=1024MB: write-back, count=1
reg04: base=0x1c0000000 (7168MB), size= 512MB: write-back, count=1
reg05: base=0x1e0000000 (7680MB), size= 256MB: write-back, count=1

i'd suggest using the mem=exact feature to force different type of memory
maps. Eg. i'm using the following append= line to force a 800 MB setup:

        append="mem=exactmap mem=0x0009d800@0x00000000 mem=0x03ef8000@0x00100000 mem=0x2bffe000@0x04000000"

such mem=exactmap lines can be constructed based on the BIOS output.

	Ingo