Memory Write Ordering Question

[James W. Lake]

Hello,

I've run into a problem with memory write ordering on a PCI Device.  

The  card has 2 memory regions, pthis->pConfigMem (PCI Chip
Configuration), and pthis->pDspMem (A DSP's memory on the PCI Card)

The code sequence below doesn't work out as expected.  The
writel(temp,addr); at the bottom actually occurs in the middle of the
memcpy_toio() operation.

The slew of barrier, mb, wmb, rmb's seem to be completely ignored.  
If I uncomment the line: junk = readl(pthis->pDspMem);  everything
behaves as expected.

I'm wondering if anyone has any idea what exactly is causing this.  The
readl is a so-so work around.  I'd like to figure out how to do it
correctly.  Does anyone who knows more about Intel CPU's and write
ordering and PCI have any ideas?

I can very reliably monitor the code's behavior, so it's easy to tell if
its fixed.  :-)

Also,  the LSPAM() lines are simply printk macros.  If I have them
actually outputting, the code also works correctly.  If they are
suppressed, the bug still happens.

Thanks,
Jim Lake

// Code Snip it

static int DspDebugSpeed(Board *pthis, void *pbuf, u16 len)
{

    u32 temp;
    unsigned long addr;
    u32 junk;
    LDEBUG("Entry");

    memcpy_toio(pthis->pDspMem,pbuf,len);
//    junk = readl(pthis->pDspMem);
    
    LSPAM("Setting DSP Address to 0x%x",0x5001);
    
    mb();
    barrier();
    mb();
    wmb();
    rmb();

    addr = (unsigned long)pthis->pConfigMem + PLX_REG_GPIO;
    
    temp = readl(addr);
    LSPAM("Got Long (0x%x) from addr (0x%lx)",temp,addr);
   
    temp &= ~PLX_DSP_ADDR_DISABLE;
    temp |= PLX_DSP_ADDR_OUTPUT;
    temp &= ~PLX_DSP_ADDR_HIGH;
    
    mb();
    barrier();
    mb();
    wmb();
    rmb();

    addr = (unsigned long)pthis->pConfigMem + PLX_REG_GPIO;
    LSPAM("Writing Data (0x%x) to addr (0x%lx)",temp,addr);
    writel(temp,addr);
    return(0);
}