Re: The IO problem on multiple PCI busses

[Grant Grundler]

Benjamin Herrenschmidt wrote:
> Hi Grant !
> 
> Alan Cox suggested I contact you about this. I'm trying to figure out a
> way to cleanly resolve the problem of doing IO accesses on machines with
> multiple PCI host bridges (and multiple IO bases when IO cycles are not
> generated by the CPU). I'd be glad if you could catch on the 
> "The IO problem on multiple PCI busses" thread on linux-kernel list
> and let us share your point of viw.

To l-k, Benjamin wrote:
| I've looked at the parisc code (thanks Alan for pointing that out), and 
| it seem they implement all inb/outb as quite big functions that decypher 
| the address, retreive the bus, and do the proper IO call. Unfortunately,
| that's a bit bloated, and I don't think I'll ever get other PPC 
| maintainers to agree with such a mecanism (everybody seem to be quite 
| concerned with IO speed, I admit including me).

Benjamin,
As the main author/maintainer of that code, let me explain why
it's so ugly. Hopefully this will give you insight into a "better"
(arch independent) solution. Apologies for the length.

For IO Port space, I didn't worry about the bloat. A nice side effect of
this bloat is it will discourage use of I/O Port space. That's good for
everyone, AFAICT. (I know some devices *only* support I/O port space and
I personnally don't care about them. If someone who does care about one
wants to talk to me about it...fine...I'll help)

[ Caveat: I've simplified the following *alot* to keep it short. ]

parisc supports two different PCI host bus adapters with each having
variants that behave differently. All work under the model we are using
with one binary. One kernel binary is important since we want to make
install's easy for users.

Under Dino (GSCtoPCI), each PCI HBA has it's own 64K I/O port space.
I/O port space transactions are generated by poking registers on Dino.
Yes - performance sucks - that's why HPUX (almost) exclusively
uses devices which support MMIO.

Under Elroy (aka LBA or RopesToPCI), we have two methods of accessing
I/O port space. One view of I/O space can be shared across all Elroy's
which share the same IOMMU (aka SBA). This method distributes the 64K
I/O space over the 8 (or 16) "ropes" with rope 0 getting the first
8k (or 4k) and so on. The other view is each LBA has it's own 64K
of I/O port space. The second view is mapped above 4GB and requires
64-bit kernel to access. In both cases, processor loads/stores from/to
the region will generate an I/O cycle on the respective PCI bus.

Generally speaking, parisc doesn't support VGA or ISA legacy crud on
it's PCI busses. But I think those are orthogonal issues.

The inb/outb support hings on this definition in include/asm-parisc/pci.h:
struct pci_port_ops {
          u8 (*inb)  (struct pci_hba_data *hba, u16 port);
         u16 (*inw)  (struct pci_hba_data *hba, u16 port);   
         u32 (*inl)  (struct pci_hba_data *hba, u16 port); 
        void (*outb) (struct pci_hba_data *hba, u16 port,  u8 data);
        void (*outw) (struct pci_hba_data *hba, u16 port, u16 data);
        void (*outl) (struct pci_hba_data *hba, u16 port, u32 data);
};

Code which uses this is in arch/parisc/kernel/pci.c at:
	http://puffin.external.hp.com/cvs/linux/arch/parisc/kernel/pci.c

(look for PCI_PORT_HBA usage)

In a nut shell, the HBA number is encoded in the upper 16-bits
of the 32-bit I/O port space address. The inb() *function* uses the
decoded HBA number to lookup the matching pci_port_ops function table
and pci_hba_data * to pass in. PCI fixup_bus() code virtualizes the
I/O port addresses found by the generic PCI bus walk. inb() is
function so drivers work under *all* parisc PCI HBAs with one binary.

This scheme allows us to support "PCI-like" busses as well.
Some parisc machines have both PCI and EISA slots which are completely
independent of each other. We'd like to keep the semantics of inb/outb
the same and support both at the same time. It might be possible
to do this by feeding the drivers different versions of inb/outb
definitions at compile time. But initial attempts to do this ran
into problems (which I don't remember the details of).

Last comment is regarding who *configures* the PCI devices. On legacy PDC
(parisc's "BIOS on steriods"), the PDC sets everything up but does
not enable everything (ie pci_enable_device will set bits in PCI_COMMAND
cfg register).  On card-mode Dino, (GSC cards plugged in proprietary bus),
the firmware doesn't know *anything* about the PCI devices and the arch
support has to set everything up - PCI MMIO space is not currently
supported there. And new servers (like L2000 or A500) with "PAT PDC" only
initialize PCI devices for boot. OS has to initialize the rest.

grant

Grant Grundler
parisc-linux {PCI|IOMMU|SMP} hacker
+1.408.447.7253