Re: [ANNOUNCE] Dolphin PCI-SCI RPM Drivers 1.1-4 released

[Jeff V. Merkey]

On Mon, Jan 29, 2001 at 09:41:21PM -0700, Todd wrote:

Todd,

I ran the tests on a box that has a limit of 70MB/S PCI throughput.  
There are GaAs (Gallium Arsenide) implementations of SCI that run 
well into the Gigabyte Ranges, and I've seen some of this hardware.  
The NUMA-Q chipsets in Sequents cluster boxes are actually SCI.  
Sun Microsystems uses SCI as their clustering interconnect for their 
Sparc servers.  The adapters these drivers I posted support are a bi-CMOS 
implementation of the SCI LC3 chipsets, and even though they are 
bi-CMOS, the Link speed on the back end is still 500 MB/S --
very respectable.

The PCI-SCI cards these drivers support in PCI systems have been clocked up 
to 140 MB/S+ on those systems that have enough bandwidth to actually 
push this much data.  These boards can pump up to 500MB/S over the 
SCI fabric, however, current PCI technology doesn't allow you to 
push this much data.  I also tested on the D320 chipsets, the newer 
D330 chipsets on the PSB66 cards support the 66Mhz bus and have been 
measured up to the Max PCI speeds.

The PCI-SCI adapters run circles around G-Enet on systems that can 
really pump this much data through the PCI bus.  Also, the numbers I 
posted are doing push/pull DMA transfers between user_space -> user_space 
in another system with **NO COPYING**.  Ethernet and LAN networking always 
copies data into userspace -- SCI has the ability to dump it directly 
into user space pages without copying.  That's what is cool about SCI, 
you can pump this data around with almost no processor utilization -- 
important on a cluster if you are doing computational stuff -- you need 
every cycle you can squeeze, and don't want to waste them copying 
data all over the place.  Sure, G-Enet can pump 124 MB/S, but the 
processor utilitzation will be high, and there will be lots of 
copying going on in the system.   What numbers does G-Enet provide 
doing userspace -> userspace transfers, and at what processor 
overhead?  These are the types of things that are the metrics for 
a good comparison.  Also, G-Enet has bandwidth limitations, the 
SCI standard does not, it's only limited by the laws of Physics
(which are being reached in the Dolphin Labs in Norway).

The GaAs SCI technology I have seen has hop latencies in the SCI 
switches @ 16 nano-seconds to route a packet, with xfer rates into
the Gigabytes per second -- very fast and low latency.   

These cards will use whatever PCI bandwidth is present in the host 
system, up to 500 MB/S.  As the PCI bus gets better, nice to know 
SCI is something that will keep it's value, since 500 MB/S gives us 
a lot of room to grow into.    

I could ask Dolphin for a GaAs version of the LC3 card (one board would
cost the equivalent to the income of a small third world nation), and 
rerun the tests on a Sparc system or Sequent system, and watch G-Enet
system suck wind in comparison.  

:-)

I posted the **ACCURATE** numbers from my test, but I did clarify that I 
was using a system with a limp PCI bus.

Jeff

> folx,
> 
> i must be missing something here.  i'm not aware of a PCI bus that only
> supports 70 MBps but i am probably ignorant.  this is why i was confused
> by jeff's performance numbers.  33MHz 32-bit PCI busses should do around
> 120MB/s (just do the math 33*32/8 allowing for some overhead of PCI bus
> negotiation), much greater than the numbers jeff is reporting.  66 MHz
> 64bit busses should do on the order of 500MB/s.
> 
> the performance numbers that jeff is reporting are not very impressive
> even for the slowest PCI bus.  we're seeing 993 Mbps (124MB/s) using the
> alteon acenic gig-e cards on 32-bit cards on a 66MHz bus.  i would expect
> to get somewhat slower on a 33MHz bus but not catastrophically so
> (certainly nothing as slow as 60MB/s or 480Mb/s).
> 
> what am i misunderstanding here?
> 
> todd
> 
> On Mon, 29 Jan 2001, Jeff V. Merkey wrote:
> 
> > Date: Mon, 29 Jan 2001 16:49:53 -0700
> > From: Jeff V. Merkey <[e-mail]>
> > To: [e-mail]
> > Cc: [e-mail]
> > Subject: Re: [ANNOUNCE] Dolphin PCI-SCI RPM Drivers 1.1-4 released
> >
> >
> > Relative to some performance questions folks have asked, the SCI
> > adapters are limited by PCI bus speeds.  If your system supports
> > 64-bit PCI you get much higher numbers.  If you have a system
> > that supports 100+ Megabyte/second PCI throughput, the SCI
> > adapters will exploit it.
> >
> > This test was performed in on a 32-bit PCI system with a PCI bus
> > architecture that's limited to 70 MB/S.
> >
> > Jeff
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to [e-mail]
> > Please read the FAQ at http://www.tux.org/lkml/
> >
> 
> =========================================================
> Todd Underwood, [e-mail]
> 
> criticaltv.com
> news, analysis and criticism.  about tv.
> and other stuff.
> 
> =========================================================