Thanks for the feedback, Mssrs. Fisk and Nagle,
I think a problem for IT folks who are hearing early statements about SANs
based on GE has to do with an issue to which you both alluded.
Specifically, what parameters -- bandwidth, throughput, latency, etc. --
must designers consider when evaluating or building a storage networking
interconnect?
Put another way, when I design an aircraft, I know about lift, drag and
other engineering parameters and can plug them into calculations that will
enable me to design a wing to lift X number of pounds. When it comes to
storage networks, I cannot get a straight answer from any vendor regarding
the parameters that must be observed or satisfied -- whether stated as
straightforward quantities/formula or general rules of thumb -- in order to
develop a working storage networking interconnect!
What factors determine the amount of bandwidth required?
What amount of latency can be tolerated?
How quickly must erred information be re-sent?
Does this all depend upon the characteristics of the storage traffic itself?
Are the parameters application dependent?
Surely, traditional SCSI bus design delivered a solution that must be
equaled or improved upon by SAN interconnects such as FC, GE or Infiniband
for the latter to be regarded as viable storage interconnects. This can't
be rocket science: Is there a convenient set of storage architecture design
parameters here that I am simply overlooking?
I have no axe to grind with the FC folks, but there seems to be a holy war
shaping up around FC versus GE as a storage interconnect. I am tracking
strategies for TCP offload or ASICS speed-up and agree that the optimization
of TCP/IP functionality to support the use of GE and 10 speed GE as a SAN as
well as a LAN interconnect. I find such a solution to be quite practical,
but I do not say that FC is inferior. There are many roads that lead to
Rome, as the saying goes. What is of concern to me (and to my readers) is
to avoid deploying a technology (FC, for example) that may need to be
"forklift upgraded" within a year.
Please let me know if you are aware of any storage networking design
criteria that must be addressed by any interconnect regardless of its
underlying protocol.
Thanks,
Jon Toigo
Independent Consultant and Author
The Holy Grail of Data Storage Management
jtoigo(_at_)intnet(_dot_)net
----- Original Message -----
From: Dave Nagle <bassoon(_at_)yogi(_dot_)ece(_dot_)cmu(_dot_)edu>
To: Jon William Toigo <jtoigo(_at_)IntNet(_dot_)net>
Cc: <ietf(_at_)ietf(_dot_)org>
Sent: Thursday, May 25, 2000 7:27 PM
Subject: Re: Storage over Ethernet/IP
Jon,
Original Message
----------------
>> I am seeking a few points of clarification:
>>
>> 1. Fibre Channel folks have attempted to explain to me why TCP/IP
could =
>> NEVER be a viable interconnect for block level storage operations.
They =
>> claim:
>> a. TCP is too CPU intensive and creates too much latency for storage
=
>> I/O operations.
>> b. The IP stack is too top heavy and processing packet headers is too
=
>> slow to support storage I/O operations.
There is a lot of work to show that this is not true. Check out Van
Meter's 1998 ASPLOS paper "VISA - Netstations virtual internet SCSI
adaptor."
Perhaps more importantly, there are many companies that are building
TCP in silicon ASICs. This should make TCP's performance comparable
to Fibre Channel. Both TCP/IP and FC provide about the same
functionality ... reliable, in-order transmission.
The bottom line is that FC is done in hardware while TCP has
traditionally been done in software. Therefore, previous performance
numbers are not going to be fair. Once TCP is in silicon, its
performance should be roughly equal to FC.
>> c. The maximum throughput of a GE TCP/IP connection is 768 Mps, which
=
>> is too slow to support storage I/O operations.
I believe there are higher numbers (especially with Jumbo
Frames). Alteon's web site show's 920 Mbps. Microsoft and Duke
University have both shown TCP performance o 1Gb+/s performance over
other networks.
BTW, why is 768 Mbps too slow for storage. Many apps (e.g.,
transaction workloads) are I/O's per second bound, not bandwidth
bound. Also, even if storage over IP/ether is a bit slower than FC,
the benefits of leveraging IP's infrastructure (i.e., routers,
switches, NICs, network management, networking people) is a huge
advantage.
There is also the issue of SCSI over TCP/IP in the SAN vs. the
LAN/WAN. Some companies, focusing on the SAN, are building
SCSI/lightweight transport/IP while others, focusing on the WAN,
propose SCSI/TCP/IP. It may be the case that SAN and WAN traffic use
different transport protocols to gain a bit of extra performance in
the SAN.
>> Is any of this true?
>>
>> 2. Adaptec has posited a replacement for TCP called STP for use as a
=
>> transport for storage. Does anyone know anything about this?
From Paul von Stamwitz's posting to the ips mailing list ...
The link to the SEP draft is
http://www.ietf.org/internet-drafts/draft-wilson-sep-00.txt
The press release is at:
http://www.adaptec.com/adaptec/press/release000504.html
The demo shows a Gb ethernet controller transporting SCSI traffic to
several
targets through an off-the-shelf 100TX switch with a Gb uplink. The
targets
are ethernet to U160 SCSI bridges with one or more SCSI drives
attached. The
host controller runs under NT4.0 at appears to the OS as a SCSI host
bus
adapter.
The architecture is based on Adaptec's SCSI Encapsulation Protocol
(SEP). SEP is mapped on top of TCP/IP or a light-weight transport
protocol specifically designed for SANs.
An SEP overview was presented at the IPS BOF in Adelaide last month
and an
internet draft on SEP was submitted to IETF this week. I will forward
the
link as soon as it becomes available. This draft is informational
only and intended to aid in this group's work toward an industry
standard SCSI transport protocol over IP networks.
>> 3. Current discussions of the SCSI over IP protocol seem to ignore
the =
>> issue of TCP or any other transport protocol. Does anyone know =
>> definitively what transport is being suggested by the IBM/Cisco crowd?
Current SCSI over IP discussions are not ignoring TCP ... they are
definitely considering TCP as the primary transport. See the ips
web site at:
http://www.ece.cmu.edu/~ips
>>
>> 4. Another storage company is looking at Reliable UDP as a substitute
=
>> for TCP in storage data transfers. Where can I learn more about this
=
>> protocol, which I am told was introduced many years ago by Cisco?
Companies to look at include:
nishansystems.com
interprophet.com
san.com
arkresearch.com
Also, I believe that the IETF IP over FC working group is now
looking at FC over IP.
dave...........
David Nagle
Director, Parallel Data Lab
Senior Reseach Computer Scientist
School of Computer Science
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-3898 (office)
412-268-3890 (fax)
http://www.ece.cmu.edu/~bassoon