There is a group of protocols that you gonna find on the point of contact of LAN, SAN and Data Center, namely : Ethernet, iSCSI, Fibre Channel or FCoE. Some vendors push one of them over the others, but knowing what they are, how do they work and were are placed in network infrastructure is crucial.
Let’s glance on below diagram. This diagram consists of 3 blocks : LAN (Campus Network), SAN (Storage) and DATA CENTER (Server). In the middle of them is placed “not specified” device, in this case it is desirable. I don’t want to focus on any particular vendor, because everything depends on what protocols we are going to use. If you use Cisco Unified Computing System you come across FCoE on the point of contact of Server and Storage, but different manufacturer may prefer iSCSI and then you are going to find the ethernet switch . I am not going to focus on which one is better, faster or more reliable in this article. Every protocol has upsides and downsides. I will talk them over and how do they work in short.
ETHERNET – the most common and known protocol. We distinguish 2 types of Ethernet standards : Ethernet v2 and 802.3 .Ethernet Is placed on Layer 1 (Ethernet Physical Layer) and Layer 2 (Data Link Layer). Layer 2 is split on 2 sub-layers : Logical Link Control (framing and addressing) and MAC – Media Access Control (48 bits hard coded address). Ethernet is used in every LAN network (IP),works in star topology (switched networks) and uses NIC (Network Interface Card). Ethernet v2 standard relies on Ethertype field. The size of MTU is 1500 bytes.
FIBRE CHANNEL – in above example we are talking about switched topology which is placed in SAN block. Instead of having MAC addresses we have WWNs (World Wide Names) – 64 or 128 bits hex numbers. WWNs are divided on : Nodes WWN and Ports WWN. Fibre Channel allows us to have 16 milion nodes in our fabric topology and works with speed up to 16Gb/s (but Brocade finally released 32 and 128 Gb switch G620 ). Fibre Channel can carry other protocols like IP or SCSI and uses special HBA (Host Bus Adapter). Total MTU for FC frame is 2240 bytes. If you want to find out more about FC visit another post http://itbundle.net/archives/1935
iSCSI – carries SCSI commands over IP network. iSCSi is being encapsulated into ethernet frames. Big advantage is that we don’t have to spend a lot of money on infrastructure, even older switches will manage with a traffic, because these are simple ethernet frames. iSCSI may works with 10Gb, 40Gb and 100G/s of speed, of course the cost of infrastructure increases dramatically
FCoE (Fibre Channel over Ethernet) – FC protocol has been encapsulated in ethernet frames. Why? Among others in order to reduce needed resources (cards, cables etc). Below are some characteristics of FCoE:
– reduction of adapters NIC and HBA are swapped by single CNA (Converged Network Adapter) which can carry Ethernet and FC traffic on the same port. CNA contains drivers for each protocol which encapsulate frames as necessary. It is done based on vNIC (virtual NIC adapter) and vHBA (virtual Host Bus Adapters) which are used by given service or Virtual Machine. Usually Cisco CNA – VIC cards support up to 256 virtual vNICs and vHBAs
– simplifying cabling (uses 10Gb Ethernet) but if we use for example fabric extender 2208XP we get 160Gb throughput.
– layer 2 domains-multipath (Virtual port channel + Fabric Path). Two FCF (Fabric Channel Forwarder) switches (i.e Nexus) are visible as one device what eliminates STP loops
– uses jumbo frames with MTU 9000 bytes
– mapping pWWNs to MAC addresses
– FIP – Fibre Channel over Internet Initialization Protocol
– is losseless
– uses ethertype 0x8906
FCoE is highly supported by Cisco and their Unified Computing System. Below I put a simple diagram from Cisco site, how UCS works and which parts it consist of. As you can see the key element is Fabric Interconnect which provides connectivity between LAN, SAN and DATA CENTER.
Uplink Ports are going towards LAN
FC Uplink Ports are going towards SAN
Server Ports are going towards UCS Blade Server