Storage Area Network (SAN)

Outline
Shared Storage Architecture
Direct Access Storage (DAS)
SCSI
RAID
Network Attached Storage (NAS)
Storage Area Network (SAN)
Fiber Channel and
Fiber Channel Switch
2

The SNIA Model

SNIA Storage Networking Industry
Association
SNIA is a framework that captures
the functional layers and properties
of a storage system
Trying to become an industry
standard

The SNIA shared storage

model
Storage domain

Application

File/record layer
Database
(dbms)

File system
(FS)

Host
Network

Block
aggregation

Device

Storage devices (disks, )

Block layer
4

The SNIA storage model

A layered view

Storage Trend and Demand

2010+

40G/100G SAN and LAN

6

Three Basic Forms of

Network Storage
Direct access storage (DAS)
Network attached storage (NAS)
Storage area network (SAN)
And a number of variations on each
(especially the last two)

Quick Overview
Storage
Type
Data
Transmissio
n

DAS

NAS

SAN

sectors

shared
files

blocks

TCP/IP,
IDE/SCSI
Ethernet

Fibre
Channel

Access
Mode

clients
or
servers

clients
or
servers

servers

Capacity
(bytes)

109

109 - 1012

1012

Complexity

Easy

Moderat
e

Difficult

DAS

NAS

FC-SAN

clients

servers

FC
Switch

storage
9

Direct Access Storage (DAS)

Ethernet
Network

Used
IDE Disk Array

Small Server

SCSI
Channel

clients

Large Server

Used
Used

SCSI Disk Array

10

Small Computer System Interface

(SCSI)
From Shugarts 1979 SASI implementation
An I/O bus for peripheral device, such as hard drives, tape
drives, CD-ROM, scanners, etc.
an improvement over IDE
A single SCSI bus connects multiple elements (max 7 or 15).
High speed data transfer:
5, 10, 20, 100, 320MB/sec,
Overlapping I/O capability:
Multiple read & write commands can be outstanding simultaneously
Different SCSI drives to be processing commands concurrently rather
than serially. The data can then be buffered and transferred over the
SCSI bus at very high speeds
11

SCSI Distribution
Architecture

SCSI is a client/server architecture.

The client is called the initiator and issues request to
the server. The client is I/O subsystem under the
typical OS control.
The server is called the target, which is the SCSI
controller inside the storage device. It receives,
process, and responds to the requests from the initiator.
SCSI commands support block I/O, transferring large
amount of data in blocks.
request

Client
(Initiator)

response

Storage Device
(Target)

12

SCSI Client/Server
Architecture
Client
(Host)

Server
(Storage
Device)

13

SCSI Block I/O Operation

14

SCSI Transport Mechanism

SCSI Applications (File Systems, Databases)
SCSI Applications (File Systems, Databases)
SCSI
Device-Type
Commands
SCSI
Generic
Commands
SCSI
Transport
Protocols

SCSI Commands (Block, Stream, etc.)

SCSI Commands (Block, Stream, etc.)
SCSI Commands, Data, and Status
SCSI Commands, Data, and Status
Parallel
SCSIParallel
Transport
SCSI Transport

FCP
FCP FC
SCSI over
SCSI over FC

Network
Transport

iSCSI
iSCSI
SCSI over
TCP/IP
SCSI over TCP/IP
TCP
TCP
IP
IP

Physical
interface

Parallel SCSI
Parallel
SCSI
Interfaces
Interfaces

Fibre Channel
Fibre Channel

Ethernet
Ethernet
15

SCSI Parallel Interface

SCSI Domain
SCSI Service Delivery Subsystem

16

Redundant Array of Independent Disks (RAID)

A group of hard disks is called a disk array
RAID combines a disk array into a single virtual
device
called RAID drive

Provide fault tolerance for shared data and

applications
Different implementations: Level 0-5
Characteristics:
Storage Capacity
Speed: Fast Read and/or Fast Write
Resilience in the face of device failure
17

RAID Functions

Striping
Write consecutive logical byte/blocks on consecutive physical disks

Mirroring
Write the same block on two or more physical disks

Parity Calculation
Given N disks, N-1 consecutive blocks are data blocks, Nth block is for
parity
When any of the N-1 data blocks is altered, N-2 XOR calculations are
performed on these N-1 blocks
The Data Block(s) and Parity Block are written
Destroy one of these N blocks, and that block can be reconstructed
using N-2 XOR calculations on the remaining N-1 blocks
Destroy two or more blocks reconstruction is not possible

18

Disk Striping (example)

Example 1: 1 0 1 0 1 1

1 1 1
disk 1: odd bits

001

110

disk 2: even bits

parity bits (even parity)

Example 2: 1 0 1 0 1 1

1 0
3k+1 bits

01
3k+2 bits

11
3k bits

11

parity bits (odd parit

19

RAID Types
RAID 0
Stripe with no parity (see next slide for figure)

RAID 1
Mirror two or more disks

RAID 0+1 (or 1+0)

Stripe and Mirrors

RAID 3
Synchronous, Subdivided Block Access; Dedicated
Parity Drive

RAID 5
Like RAID 4, but parity striped across multiple drives
20

RAID 0

RAID
1

Disk Striping (no redundancy)

Disk Mirror

21

RAID
0+1
(or 1+0)

22

RAID 3

RAID
5

Disk striping with Dedicated Parity Disk

Drivestriping with Distributed Parity Data

23

Striping (parity) data is duplicate.

24

Network Attached Storage

(NAS)
NAS is adedicated storage device, and it operates
in a client/server mode.
NAS is connected to the file server via LAN.
Protocol: NFS (or CIFS) over an IP Network
Network File System (NFS) UNIX/Linux
Common Internet File System (CIFS) Windows Remote file
system (drives) mounted on the local system (drives)
evolved from Microsoft NetBIOS, NetBIOS over TCP/IP (NBT), and Server
Message Block (SMB)

SAMBA: SMB on Linux (Making Linux a Windows File Server)

Advantage: no distance limitation

Disadvantage: Speed and Latency
Weakness: Security
25

SMB
NetBIOS

TCP
IP
802.
3

26

NFS
TCP
IP
802.3
27

Network Attached Storage

(NAS)
Specialized storage device or group of storage devices
providing centralized fault-tolerant data storage for a network

Clients

Servers

Storage Devices
28

Case Study
Product: MicroNet
ProtinumNAS
Storage: 1TB and more
Price: < $1,000
Protocol: CIFS/SMB,
RAID

29

Discussion
Need: a lot more storage (hundreds of GB)
and a scalable solution (~2 TB) for home
users
USB (USB 2.0) to a server, up to 480M bps
Firewire (IEEE 1394) to a server, up to 3.2G bps
SCSI to a server: up to 320MB (3208 bps)
NAS: no need for a server

Q: What is your choice?

30

Storage Area Network (SAN)

A Storage Area Network (SAN) is a specialized,
dedicated high speed network joining servers
and storage, including disks, disk arrays, tapes,
etc.
Storage (data store) is separated from the
processors (and separated processing).
High capacity, high availability, high scalability,
ease of configuration, ease of reconfiguration.
Fiber Channel is the de facto SAN networking
architecture, although other network standards
could be used.
31

SAN Benefits

Storage consolidation
Data sharing
Non-disruptive scalability for growth
Improved backup and recovery
Tape pooling
LAN-free and server-free data movement
High performance
High availability server clustering
Data integrity
Disaster tolerance
Ease of data migration
Cost-effectives (total cost of ownership)
32

NAS vs. SAN ?

Traditionally:
NAS is used for low-volume access to a large
amount of storage by many users
SAN is the solution for terabytes (1012) of
storage and multiple, simultaneous access to
files, such as streaming audio/video.

The lines are becoming blurred between

the two technologies now, and while the
SAN-versus-NAS debate continues, the
fact is that both technologies complement
each another.
33

Fibre Channel
Fiber Channel is well established in the open
systems environment as the underlining
architecture of the SAN.
Fibre Channel is structured with independent
layers, as are other networking protocols.
There are five layers, where 0 is the lowest
layer. The physical layers are 0 to 2. These
layers carry the physical attributes of the
network and transport the data created by
the higher level protocols, such as SCSI,
TCP/IP, or FICON.
34

FC Standard ANSI T11

T11 (technical committee) has been producing
interface standards for high-performance and
mass storage applications since the 1970s.
http://www.t11.org/index.htm

Designed to transport multiple protocols, such

as HIPPI, IPI, SCSI, IP, Ethernet, etc.
Full duplex medium
Channels are established between the
originator and the responder.
Transfer rate from 100MB/s to Gigabits/s
Distance >10 km (single mode fiber)
Multi-layer stack functions (not mapped to the
OSI model)
35

FC Protocol Layers

Gbau
d

Gbau
d

IPI: Intelligent Peripheral Interface

HIPPI: High Performance Parallel Interface
SCSI
SBCCS: Single Byte Command Code Set
36

FC Layers: 0 & 1

37

FC Layer 2

Port_ID Port_ID

38

FC Address
FC node a node has many ports
FC port the end point of a link (either
transmission or reception).
Port ID: a unique 24-bit address for a
port
In Frame Header (see Slide-49), there
are two fields: Source address
(transmission port) and Destination
address (reception port)
39

FC Naming and Addressing

Each node normally has one physical interface ,
known as N_Port.
Each node has an 8-byte node name.
Assigned by manufacturer
If registered with IEEE, it is known as World Wide Name.

N_Port ID: 24-bit port address

An N_Port has a point-to-point connection with
another N_Port.
An N-Port may be attached to a fabric port, F_port.
Connection between fabric switches is via
expansion ports, E_ports.
A switch port, if configured for either one, is a
generic port, G_Port.
40

FC Port Naming
Fiber Channel
Host

N-Port

Fiber Channel
Switch

F-Port

E-Port

Fiber Channel
Switch

E-Port

F-Port

Fiber Channel
Stores

N-Port

Node port, fabric port, expansion port, generic port

41

FC Layers 3 & 4

(one)

42

SAN Topologies
Fibre Channel based networks
support three types of topologies:
Point-to-point
Loop (arbitrated) shared media
Switched

43

FC - Point-to-Point

The point-to-point topology is the easiest

Fibre Channel configuration to implement,
and it is also the easiest to administer.
The distance between nodes can be up to
10 km
44

Data Access over FC

Data

Data

SCSI

SCSI

FC

FC

45

Arbitrated Loop
Shared Media Transport
Similar in concept to shared Ethernet

Not common for FC-based SAN

Commonly used for JBOD (Just a
Bunch of Disks)
An arbitration protocol determines
who can access the media.
ARB primitive
46

Arbitrated Loop (Daisy

Chain)
Rx

Tx

Tx

Rx

Tx
Rx
Tx

Rx

47

FC Arbitrated Loop (FC Hub)

48

RAID Controller

RAID Controller

RAID, SCSI, and Fibre

Channel

SCSI Disks

Fibre Channel
Loop

49

Switched FC SAN
Fibre Channel-switches function in a manner
similar to traditional network switches to provide
increased bandwidth, scalable performance, an
increased number of devices, and, in some
cases, increased redundancy. Fibre Channelswitches vary in the number of ports and media
types they support.
Multiple switches can be connected to form a
switch fabric capable of supporting a large
number of host servers and storage subsystems

50

FC Switched SAN
Servers

Fiber Channel
Switch

Clients

Fiber Channel
Stores

51

Data Access over Switched

SAN
Fiber Channel
Servers

Switch

Data

Storage
Device

Data

SCSI

SCSI

SCSI

FC

FC

FC

52

FC - Storage Area Network

(redundant architecture)
Servers

Fiber Channel
Switch

Clients

Fiber Channel
Stores

53

Repeat Overview
Storage
Type
Data
Transmissio
n

DAS

NAS

SAN

sectors

shared
files

blocks

TCP/IP,
IDE/SCSI
Ethernet

Fibre
Channel

Access
Mode

clients
or
servers

clients
or
servers

servers

Capacity
(bytes)

109

109 - 1012

1012

Complexity

Easy

Moderat
e

Difficult

54

IP-based Storage Area Networks

55

Course Outline
IP over FC (RFC 2625)
IP-SAN
iSCCI (RFC 3720)

IP and FC-SAN Interworking

FC Encapsulation (RFC 3643)
FCIP (RFC 3821) FC over IP
iFCP (RFC 4172)

Storage Virtualization
56

RFC 2625 IP and ARP

over Fiber Channel (FC)
FC supports multiple higher layer
protocols, and SCSI is the most widely
used one.
What about IP over FC?
Access data in SAN from IP-based servers
interworking between NAS and SAN

RFC 2625 addresses two issues.

A scheme to encapsulate IP and ARP packets
inside the FC frame (as the FC payload)
A procedure to resolve the address mapping
57

IP over FC (RFC 2625)

App-1: accessing SAN from IP-based servers

SAN
FC-based
Storage
Device
Data

FC/IP
Gateway
Data
IP

IP

IP

IP

L2

L2

PHY

PHY

RFC
2625
FC

RFC
2625
FC

FC
SAN

58

IP over FC (RFC 2625)

(App-2: interworking between SAN and NAS)
SAN
NAS-based
Storage
Device

FC/IP
Gateway

FC/IP
Gateway
Data

Data

IP

IP

IP

IP

IP

IP

L2

L2

L2

PHY

RFC
2625
FC

L2

PHY

RFC
2625
FC

PHY

PHY

FC
SAN

59

IP-SAN

60

Advantages of IP for SAN

61

IP Network Capabilities

62

IP-SAN Protocols

63

IP - SAN
IP

IP
FC
-SAN

Storage Devices

64

IP - Storage Area Network

(SAN)

IP storage networking carrying storage

traffic over IP
Uses TCP, a reliable transport for delivery
Can be used for local data center and long
haul applications
Two primary IETF protocols/standards:
iSCSI Internet SCSI
allows
block storage to be
IP
IP
TCP
TCP iSCSI
iSCSI
SCSI
SCSI Data
Data
accessed
over a TCP/IP network as though it were locally
attached

IP
IP

TCP
TCP

FCIP
FCIP FC
FC SCSI
SCSI

Data
Data

FCIP Fibre-Channel-over-IP used to tunnel

65

Internet SCSI (iSCSI)

iSCSI is a proposed industry standard that allows SCSI
block I/O protocols (commands, sequences, and
attributes) to be sent over a network using the
popular TCP/IP protocol.
A way to access storage across an IP network as
though it was locally attached.
Transports SCSI protocol commands and data across
an IP network
Cisco and IBM co-authored original iSCSI protocol draft
iSCSI Protocol is a standard maintained by the IETF
IP Storage (IPS) Working Group
RFC 3720

66

iSCSI Benefit

67

Is IP-SAN similar to NAS?

What are the
advantages, if any, of
IP-SAN vs. NAS?

68

SAN, NAS, and IP-SAN

IP

iSCSI
IP
FC
-SAN

iSCSI
IP

iSCSI

69

Performance Analysis: iSCSI vs.

NAS
(software based no HBA)

iSCSI

NFS
70

Sequential Read/Write Tests

Conclusion:
1. Comparable performance in character read/write and
2. Significant advantage of iSCSI in block read (20-25%
71

Small Files Read/Write

72

IOGen Test (Emulation of

Database)

73

FC-SAN vs. iSCSI

Since the iSCSI appliance attaches to the existing Ethernet
network, NAS and iSCSI are very similar in network
architecture
However, the performance would be significantly different.

Both iSCSI and SAN use Block I/O to transport data, whereas
NAS uses File I/O.
SAN offers better performance (c.f. NAS), but is more
expensive and requires a higher skill set to implement. iSCSI
and NAS offer better pricing and skills may already be in
place to implement them.
Both SAN and iSCSI offer the performance benefit of Block I/O.

74

FC over IP
SAN

Data

IP

SAN

an IP tunnel for FC-based SAN

Application: interconnect SAN over IP-WAN.

SCSI
FCP
FC 0-2

Data
SCSI

FCIP
FC 02

FCIP

TCP

TCP

IP

IP

IP

L2

L2

L2

PHY

PHY

PHY

FC 02

FCP
FC 0-2

75

Storage Virtualization
PHYSICAL

LOGICAL
Virtualization

FC
-SAN

IP

Logical storage Pool

(Direct Attached Storage)
JBOD
RAID
76

Storage Virtualization
Definition: storage virtualization hides the
physical storage from applications on host
systems, and presents a simplified (logical)
view of storage resources to the applications.
Virtualization allows the application to
reference the storage resource by its common
name where the actual storage could be on a
complex, multilayered, multipath storage
networks.
RAID is an early example of storage
virtualization.

77

Virtualization Intelligence
Host-Based: storage virtualization could be
implemented on the host through Logical Volume
Management (LVM) which provides the logical view of
the storage to the host operating system.
Switch-based: intelligence of storage virtualization
could be implemented on the SAN switches. Each
server is assigned a Logical Unit Number (LUN) to
access the storage resources.
Switch-based virtualization could be in dual configuration for
high availability.
Pros: ease of configuration and management ;
redundancy/high availability
Cons: potential bottleneck on the switch; higher cost

78

Storage Virtualization
LVM

LVM

LUN
SAN Switch

RAID

RAID

JBOD

RAID

LUN SAN Switch

w/ Virtualization
Intelligence

RAID

JBOD
79

SAN Challenges
Standards
ANSI T10 (SCSI) ANSI T11 (FC), IETF (IP-SAN),
Ethernet (IEEE 802.3), SNIA, etc.

Interoperability
High availability and data synchronization
between remote locations
Convergence
DAS, NAS, FC-SAN => IP-SAN

Management
Security

80

Summary
Needs for large storage continual growth

109 (G) => 1012 (T) => 1015 (P) => 1018 (E) .

From dedicated solution to network-based

solution
DAS => NAS => SAN => IP-SAN

Convergence of SAN and IP-LAN/WAN

It is an IP world!

SCSI is the protocol for block data

transmission
SCSI over FC - legacy
SCSI over IP (iSCSI)

FC and IP interworking protocols

IP over FC
FC over IP (FCIP) and iFCP

81