Solaris 10’s new multipath storage tool (Part 1)

Solaris 10 11/06 was released in late 2006 with a plethora of new features, and among them a new tool called mpathadm, which comes as part of the SUNWmpathadm package.

Before I delve into how this specific tool works and how it helps when managing multipathed storage in Solaris, I’ll give some background on what multipathing is and how it is implemented in Solaris.

What is multipathing?

Multipathing is a generic term relating to a certain SAN configuration where a host on that SAN (ie; a server) has multiple physical paths to a target (ie; a disk array) on said SAN. A common multipath configuration consists of a server with multiple fibre channel ports, and each port is connected to a separate fibre channel switch in the SAN. Optionally, the target device or disk array may also have multiple physical links in the same manner. In the end, though, the host “sees” multiple instances of the same target – once through each connection to the SAN. It’s up to the host to then manage this. The end result is higher reliability. If a switch were to go down or the optical cable between the host and a switch were damaged, the host’s connectivity to its SAN disks would not be interrupted. IO traffic is also multiplexed over the multiple links, thus providing fail-over and higher throughput.

How multipathing works in Solaris

Multipathing has existed in Solaris since Solaris 8 via an add-on product called Sun SAN Foundation Kit. With the release of Solaris 10, all functionality of the SAN Foundation Kit was integrated into the OS and no longer requires an add-on to function.

When multipathing is enabled, the scsi_vhci driver looks for multiple instances of the same target (eg; a LUN) and upon finding one, creates a “meta” disk device which it manages. IO to and from this device are then multiplexed over however many paths scsi_vhci sees to the LUN.

Below is an example of a host which has two fibre channel controllers (c4 and c5) connected to the sam SAN, and both see the same target LUNs, of which there are four:

[root@ackpthh]/>cfgadm -al -o show_SCSI_LUN
Ap_Id                          Type         Receptacle   Occupant     Condition
c4                             fc-fabric    connected    configured   unknown
c4::60003930000153a5,0         disk         connected    configured   unknown
c4::60003930000153da,0         disk         connected    configured   unknown
c4::60003930000153eb,0         disk         connected    configured   unknown
c4::6000393000015463,0         disk         connected    configured   unknown
c5                             fc-fabric    connected    configured   unknown
c5::60003930000153a5,0         disk         connected    configured   unknown
c5::60003930000153da,0         disk         connected    configured   unknown
c5::60003930000153eb,0         disk         connected    configured   unknown
c5::6000393000015463,0         disk         connected    configured   unknown

In the above cfgadm output, the name of the LUNs are in the following format:

So, the aforementioned scsi_vhci driver sees that LUN 6000393000015463,0 is presented by two controllers. It will then make a new disk device using that device’s WWN, LUN, and other information to construct a globally-unique ID (GUID) device name and then create device entries under /dev/dsk and /dev/rdsk to represent a multipathed view:

[root@ackpthh]/>ls -l /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0*
lrwxrwxrwx   1 root     root          69 Jul  6  2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0 -> ../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:wd,raw
lrwxrwxrwx   1 root     root          68 Jul  6  2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s0 -> ../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:a,raw
lrwxrwxrwx   1 root     root          68 Jul  6  2006 /dev/rdsk/c6t600039300001546301000000D52ACC7Ed0s1 -> ../../devices/scsi_vhci/disk@g600039300001546301000000d52acc7e:b,raw
...

You might notice the new controller, c6. This is a “virtual” controller implemented by scsi_vhci since making such virtual disk devices under a real physical controller would be a broken idea since, technically, there is no such physical device. Remember, this is a meta device.

A note about those big, long multipath device names: there’s actually a format to them, so don’t let their length turn you off. The format is as follows, using the above device as an example:

We can now use the multipath device as we would any other kind of disk device, with format for example:

[root@ackpthh]/>format
Searching for disks...done

c6t600039300001546301000000D52ACC7Ed0: configured with capacity of 2794.54GB


AVAILABLE DISK SELECTIONS:
       0. c0t2d0 <DEFAULT cyl 8921 alt 2 hd 255 sec 63>
          /pci@0,0/pci1022,7450@2/pci1000,3060@3/sd@2,0
       1. c6t60003930000153A501000000D52AE0C4d0 <APPLE-Xserve RAID-1.50-2.73TB>
          /scsi_vhci/disk@g60003930000153a501000000d52ae0c4
       2. c6t600039300001546301000000D52ACC7Ed0 <APPLE-Xserve RAID-1.50-2.73TB>
          /scsi_vhci/disk@g600039300001546301000000d52acc7e
       3. c6t60003930000153DA01000000D767690Ed0 <APPLE-Xserve RAID-1.50-1.82TB>
          /scsi_vhci/disk@g60003930000153da01000000d767690e
       4. c6t60003930000153EB01000000D767D3C0d0 <APPLE-Xserve RAID-1.50-2.73TB>
          /scsi_vhci/disk@g60003930000153eb01000000d767d3c0
Specify disk (enter its number): 

The next part will cover mpathadm itself and how it fits into managing these devices.