多路径软件multipath.conf配置详解

Multipath简称为多路径访问，主要功能就是和存储设备一块儿配合实现三大功能： 1. 故障的切换和恢复 2. IO流量的负载均衡 3. 磁盘的虚拟化

1、multipath在redhat 7.2中的基本配置：

wKioL1hck-_A9WpgAADUF9Ah-cM128.png

从上面的提示能够看出，默认没有/etc/multipath,DM模块没有加载，在redhat linux 7.2能够用命令/sbin/mpathconf/ --enable 生成配置文件

在/etc/multipath.conf配置文件中内容是被注释掉的

wKiom1hclDyx0blSAAEcSNr1nl0957.png

全部要注意些，要修改的地方去掉注释；

为节省时间，下面有6.2的配置详情，和7.2差异不大

1、multipath在redhat 6.2中的基本配置：

1. 经过命令：lsmod |grep dm_multipath 检查是否正常安装成功。若是没有输出说明没有安装那么经过yum功能安装一下软件包：yum –y install device-mapper device-mapper-multipath

接着经过命令：multipath –ll 查看多路径状态查看模块是否加载成功

[root@liujing ~]# multipath –ll 查看多路径状态

Mar 10 19:18:28 | /etc/multipath.conf does not exist, blacklisting alldevices.

Mar 10 19:18:28 | A sample multipath.conf file is located at

Mar 10 19:18:28 |/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf

Mar 10 19:18:28 | You can run /sbin/mpathconf to create or modify/etc/multipath.conf

Mar 10 19:18:28 | DM multipath kernel driver notloaded ----DM模块没有加载

若是模块没有加载成功请使用下列命初始化DM,或重启系统
---Use the following commands to initialize and start DM for the first time:
modprobe dm-multipath
modprobe dm-round-robin
service multipathd start
multipath –v2

初始化完了以后再经过multipath -ll命令查看是否加载成功

[root@liujing ~]# multipath -ll

Mar 10 19:21:14 | /etc/multipath.conf does not exist, blacklisting alldevices.

Mar 10 19:21:14 | A sample multipath.conf file is located at

Mar 10 19:21:14 |/usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf

Mar 10 19:21:14 | You can run /sbin/mpathconf to create or modify/etc/multipath.conf

DM multipath kernel driver not loaded ----这个提示没了说明DM模块已加载成功。

从上面的提示能够看到，DM模块是成功加载，可是/etc/下没有multipath.conf 配置文件，下一步介绍如何配置multipath.conf 文件。

2. 配置multipath：

经过vi命令建立一个Multipath的配置文件路径是/etc/multipath.conf ,在配置文件中添加multipath正常工做的最简配置以下：

vi /etc/multipath.conf

blacklist {

devnode "^sda"

}

defaults {

user_friendly_names yes

path_grouping_policy multibus

failback immediate

no_path_retry fail

}

编辑完成后保存配置，同时经过命令：

/etc/init.d/multipathd start #开启mulitipath服务

若是出现没法开启服务的状况，没有提示OK的话以下：

[root@liujing mapper]# service multipathd start

Starting multipathd daemon: 没有提示OK

从新开关一下服务就能够解决了。

[root@liujing mapper]# /etc/init.d/multipathd stop

Stopping multipathddaemon: [ OK ]

[root@localhost mapper]# /etc/init.d/multipathd start

Starting multipathddaemon: [ OK ] -----提示OK 正常开启服务

经过命令查看：

[root@liujing mapper]# multipath -ll

mpatha (360a9800064665072443469563477396c) dm-0NETAPP,LUN ----建立了一个lun

size=3.5G features='0' hwhandler='0' wp=rw

`-+- policy='round-robin 0' prio=4 status=active

|- 1:0:0:0 sdb 8:16 active ready running ----多路径下的两个盘符sdb和sde.

`- 2:0:0:0 sde 8:64 active ready running

目录/dev/mapper/ 下多了两个文件夹mpatha 和mpathap1。

[root@liujing mapper]# cd /dev/mapper/

[root@liujing mapper]# ls

control mpatha mpathap1

同时fdisk –l的命令下也多了两个设备标识：

没有配置多路径时：

[root@liujing~]# fdisk -l

Disk /dev/sda: 146.8 GB, 146815733760 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000a6cdd

DeviceBoot Start End Blocks Id System

/dev/sda1 * 1 26 204800 83 Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2 26 287 2097152 82 Linuxswap / Solaris

Partition 2 does not end on cylinder boundary.

/dev/sda3 287 17850 141071360 83 Linux

Disk /dev/sdb: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

DeviceBoot Start End Blocks Id System

/dev/sdb1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/sde: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

DeviceBoot Start End Blocks Id System

/dev/sde1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

两个CAN网卡获取到同一盘符：

/dev/sde和/dev/sdb.

配置后多了/dev/mapper/mpatha和/dev/mapper/mpathap1：

[root@localhost mapper]# fdisk -l

Disk /dev/sda: 146.8 GB, 146815733760 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000a6cdd

DeviceBoot Start End Blocks Id System

/dev/sda1 * 1 26 204800 83 Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2 26 287 2097152 82 Linuxswap / Solaris

Partition 2 does not end on cylinder boundary.

/dev/sda3 287 17850 141071360 83 Linux

Disk /dev/sdb: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

DeviceBoot Start End Blocks Id System

/dev/sdb1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/sde: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/sde1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/mpatha: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

DeviceBoot Start End Blocks Id System

/dev/mapper/mpathap1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/mpathap1: 3773 MB, 3773441024 bytes

255 heads, 63 sectors/track, 458 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Alignment offset: 1024 bytes

Disk identifier: 0x00000000

Disk /dev/mapper/mpathap1 doesn't contain a valid partition table

multipath -F #删除现有路径 两个新的路径就会被删除
multipath -v2 #格式化路径 格式化后又出现

3. multipath磁盘的基本操做

要对多路径软件生成的磁盘进行操做直接操做/dev/mapper/目录下的磁盘就行.

在对多路径软件生成的磁盘进行分区以前最好运行一下pvcreate命令:

pvcreate /dev/mapper/mpatha

#fdisk /dev/mapper/mpatha 分区时用这个目录/dev/mapper/mpatha

用fdisk对多路径软件生成的磁盘进行分区保存时会有一个报错,此报错不用理会.

ls -l /dev/mapper/

[root@liujing mnt]# ls -l /dev/mapper/

total 0

crw-rw----. 1 root root 10, 58 Mar 10 19:10 control

lrwxrwxrwx. 1 root root 7 Mar 10 20:28mpatha -> ../dm-0

lrwxrwxrwx. 1 root root 7 Mar 10 20:33 mpathap1-> ../dm-1

的mpathap1就是咱们对multipath磁盘进行的分区

mkfs.ext4 /dev/mapper/mpathap1 #对mpath1p1分区格式化成ext4文件系统

mount /dev/mapper/mpathap1 /mnt/ #挂载mpathap1分区

格式化和挂载时用/dev/mapper/mpathap1

4. 分区磁盘：

上面有提到分区时用目录/dev/mapper/mpatha

[root@liujing~]# fdisk /dev/mapper/mpatha

Device contains neither a valid DOS partition table, nor Sun, SGI or OSFdisklabel

Building a new DOS disklabel with disk identifier 0xac956c3a.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected byw(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to

switch off the mode (command 'c') and change display units to

sectors (command 'u').

Command (m for help): n------------------------新建分区

Command action

e extended

p primary partition (1-4)

p-----------------------------主分区

Partition number (1-4): 1

First cylinder (1-1016, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-1016, default 1016):

Using default value 1016

Command (m for help): w ---------------------写入列表至关于保存

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

注：若是同一台设备的两个node挂一样的盘符，另外一个盘符还须要再次写入w就行。不须要n了。

5. 格式化：

[root@liujing ~]# mkfs.ext4 /dev/mapper/mpathap1

mke2fs 1.41.12 (17-May-2010)

/dev/sdd1 alignment is offset by 1024 bytes.

This may result in very poor performance, (re)-partitioning suggested.

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=1 blocks, Stripe width=16 blocks

230608 inodes, 921250 blocks

46062 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=943718400

29 block groups

32768 blocks per group, 32768 fragments per group

7952 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (16384 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or

180 days, whichever comes first. Use tune2fs -c or -i tooverride.

6. 挂载 /dev/mapper/mpathap1 到 /mnt

[root@liujing ~]# mount /dev/mapper/mpathap1 /mnt

3、multipath的高级配置以前的配置都是用multipath的默认配置来完成multipath，好比映射设备的名称，multipath负载均衡的方法都是默认设置。那有没有按照咱们本身定义的方法来配置multipath呢，答案是OK。

一、multipath.conf文件的配置

接下来的工做就是要编辑/etc/multipath.conf的配置文件

multipath.conf主要包括blacklist、multipaths、devices三部份的配置

blacklist配置

blacklist {

devnode "^sda"

}

Multipaths部分配置multipaths和devices两部份的配置。

multipaths {

multipath {

wwid **** #此值multipath -v3能够看到

alias iscsi-dm0 #映射后的别名,能够随便取

path_grouping_policy multibus #路径组策略

path_checker tur #决定路径状态的方法

path_selector "round-robin 0" #选择那条路径进行下一个IO操做的方法

}

}

Devices部分配置

devices {

device {

vendor "iSCSI-Enterprise" #厂商名称

product "Virtual disk" #产品型号

path_grouping_policy multibus #默认的路径组策略

getuid_callout "/sbin/scsi_id -g -u -s /block/%n" #得到惟一设备号使用的默认程序

prio_callout "/sbin/acs_prio_alua%d" #获取有限级数值使用的默认程序

path_checker readsector0 #决定路径状态的方法

path_selector "round-robin 0" #选择那条路径进行下一个IO操做的方法

failback immediate #故障恢复的模式

no_path_retry queue #在disable queue以前系统尝试使用失效路径的次数的数值

rr_min_io 100 #在当前的用户组中，在切换到另一条路径以前的IO请求的数目

}

}

下面是相关参数的标准文档的介绍：

Attribute

Description

wwid

Specifies the WWID of the multipath device to which themultipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.

alias

Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are usinguser_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.

path_grouping_policy

Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include:

failover = 1 path per priority group

multibus = all valid paths in 1 priority group

group_by_serial = 1 priority group per detected serial number

group_by_prio = 1 priority group per path priority value

group_by_node_name = 1 priority group per target node name

path_selector

Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include:

round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each.

queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests.

service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput.

failback

Manages path group failback.

A value of immediate specifies immediate failback to the highest priority path group that contains active paths.

A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention.

A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover.

A numeric value greater than zero specifies deferred failback, expressed in seconds.

prio

Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include:

const: Set a priority of 1 to all paths.

emc: Generate the path priority for EMC arrays.

alua: Generate the path priority based on the SCSI-3 ALUA settings.

tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit.

ontap: Generate the path priority for NetApp arrays.

rdac: Generate the path priority for LSI/Engenio RDAC controller.

hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode.

hds: Generate the path priority for Hitachi HDS Modular storage arrays.

no_path_retry

A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing.

A value of fail indicates immediate failure, without queueing.

A value of queue indicates that queueing should not stop until the path is fixed.

rr_min_io

Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.

rr_min_io_rq

Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1.

rr_weight

If set to priorities, then instead of sending rr_min_iorequests to a path before calling path_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by theprio function. If set to uniform, all path weights are equal.

flush_on_last_del

If set to yes, then multipath will disable queueing when the last path to a device has been deleted.

在我本地的一个完整的高级配置以下：

[root@liujing ~]# vi /etc/multipath.conf

blacklist {

devnode "^sda"

}

multipaths {

multipath {

wwid 360a98000646650724434697454546156

alias mpathb_fcoe

path_grouping_policy multibus

#path_checker "directio"

prio "random"

path_selector "round-robin0"

}

}

devices {

device {

vendor "NETAPP"

product "LUN"

getuid_callout "/lib/udev/scsi_id--whitelisted --device=/dev/%n"

#path_checker "directio"

#path_selector "round-robin0"

failback immediate

no_path_retry fail

}

}

其中 wwid，vendor，product， getuid_callout这些参数能够经过：multipath -v3命令来获取。若是在/etc/multipath.conf中有设定各wwid 别名,别名会覆盖此设定。

4、负载均衡测试：

可使用dd命令来对设备进行读写操做，并同时经过iostat来查看I／0状态，流量从哪一个路径出去：

DD命令：dd if=/dev/zero of=/mnt/1Gfile bs=8k count=131072 在上面咱们已经把磁盘挂载在/MNT文件夹下因此咱们在读写磁盘时直接对/mnt文件夹直接读写就能够了。

若是想对磁盘重复读写能够用以下语句：

[root@liujing ~]# for ((i=1;i<=5;i++));do dd if=/dev/zeroof=/mnt/1Gfile bs=8k count=131072 2>&1|grep MB;done; ---重复读写5次这个值能够根据本身测试需求修改。

另外一个控制台输入iostat 2 10查看IO读写状态：

能够看到sdc和sdd是两个多路径的盘符，流量均匀的负载在两条路径中，负载均衡很成功。

5、路径冗余备份测试

将其中一条路径的端口down掉，全部流量会直接切换到另外一个路径中

转载自：https://blog.51cto.com/zjwliunx/1885408