Linux分布式mogilefs文件系统

一、初识分布式文件系统

在生产环境中，后端存储服务器服务器是一个很重要的环节。若是一旦前段服务器须要大量进行IO操做，会对后端存储的文件系统形成很大的压力。一旦后端的存储文件系统down机，那会致使整个生产环境奔溃。由于在生产环境中，后端的存储文件系统会成为一个单点故障。而为了不由于单点故障致使整个生产环境没法正常使用，全部你们都会采用分布式文件系统，以免由于单点故障致使生产环境奔溃。

2. mogilefs组成部分

MogileFS由3个部分组成：

(1) server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它经过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它实际上是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将全部的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。

(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。

(3) 客户端API：MogileFS的客户端API不少，例如Perl、PHP、Java、Python等，用这个模块能够编写客户端程序，实现文件的备份管理功能等。

存储主机(节点)

这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增长这些机器.

设备(device)

一个存储节点,以就是上面的主机,能够有多个 device, 就是用来存放文件的目录(例如挂载的目录),每一个设备都有一个设备id,须要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面建立相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 以后,就真的 dead了,里面的数据也没法恢复了,且这个dead了的设备的 id 也不能再用.

3、分布式文件系统须要的程序包

MogileFS-Server-2.46-2.el6.noarch.rpm

MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm

MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm

MogileFS-Utils-2.19-1.el6.noarch.rpm

Perlbal-1.78-1.el6.noarch.rpm

Perlbal-doc-1.78-1.el6.noarch.rpm

perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

perl-IO-AIO-3.71-2.el6.x86_64

perl-MogileFS-Client-1.14-1.el6.noarch.rpm

perl-Net-Netmask-1.9015-8.el6.noarch.rpm

perl-Perlbal-1.78-1.el6.noarch.rpm

1. 接着去安装所提供的全部的软件包

[root@mogilefs1 mogilefs]# yum install perl-IO-AIO perl-Net-Netmask perl-Sys-Syslog

[root@mogilefs1 mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm  

perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm  perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm

或者在安装包的目录下把全部rpm一次性安装

[root@mogilefs1 mogilefs]# yum install *.rpm

2. 接着去mysql数据库上进行受权

[root@mysql1 ~]# mysql

mysql> grant all on *.* to 'root'@'172.16.17.%' identified by 'xiaopeng';

Query OK, 0 rows affected (0.00 sec)

mysql> create database mogilefs;

Query OK, 1 row affected (0.03 sec)

mysql> grant all on mogilefs.* to 'moguser'@'172.16.17.%' identified by 'mogpass';

Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;

Query OK, 0 rows affected (0.00 sec)

[root@mysql1 ~]#

6、数据库受权完成以后，接着去mogilefs文件系统的主控制台上，对数据库进行初始化。（建议此命令慎用，若是mogilefs文件系统在运行的时候，当心使用。）

[root@mogilefs1 mogilefs]# mogdbsetup --dbhost=172.16.17.144 --dbrootpass='xiaopeng' --dbuser='moguser' --dbpass='mogpass'

This will attempt to setup or upgrade your MogileFS database.

It won't destroy existing data.

Run with --help for more information.  Run with --yes to shut up these prompts.

Continue? [N/y]: y

完成以后，能够去数据库确认初始化状况

[root@mogilefs1 mogilefs]#

mysql> use mogilefs

Database changed

mysql> show tables;

+----------------------+

| Tables_in_mogilefs   |

+----------------------+

1. 数据库准备完成以后，接着去修改控制台文件

[root@mogilefs1 ~]# vim /etc/mogilefs/mogilefsd.conf

daemonize = 1

pidfile = /var/run/mogilefsd/mogilefsd.pid

db_dsn = DBI:mysql:mogilefs:host=172.16.100.68

db_user = moguser

db_pass = mogpass

listen = 0.0.0.0:7001

conf_port = 7001

query_jobs = 5

delete_jobs = 1

replicate_jobs = 2

reaper_jobs = 1

2. 主配置文件修改完成以后，接着全部的存储节点都须要进行一下的配置

[root@mogilefs1 ~]# mkdir -pv /data/mogilefs/dev1

[root@mogilefs1 ~]# chown -R mogilefs.mogilefs /data/mogilefs/     

[root@mogilefs1 ~]# vim /etc/mogilefs/mogstored.conf

maxconns = 10000

httplisten = 0.0.0.0:7500

mgmtlisten = 0.0.0.0:7501

docroot = /data/mogilefs/

3. 对于mogstord的启动文件（全部的节点都的mogstored的启动脚本文件须要加如下的几项）

为脚本添加：

pidfile=/var/run/mogilefsd/mogstored.pid

echo $(pidof mogstored) > ${pidfile}

rm -f ${pidfile}

4. 全部的配置修改完成以后，接着去启动服务器，进行配置资源

[root@mysql1 ~]# service mogstored start

Starting mogstored                                         [  OK  ]

[root@mysql1 ~]# ss -tanlp

[root@mogilefs1 init.d]# service mogilefsd start

Starting mogilefsd                                         [  OK  ]

[root@mogilefs1 ~]# mogadm check

Checking trackers...

  127.0.0.1:7001 ... OK

Checking hosts...

No devices found on tracker(s).

[root@mogilefs1 ~]# clear

追踪节点所能适应的管理命令

[root@mogilefs1 ~]# mogadm --tracker=172.16.17.141:7001

  mogadm check                     Check the state of the MogileFS world.

  mogadm stats                     Show MogileFS system statistics.   mogadm host ...

         host add ...              Add a host to MogileFS.

         host delete ...           Delete a host.

         host list                 List all hosts.

         host mark ...             Change the status of a host.  (equivalent to 'modify --status')

         host modify ...           Modify a host's properties.

  mogadm device ...

         device add ...            Add a device to a host.

         device list ...           List all devices, for each host.

         device mark ...           Mark a device as {alive,dead,down,drain,readonly}

         device modify ...         Modify a device's properties.

         device summary ...        List the summary of devices, for each host.

  mogadm domain ...

         domain add ...            Add a domain (namespace)

         domain delete ...         Delete a domain.

         domain list               List all hosts.

  mogadm class ...

         class add ...             Add a file class to a domain.

         class delete ...          Delete a file class from a domain.

         class list                List all classes, for each domain.

         class modify ...          Modify properties of a file class.

  mogadm slave ...

         slave add ...             Add a slave node for store usage

         slave delete ...          Delete a slave node for store usage

         slave list                List current store slave nodes.

         slave modify ...          Modify a slave node for store usage

  mogadm fsck ...

         fsck clearlog             Clear the fsck log

         fsck printlog             Display the fsck log

         fsck reset ...            Reset fsck position back to the beginning

         fsck start                Start (or resume) background fsck

         fsck status               Show fsck status

         fsck stop                 Stop (pause) background fsck

         fsck taillog              Tail the fsck log

  mogadm rebalance ...

         rebalance policy ...      Add or adjust the current policy

         rebalance reset           Reset an existing policy

         rebalance settings        Display rebalance settings

         rebalance start           Start a rebalance job

         rebalance status          Show status of current rebalance job

         rebalance stop            Stop a rebalance job

         rebalance test            Show what devices the current policy would match

  mogadm settings ...

         settings list             List all server settings

         settings set ...          Set server setting 'key' to 'value'.

5. 首先须要把节点设备添加至管理设备中。使其可以正常管理

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.140 --ip=172.16.17.140 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.141 --ip=172.16.17.141 --status=alive

[root@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.144 --ip=172.16.17.144 --status=alive

[root@mogilefs1 ~]#

添加完成以后，去确认设备添加状况

[root@mogilefs1 ~]# mogadm host list

172.16.17.140 [1]: alive

  IP:       172.16.17.140:7500

172.16.17.141 [2]: alive

  IP:       172.16.17.141:7500

172.16.17.144 [3]: alive

  IP:       172.16.17.144:7500

12、后端存储的节点添加完成以后，接着去添加设备文件。

[root@mogilefs1 ~]# mogadm device add 172.16.17.140 1

[root@mogilefs1 ~]# mogadm device add 172.16.17.141 2

[root@mogilefs1 ~]# mogadm device add 172.16.17.144 3

[root@mogilefs1 ~]# mogadm device list

172.16.17.140 [1]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev1:   alive      0.280     18.280     18.561        100

172.16.17.141 [2]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev2:   alive      0.280     18.280     18.561        100

172.16.17.144 [3]: alive

                    used(G)    free(G)   total(G)  weight(%)

   dev3:   alive      1.671     16.890     18.561        100

再次检查mogadm的状态信息

[root@mogilefs1 ~]# mogadm check

Checking trackers...

  127.0.0.1:7001 ... OK

Checking hosts...

  [ 1] 172.16.17.140 ... OK

  [ 2] 172.16.17.141 ... OK

  [ 3] 172.16.17.144 ... OK

Checking devices...

  host device         size(G)    used(G)    free(G)   use%   ob state   I/O%

  ---- ------------ ---------- ---------- ---------- ------ ---------- -----

  [ 1] dev1            18.561      0.281     18.280   1.51%  writeable   0.0

  [ 2] dev2            18.561      0.281     18.280   1.51%  writeable   0.0

  [ 3] dev3            18.561      1.671     16.890   9.00%  writeable   1.7

  ---- ------------ ---------- ---------- ---------- ------

             total:    55.683      2.233     53.451   4.01%

[root@mogilefs1 ~]#

1. 添加domain域空间存储目录

[root@mogilefs1 ~]# mogadm domain add files

[root@mogilefs1 ~]# mogadm domain add p_w_picpaths

[root@mogilefs1 ~]# mogadm domain list

 domain               class                mindevcount   replpolicy   hashtype

-------------------- -------------------- ------------- ------------ -------

 files                default                   2        MultipleHosts() NONE   

 p_w_picpaths               default                   2        MultipleHosts() NONE   

在建立domain的时候，根绝本身的须要进行建立。

2. 建立好域名空间以后，若是须要有建立class类别。也能建立，这样可以分类别存储文件

基本使用方法

[root@mogilefs1 ~]# mogadm class help

  mogadm class add <domain> <class> [opts]           Add a file class to a domain.

      <class>              Name of class to add.

      <domain>             Domain to add class to.

      --hashtype=s         Hash algorithm string ('MD5', 'NONE').

      --mindevcount=i      Minimum number of replicas.

      --replpolicy=s       Replication policy string.

添加几个基本的类试试

[root@mogilefs1 ~]# mogadm class add p_w_picpaths jpg

[root@mogilefs1 ~]# mogadm class add p_w_picpaths png

[root@mogilefs1 ~]# mogadm class add p_w_picpaths gif

[root@mogilefs1 ~]# mogadm class add files text --mindevcount=1

[root@mogilefs1 ~]# mogadm class add files html --replpolicy="MultipleHosts(3)"

[root@mogilefs1 ~]# mogadm class list

 domain               class                mindevcount   replpolicy   hashtype

 files                default                   2        MultipleHosts() NONE   

 files                html                      2        MultipleHosts(3) NONE   

 files                plaintext                 1        MultipleHosts() NONE   

 

 p_w_picpaths               default                   2        MultipleHosts() NONE   

 p_w_picpaths               gif                       2        MultipleHosts() NONE   

 p_w_picpaths               jpeg                      2        MultipleHosts() NONE   

 p_w_picpaths               png                       2        MultipleHosts() NONE   

3. 若是基本的属性定义完成以后，接着能够尝试去上传资源试试

[root@mogilefs1 ~]# mogupload

Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'

[root@mogilefs1 ~]# mogupload --trackers=172.16.17.140 --domain=files --key='/fstab.txt' --file='/etc/fstab' --class=plaintext

上传完成以后，确认上传状况。可以，会生成http链接。可是链接会比较特殊。

[root@mogilefs1 ~]# mogfileinfo --trackers=172.16.17.140 --domain=files --key='/fstab.txt'

- file: /fstab.txt

     class:            plaintext

  devcount:                    1

    domain:                files

       fid:                    2

       key:           /fstab.txt

    length:                  921

 - http://172.16.17.141:7500/dev2/0/000/000/0000000002.fid

[root@mogilefs1 ~]#

 

[root@mogilefs1 mogilefs]# mogupload --trackers=172.16.17.140 --domain=p_w_picpaths --key='/1.jpg' --file='1.jpg' --class=jpeg

[root@mogilefs1 mogilefs]# mogfileinfo --trackers=172.16.17.140 --domain=p_w_picpaths --key='/1.jpg'

- file: /1.jpg

     class:                 jpeg

  devcount:                    2

    domain:               p_w_picpaths

       fid:                    3

       key:               /1.jpg

    length:               170625

 - http://172.16.17.140:7500/dev1/0/000/000/0000000003.fid

 - http://172.16.17.141:7500/dev2/0/000/000/0000000003.fid

 

[root@mogilefs1 mogilefs]# mogstats --db_dsn="DBI:mysql:mogilefs:host=172.16.17.144" --db_user="moguser" --db_pass="mogpass" --verbose  --stats="all"

Statistics for devices...

  device     host                   files     status

  ---------- ---------------- ------------ ----------

  dev1       172.16.17.140          1      alive

  dev2       172.16.17.141          2      alive

  ---------- ---------------- ------------ ----------

Statistics for file ids...

  Max file id: 3

Statistics for files...

  domain               class           files    size (m)  fullsize (m)

  -------------------- ----------- ---------- ----------- -------------

  files                plaintext           1           0             0

  p_w_picpaths               jpeg                1           0             0

  -------------------- ----------- ---------- ----------- -------------

Statistics for replication...

  domain               class        devcount      files

  -------------------- ----------- ---------- ----------

  files                plaintext           1          1

  p_w_picpaths               jpeg                2          1

  -------------------- ----------- ---------- ----------

若是想要经过更加便捷的方式进行上传数据须要，程序员本身写API接口

 

前段基于nginx进行反代mogilefs文件系统

1. 须要的安装包组

[root@backproxy ~]# ll

-rw-r--r--  1 root root 832104 Dec  4 15:32 nginx-1.8.0.tar.gz

-rw-r--r--  1 root root  11208 Dec  4 18:54 nginx_mogilefs_module-1.0.4.tar.gz

须要注意的一点是，对于nginx而言，若是想实现对后端的mogilefs服务器的代理须要nginx_mogilefs_module模块。不然不能实现此功能；同时，还须要注意的一点是模板的版本须要严格要求，模块的版本信息要和nginx版本信息匹配不然会报内核溢出之类的错误

2015/12/04 18:36:42 [alert] 1880#0: worker process 1890 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1892 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1894 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1896 exited on signal 11 (core dumped)

2. 在安装以前须要解决须要的依赖环境

[root@backproxy ~]#  yum groupinstall "Development Tools" "Server Platform Deveopment"

[root@backproxy ~]# yum install openssl-devel pcre-devel

还须要建立一个nginx用户

[root@backproxy ~]# groupadd -r nginx

[root@backproxy ~]# useradd -r -g nginx nginx

3. 接着解压安装包进行安装

[root@backproxy ~]# ./configure \

  --prefix=/usr \

  --sbin-path=/usr/sbin/nginx \

  --conf-path=/etc/nginx/nginx.conf \

  --error-log-path=/var/log/nginx/error.log \

  --http-log-path=/var/log/nginx/access.log \

  --pid-path=/var/run/nginx/nginx.pid  \

  --lock-path=/var/lock/nginx.lock \

  --user=nginx \

  --group=nginx \

  --with-http_ssl_module \

  --with-http_flv_module \

  --with-http_stub_status_module \

  --with-http_gzip_static_module \

  --http-client-body-temp-path=/var/tmp/nginx/client/ \

  --http-proxy-temp-path=/var/tmp/nginx/proxy/ \

  --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \

  --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \

  --http-scgi-temp-path=/var/tmp/nginx/scgi \

  --with-pcre \

  --with-debug \

  --add-module=

这里须要特别注意的一项是，在编译安装nginx的时候，须要为期指定mogilefs的模块相对于的文件--add-module=../nginx_mogilefs_module-1.0.4

编译完成以后进行

Make && make install

4. 为nginx提供一个启动脚本以下

#!/bin/sh

# nginx - this script starts and stops the nginx daemon

# chkconfig:   - 85 15

# description:  Nginx is an HTTP(S) server, HTTP(S) reverse \

#               proxy and IMAP/POP3 proxy server

# processname: nginx

# config:      /etc/nginx/nginx.conf

# config:      /etc/sysconfig/nginx

# pidfile:     /var/run/nginx.pid

# Source function library.

. /etc/rc.d/init.d/functions

# Source networking configuration.

. /etc/sysconfig/network

# Check that networking is up.

[ "$NETWORKING" = "no" ] && exit 0

nginx="/usr/sbin/nginx"

prog=$(basename $nginx)

 

NGINX_CONF_FILE="/etc/nginx/nginx.conf"

 

[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx

 

lockfile=/var/lock/subsys/nginx

 

make_dirs() {

   # make required directories

   user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`

   options=`$nginx -V 2>&1 | grep 'configure arguments:'`

   for opt in $options; do

       if [ `echo $opt | grep '.*-temp-path'` ]; then

           value=`echo $opt | cut -d "=" -f 2`

           if [ ! -d "$value" ]; then

               # echo "creating" $value

               mkdir -p $value && chown -R $user $value

           fi

       fi

   done

}

 

start() {

    [ -x $nginx ] || exit 5

    [ -f $NGINX_CONF_FILE ] || exit 6

    make_dirs

    echo -n $"Starting $prog: "

    daemon $nginx -c $NGINX_CONF_FILE

    retval=$?

    echo

    [ $retval -eq 0 ] && touch $lockfile

    return $retval

}

 

stop() {

    echo -n $"Stopping $prog: "

    killproc $prog -QUIT

    retval=$?

    echo

    [ $retval -eq 0 ] && rm -f $lockfile

    return $retval

}

 

restart() {

    configtest || return $?

    stop

    sleep 1

    start

}

 

reload() {

    configtest || return $?

    echo -n $"Reloading $prog: "

    killproc $nginx -HUP

    RETVAL=$?

    echo

}

 

force_reload() {

    restart

}

 

configtest() {

  $nginx -t -c $NGINX_CONF_FILE

}

 

rh_status() {

    status $prog

}

 

rh_status_q() {

    rh_status >/dev/null 2>&1

}

 

case "$1" in

    start)

        rh_status_q && exit 0

        $1

        ;;

    stop)

        rh_status_q || exit 0

        $1

        ;;

    restart|configtest)

        $1

        ;;

    reload)

        rh_status_q || exit 7

        $1

        ;;

    force-reload)

        force_reload

        ;;

    status)

        rh_status

        ;;

    condrestart|try-restart)

        rh_status_q || exit 0

            ;;

    *)

        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"

        exit 2

esac

然后为此脚本赋予执行权限：

[root@backproxy ~]# chmod +x /etc/rc.d/init.d/nginx

添加至服务管理列表，并让其开机自动启动：

[root@backproxy ~]# chkconfig --add nginx

[root@backproxy ~]# chkconfig nginx on

然后就能够启动服务并测试了：

[root@backproxy ~]# service nginx start

5. 接着去修改nginx的主配置文件

upstream trackers {

        server 172.16.17.140:7001 weight=1;

        server 172.16.17.141:7001 weight=1;

        server 172.16.17.144:7001 weight=1;

    }

Server {

        location /p_w_picpaths {

            mogilefs_tracker trackers;

            mogilefs_domain p_w_picpaths;

            mogilefs_methods GET;

            mogilefs_noverify on;

            mogilefs_pass {

                  proxy_pass $mogilefs_path;

                  proxy_hide_header Content-Type;

                  proxy_buffering off;

            }

        }

        location /files {

            mogilefs_tracker trackers;

            mogilefs_domain files;

            mogilefs_methods GET;

            mogilefs_noverify on;

            mogilefs_pass {

                  proxy_pass $mogilefs_path;

                  proxy_hide_header Content-Type;

                  proxy_buffering off;

            }

        }

}

6. 配置结束以后，重启服务器能够去测试反代效果如何？

 

查看反代日志

[root@backproxy ~]# tail -2 /var/log/nginx/access.log

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /p_w_picpaths/love.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /p_w_picpaths/love.jpg HTTP/1.1" 200 190166 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36"

[root@backproxy ~]#

7. 接着尝试down机一台存储服务器测试效果如何

[root@mogilefs2 ~]# service mogstored stop

Stop mogstored                                         [  OK  ]

 

对于前段访问后端存储服务器不会出现任何问题

 

注意点：对于这个版本的mogilefs分布式文件系统，在centos7上，只能存储一份，不能有其余多余的副本