分布式文件系统之MogileFS的安装使用

时间 2020-09-21

原文原文链接

　　1、简介html

　　MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发；它主要由三部分组成，第一部分是server端，server端主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它经过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它实际上是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将全部的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。第二部是mogilefs的utils,这部分主要是一些工具，好比mogadm、mogupload、mogfileinfo 等等；第三部分是客户端API，mogilefs支持众多编程语言的客户端API接口，使用对应的客户端API接口，咱们能够编写mogilefs的客户端，从而实现来管理mogilefs上的文件；前端

　　2、Mogilefs架构图node

　　提示：tracker的主要做用是代理前端用户的请求到数据库查询相关文件的元信息，同时它还要监控注册到tracker的storagenode的健康状态，一旦后端storage挂掉，它会马上把对应的storage节点上的device从集群信息中删除（对应数据还在，只是在集群的状态信息看不到对应设备），上线时再加入到集群信息；客户端经过向tracker发送请求文件的信息，tracker经过去数据库中找到对应的文件所在storage节点的ip地址、端口、设备id以及文件fid，而后经过http连接的形式从tracker返回给客户端，客户端拿到这tracker的返回的文件地址，而后在去对应的storage请求文件，从而在客户端就访问到对应的文件内容信息；客户端和tracker，storage之间通讯都是http协议，从本质上讲storage就是一个web服务器；因此对于客户端咱们可使用web服务器的反向代理，也能够是调用mogilefsAPI写的客户端程序；mysql

　　3、Mogilefs安装部署nginx

　　环境说明web

名称	角色	ip地址	端口
node01	tracker+mysql+storage	192.168.0.41	7001/7500/7501/3306
node02	tracker+storage	192.168.0.42	7001/7500/7501
node03	storage	192.168.0.43	7500/7501

　　各server时间同步sql

　　提示：若是有必要，能够配置各节点用hosts文件解析；数据库

　　安装环境依赖包编程

[root@node01 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y 
Loaded plugins: fastestmirror
base                                                                                                            | 3.6 kB  00:00:00     
epel                                                                                                            | 4.7 kB  00:00:00     
extras                                                                                                          | 2.9 kB  00:00:00     
updates                                                                                                         | 2.9 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package perl-IO-AIO.x86_64 0:4.20-1.el7 will be installed
--> Processing Dependency: perl(common::sense) for package: perl-IO-AIO-4.20-1.el7.x86_64
---> Package perl-IO-stringy.noarch 0:2.110-22.el7 will be installed
---> Package perl-Net-Netmask.noarch 0:1.9015-13.el7 will be installed
---> Package perl-Sys-Syslog.x86_64 0:0.33-3.el7 will be installed
--> Running transaction check
---> Package perl-common-sense.noarch 0:3.6-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================================
 Package                                Arch                        Version                            Repository                 Size
=======================================================================================================================================
Installing:
 perl-IO-AIO                            x86_64                      4.20-1.el7                         epel                      117 k
 perl-IO-stringy                        noarch                      2.110-22.el7                       base                       71 k
 perl-Net-Netmask                       noarch                      1.9015-13.el7                      epel                       25 k
 perl-Sys-Syslog                        x86_64                      0.33-3.el7                         base                       42 k
Installing for dependencies:
 perl-common-sense                      noarch                      3.6-4.el7                          epel                       28 k

Transaction Summary
=======================================================================================================================================
Install  4 Packages (+1 Dependent package)

Total download size: 283 k
Installed size: 636 k
Downloading packages:
(1/5): perl-Sys-Syslog-0.33-3.el7.x86_64.rpm                                                                    |  42 kB  00:00:00     
(2/5): perl-Net-Netmask-1.9015-13.el7.noarch.rpm                                                                |  25 kB  00:00:00     
(3/5): perl-IO-AIO-4.20-1.el7.x86_64.rpm                                                                        | 117 kB  00:00:00     
(4/5): perl-common-sense-3.6-4.el7.noarch.rpm                                                                   |  28 kB  00:00:00     
(5/5): perl-IO-stringy-2.110-22.el7.noarch.rpm                                                                  |  71 kB  00:00:00     
---------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                  642 kB/s | 283 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : perl-common-sense-3.6-4.el7.noarch                                                                                  1/5 
  Installing : perl-IO-AIO-4.20-1.el7.x86_64                                                                                       2/5 
  Installing : perl-Net-Netmask-1.9015-13.el7.noarch                                                                               3/5 
  Installing : perl-Sys-Syslog-0.33-3.el7.x86_64                                                                                   4/5 
  Installing : perl-IO-stringy-2.110-22.el7.noarch                                                                                 5/5 
  Verifying  : perl-IO-AIO-4.20-1.el7.x86_64                                                                                       1/5 
  Verifying  : perl-common-sense-3.6-4.el7.noarch                                                                                  2/5 
  Verifying  : perl-IO-stringy-2.110-22.el7.noarch                                                                                 3/5 
  Verifying  : perl-Sys-Syslog-0.33-3.el7.x86_64                                                                                   4/5 
  Verifying  : perl-Net-Netmask-1.9015-13.el7.noarch                                                                               5/5 

Installed:
  perl-IO-AIO.x86_64 0:4.20-1.el7           perl-IO-stringy.noarch 0:2.110-22.el7       perl-Net-Netmask.noarch 0:1.9015-13.el7      
  perl-Sys-Syslog.x86_64 0:0.33-3.el7      

Dependency Installed:
  perl-common-sense.noarch 0:3.6-4.el7                                                                                                 

Complete!
[root@node01 ~]#

　　提示：三个节点都须要安装环境依赖的包；后端

　　上传rpm包到服务器并安装

[root@node01 ~]# rz
rz waiting to receive.
 zmodem trl+C ȡ

  100%       1 KB    1 KB/s 00:00:01       0 Errors.el7.centos.noarch.rpm...
  100%     173 KB  173 KB/s 00:00:01       0 Errorsentos.noarch.rpm...
  100%      26 KB   26 KB/s 00:00:01       0 Errorsentos.noarch.rpm...
  100%      75 KB   75 KB/s 00:00:01       0 Errorsh.rpm...
  100%      27 KB   27 KB/s 00:00:01       0 Errors.rpm...
  100%      29 KB   29 KB/s 00:00:01       0 Errors.noarch.rpm...
  100%     262 KB  262 KB/s 00:00:01       0 Errors

[root@node01 ~]# ls
MogileFS-Server-2.46-2.el7.centos.noarch.rpm            perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm  perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm  perl-Perlbal-1.78-1.el6.noarch.rpm
MogileFS-Utils-2.19-1.el7.centos.noarch.rpm
[root@node01 ~]# yum install ./*.rpm -y
Loaded plugins: fastestmirror
Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch
Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch
Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed
Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch
Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch
Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed
---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed
--> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
--> Processing Dependency: perl(DBD::mysql) >= 3 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch
---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed
……省略部份内容……
Installed:
  MogileFS-Server.noarch 0:2.46-2.el7.centos                         MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos              
  MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos               MogileFS-Utils.noarch 0:2.19-1.el7.centos                         
  perl-Danga-Socket.noarch 0:1.61-1.el6.rf                           perl-MogileFS-Client.noarch 0:1.14-1.el7.centos                   
  perl-Perlbal.noarch 0:1.78-1.el6                                  

Dependency Installed:
  mailcap.noarch 0:2.1.41-2.el7                                  perl-BSD-Resource.x86_64 0:1.29.07-1.el7                             
  perl-Business-ISBN.noarch 0:2.06-2.el7                         perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7                  
  perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7                   perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7                          
  perl-DBD-MySQL.x86_64 0:4.023-6.el7                            perl-DBI.x86_64 0:1.627-4.el7                                        
  perl-Data-Dumper.x86_64 0:2.145-3.el7                          perl-Digest.noarch 0:1.17-245.el7                                    
  perl-Digest-MD5.x86_64 0:2.52-3.el7                            perl-Encode-Locale.noarch 0:1.03-5.el7                               
  perl-File-Listing.noarch 0:6.04-7.el7                          perl-HTML-Parser.x86_64 0:3.71-4.el7                                 
  perl-HTML-Tagset.noarch 0:3.20-15.el7                          perl-HTTP-Cookies.noarch 0:6.01-5.el7                                
  perl-HTTP-Daemon.noarch 0:6.01-8.el7                           perl-HTTP-Date.noarch 0:6.02-8.el7                                   
  perl-HTTP-Message.noarch 0:6.06-6.el7                          perl-HTTP-Negotiate.noarch 0:6.01-5.el7                              
  perl-IO-Compress.noarch 0:2.061-2.el7                          perl-IO-HTML.noarch 0:1.00-2.el7                                     
  perl-IO-Socket-IP.noarch 0:0.21-5.el7                          perl-IO-Socket-SSL.noarch 0:1.94-7.el7                               
  perl-LWP-MediaTypes.noarch 0:6.02-2.el7                        perl-Mozilla-CA.noarch 0:20130114-5.el7                              
  perl-Net-Daemon.noarch 0:0.48-5.el7                            perl-Net-HTTP.noarch 0:6.06-2.el7                                    
  perl-Net-LibIDN.x86_64 0:0.12-15.el7                           perl-Net-SSLeay.x86_64 0:1.55-6.el7                                  
  perl-PlRPC.noarch 0:0.2020-14.el7                              perl-Sys-Syscall.noarch 0:0.25-10.el7                                
  perl-TimeDate.noarch 1:2.30-2.el7                              perl-URI.noarch 0:1.60-9.el7                                         
  perl-WWW-RobotRules.noarch 0:6.02-5.el7                        perl-libwww-perl.noarch 0:6.05-2.el7                                 

Complete!
[root@node01 ~]#

　　提示：安装环境依赖和以上rpm包，首先要配置好epel源；由于有些依赖包，它来自epel源；相关包我已经上传至个人网盘http://www.qiuhom.com/d/383f34f57cf847beb296/；

　　安装mariadb-server

　　配置yum源

[root@node01 ~]# cat /etc/yum.repos.d/mairadb.repo 
[mariadb]
name=mariadb repo
baseurl=https://mirrors.tuna.tsinghua.edu.cn/mariadb//mariadb-10.1.46/yum/centos/7/x86_64/
gpgcheck=0
[root@node01 ~]#

　　安装mariadb-server

[root@node01 ~]# yum install MariaDB-server -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package MariaDB-server.x86_64 0:10.1.46-1.el7.centos will be installed
--> Processing Dependency: MariaDB-client for package: MariaDB-server-10.1.46-1.el7.centos.x86_64
--> Processing Dependency: rsync for package: MariaDB-server-10.1.46-1.el7.centos.x86_64
--> Processing Dependency: MariaDB-common for package: MariaDB-server-10.1.46-1.el7.centos.x86_64
--> Processing Dependency: galera for package: MariaDB-server-10.1.46-1.el7.centos.x86_64
--> Processing Dependency: libjemalloc.so.1()(64bit) for package: MariaDB-server-10.1.46-1.el7.centos.x86_64
--> Running transaction check
……省略部份内容……

Installed:
  MariaDB-server.x86_64 0:10.1.46-1.el7.centos                       MariaDB-shared.x86_64 0:10.1.46-1.el7.centos                      

Dependency Installed:
  MariaDB-client.x86_64 0:10.1.46-1.el7.centos                       MariaDB-common.x86_64 0:10.1.46-1.el7.centos                      
  boost-program-options.x86_64 0:1.53.0-28.el7                       galera.x86_64 0:25.3.29-1.rhel7.el7.centos                        
  jemalloc.x86_64 0:3.6.0-1.el7                                      rsync.x86_64 0:3.1.2-10.el7                                       

Replaced:
  mariadb-libs.x86_64 1:5.5.56-2.el7                                                                                                   

Complete!
[root@node01 ~]#

　　提示：mariadb-server包名是大写的MariaDB-server；

　　配置mariadb忽略名称解析

　　设置mariadb开机自动启动，并启动mariadb

　　添加用于mogilefs访问数据库的帐号

　　初始化数据库

[root@node01 ~]# mogdbsetup --help
Usage: mogdbsetup [opts]

Options:

                  Default      Description
                  ============ ===========================================
 --verbose        <off>        Be verbose about what's happening.

 --dbhost=        localhost    hostname or IP to database server.

 --dbport=        dbd default  port number to database server.

 --dbname=        mogilefs     database name to create/upgrade.

 --dbrootuser=    root         Database administrator username.  Only needed
                               for initial setup, not subsequent upgrades.

 --dbrootpass=    <blank>      Database administrator password.  Only needed
                               for initial setup, not subsequent upgrades.

 --dbuser=        mogile       Regular database user to create and/or use
                               for MogileFS database.  This is what the
                               mogilefsd trackers connect as.

 --dbpass=        <blank>      You should change this, especially if your
                               database servers are accessible to other users
                               on the network.  But they shouldn't be
                               if you're running MogileFS, because MogileFS
                               assumes your network is closed.

  --type=         MySQL        Which MogileFS::Store implementation to use.
                               Available: MySQL, Postgres

  --yes                        Run without questions.

[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs

This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.  Run with --yes to shut up these prompts.

Continue? [N/y]: y

Create/Upgrade database name 'mogilefs'? [Y/n]: y

Grant all privileges to user 'mogilefs', connecting from anywhere, to the mogilefs database 'mogilefs'? [Y/n]: n
Stopped.
[root@node01 ~]# mogdbsetup --dbhost=192.168.0.41 --dbport=3306 --dbname=mogilefs --dbrootpass=admin --dbuser=mogilefs --dbpass=mogilefs

This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.  Run with --yes to shut up these prompts.

Continue? [N/y]: y
[root@node01 ~]#

　　提示：若是不想让mogdbsetup在数据库中建立一个任何地址均可访问的mogilefs的用户，在最后的选项中使用n便可，而后在重复执行上述命令便可；若是不想交互式输入y，也能够直接在命令后面用--yes；

　　验证：在数据库中查看对应的库是否都已建立，对应库中是否都有表？

[root@node01 ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 7
Server version: 10.1.46-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mogilefs           |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.01 sec)

MariaDB [(none)]> use mogilefs
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
MariaDB [mogilefs]> show tables;
+----------------------+
| Tables_in_mogilefs   |
+----------------------+
| checksum             |
| class                |
| device               |
| domain               |
| file                 |
| file_on              |
| file_on_corrupt      |
| file_to_delete       |
| file_to_delete2      |
| file_to_delete_later |
| file_to_queue        |
| file_to_replicate    |
| fsck_log             |
| host                 |
| server_settings      |
| tempfile             |
| unreachable_fids     |
+----------------------+
17 rows in set (0.00 sec)

MariaDB [mogilefs]>

　　到此数据库就准备好了

　　配置mogilefsd链接数据库

　　提示：咱们只须要修改红框中的内容便可；db_dsn用于指定链接数据库的驱动，以及数据库库名和主机地址；db_user用于指定链接数据库所使用的用户，db_pass用于指定链接数据库的密码；listen用于指定mogilefsd监听的ip地址和端口；后面的配置就是和启动的进程数量的配置，这个能够根据业务的规模来设置；

　　启动mogilefsd

　　提示：在centos7上启动mogilefsd时，会报错，咱们能够不用管这个错误，实际上mogilefsd是启动成功的，只要看到对应的端口和进程起来，表示mogilefsd启动就没有问题；

　　配置node01上的mogstored

[root@node01 ~]# cat /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data/mogilefs
[root@node01 ~]# mkdir /data/mogilefs/ -pv
mkdir: created directory ‘/data’
mkdir: created directory ‘/data/mogilefs/’
[root@node01 ~]# chown -R mogilefs.mogilefs /data/mogilefs/
[root@node01 ~]# ll -d /data/mogilefs/
drwxr-xr-x 2 mogilefs mogilefs 6 Sep 16 13:37 /data/mogilefs/
[root@node01 ~]#

　　启动mogstored

　　提示：在centos7上启动mogstored也会同启动mogilefsd同样报错，这个主要是由于systemd的缘故，在centos6不会报错；到此，node01上的mariadb、mogilefsd和mogstored都已经安装配置好了；

　　复制安装包到node02和node03

　　在node02上安装

[root@node02 ~]# yum install perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO ./*.rpm -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version
Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version
Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version
Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version
Examining ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch
Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch
Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed
Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch
Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch
Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package MogileFS-Server.noarch 0:2.46-2.el7.centos will be installed
---> Package MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos will be installed
--> Processing Dependency: perl(DBI) >= 1.44 for package: MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch
……省略部份内容……
Installed:
  MogileFS-Server.noarch 0:2.46-2.el7.centos                                                              
  MogileFS-Server-mogilefsd.noarch 0:2.46-2.el7.centos                                                    
  MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos                                                    
  MogileFS-Utils.noarch 0:2.19-1.el7.centos                                                               
  perl-Danga-Socket.noarch 0:1.61-1.el6.rf                                                                
  perl-MogileFS-Client.noarch 0:1.14-1.el7.centos                                                         
  perl-Perlbal.noarch 0:1.78-1.el6                                                                        

Dependency Installed:
  mailcap.noarch 0:2.1.41-2.el7                    perl-BSD-Resource.x86_64 0:1.29.07-1.el7               
  perl-Business-ISBN.noarch 0:2.06-2.el7           perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7    
  perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7     perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7            
  perl-DBD-MySQL.x86_64 0:4.023-6.el7              perl-DBI.x86_64 0:1.627-4.el7                          
  perl-Data-Dumper.x86_64 0:2.145-3.el7            perl-Digest.noarch 0:1.17-245.el7                      
  perl-Digest-MD5.x86_64 0:2.52-3.el7              perl-Encode-Locale.noarch 0:1.03-5.el7                 
  perl-File-Listing.noarch 0:6.04-7.el7            perl-HTML-Parser.x86_64 0:3.71-4.el7                   
  perl-HTML-Tagset.noarch 0:3.20-15.el7            perl-HTTP-Cookies.noarch 0:6.01-5.el7                  
  perl-HTTP-Daemon.noarch 0:6.01-8.el7             perl-HTTP-Date.noarch 0:6.02-8.el7                     
  perl-HTTP-Message.noarch 0:6.06-6.el7            perl-HTTP-Negotiate.noarch 0:6.01-5.el7                
  perl-IO-Compress.noarch 0:2.061-2.el7            perl-IO-HTML.noarch 0:1.00-2.el7                       
  perl-IO-Socket-IP.noarch 0:0.21-5.el7            perl-IO-Socket-SSL.noarch 0:1.94-7.el7                 
  perl-LWP-MediaTypes.noarch 0:6.02-2.el7          perl-Mozilla-CA.noarch 0:20130114-5.el7                
  perl-Net-Daemon.noarch 0:0.48-5.el7              perl-Net-HTTP.noarch 0:6.06-2.el7                      
  perl-Net-LibIDN.x86_64 0:0.12-15.el7             perl-Net-SSLeay.x86_64 0:1.55-6.el7                    
  perl-PlRPC.noarch 0:0.2020-14.el7                perl-Sys-Syscall.noarch 0:0.25-10.el7                  
  perl-TimeDate.noarch 1:2.30-2.el7                perl-URI.noarch 0:1.60-9.el7                           
  perl-WWW-RobotRules.noarch 0:6.02-5.el7          perl-libwww-perl.noarch 0:6.05-2.el7                   

Complete!
[root@node02 ~]#

　　把node01上的mogilefsd的配置文件复制到node02上，而后启动mogilefsd

　　把node01上的mogstored配置文件复制到node02上，而后在node02上建立对应的目录并修改属主和属组为mogilefs，而后启动mogstored

　　到此node02上的mogilefsd和mogstored就安装配置启动好了；

　　在node03上安装配置mogstored

[root@node03 ~]# ls
MogileFS-Server-2.46-2.el7.centos.noarch.rpm            perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el7.centos.noarch.rpm  perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm
MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm  perl-Perlbal-1.78-1.el6.noarch.rpm
MogileFS-Utils-2.19-1.el7.centos.noarch.rpm
[root@node03 ~]# yum install ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm ./perl-*.rpm perl-Net-Netmask perl-IO-stringy perl-Sys-Syslog perl-IO-AIO -y    
Loaded plugins: fastestmirror
Examining ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm: MogileFS-Server-mogstored-2.46-2.el7.centos.noarch
Marking ./MogileFS-Server-mogstored-2.46-2.el7.centos.noarch.rpm to be installed
Examining ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm: MogileFS-Utils-2.19-1.el7.centos.noarch
Marking ./MogileFS-Utils-2.19-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm: perl-Danga-Socket-1.61-1.el6.rf.noarch
Marking ./perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm to be installed
Examining ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm: perl-MogileFS-Client-1.14-1.el7.centos.noarch
Marking ./perl-MogileFS-Client-1.14-1.el7.centos.noarch.rpm to be installed
Examining ./perl-Perlbal-1.78-1.el6.noarch.rpm: perl-Perlbal-1.78-1.el6.noarch
Marking ./perl-Perlbal-1.78-1.el6.noarch.rpm to be installed
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Package perl-Net-Netmask-1.9015-13.el7.noarch already installed and latest version
Package perl-IO-stringy-2.110-22.el7.noarch already installed and latest version
Package perl-Sys-Syslog-0.33-3.el7.x86_64 already installed and latest version
Package perl-IO-AIO-4.20-1.el7.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos will be installed
---> Package MogileFS-Utils.noarch 0:2.19-1.el7.centos will be installed
--> Processing Dependency: perl(Compress::Zlib) for package: MogileFS-Utils-2.19-1.el7.centos.noarch
……省略部份内容……
Installed:
  MogileFS-Server-mogstored.noarch 0:2.46-2.el7.centos   MogileFS-Utils.noarch 0:2.19-1.el7.centos        
  perl-Danga-Socket.noarch 0:1.61-1.el6.rf               perl-MogileFS-Client.noarch 0:1.14-1.el7.centos  
  perl-Perlbal.noarch 0:1.78-1.el6                      

Dependency Installed:
  mailcap.noarch 0:2.1.41-2.el7                    perl-BSD-Resource.x86_64 0:1.29.07-1.el7               
  perl-Business-ISBN.noarch 0:2.06-2.el7           perl-Business-ISBN-Data.noarch 0:20120719.001-2.el7    
  perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7     perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7            
  perl-Data-Dumper.x86_64 0:2.145-3.el7            perl-Digest.noarch 0:1.17-245.el7                      
  perl-Digest-MD5.x86_64 0:2.52-3.el7              perl-Encode-Locale.noarch 0:1.03-5.el7                 
  perl-File-Listing.noarch 0:6.04-7.el7            perl-HTML-Parser.x86_64 0:3.71-4.el7                   
  perl-HTML-Tagset.noarch 0:3.20-15.el7            perl-HTTP-Cookies.noarch 0:6.01-5.el7                  
  perl-HTTP-Daemon.noarch 0:6.01-8.el7             perl-HTTP-Date.noarch 0:6.02-8.el7                     
  perl-HTTP-Message.noarch 0:6.06-6.el7            perl-HTTP-Negotiate.noarch 0:6.01-5.el7                
  perl-IO-Compress.noarch 0:2.061-2.el7            perl-IO-HTML.noarch 0:1.00-2.el7                       
  perl-IO-Socket-IP.noarch 0:0.21-5.el7            perl-IO-Socket-SSL.noarch 0:1.94-7.el7                 
  perl-LWP-MediaTypes.noarch 0:6.02-2.el7          perl-Mozilla-CA.noarch 0:20130114-5.el7                
  perl-Net-HTTP.noarch 0:6.06-2.el7                perl-Net-LibIDN.x86_64 0:0.12-15.el7                   
  perl-Net-SSLeay.x86_64 0:1.55-6.el7              perl-Sys-Syscall.noarch 0:0.25-10.el7                  
  perl-TimeDate.noarch 1:2.30-2.el7                perl-URI.noarch 0:1.60-9.el7                           
  perl-WWW-RobotRules.noarch 0:6.02-5.el7          perl-libwww-perl.noarch 0:6.05-2.el7                   

Complete!
[root@node03 ~]#

　　复制node01上的mogstored配置文件到node03上，而后在node03上建立对应的目录并修改属主和属组为mogilefs，而后启动mogstored

　　提示：在node03上没有安装mogilefs-server这个包，因此在对应系统上没有mogilefs这个帐号，咱们须要手动建立这个帐号；

　　启动mogstored

　　到此三个节点的环境配置，以及服务都已经正常启动了；

　　4、使用mogadm管理mogilefs

　　把三个storage节点注册到tracker上

　　提示：若是集群各节点没有作hosts主机名解析,那么对应的的节点就须要写清楚ip地址便可，若是端口不是7500，咱们还须要用--port去指定端口；

　　检查集群

　　提示：在tracker节点监听在本机全部地址的7001，执行上述命令不须要加--trackers选项来指明tracker是谁，它默认会去找本机的7001；不然执行上述命令须要加上--trackers来指定tracker的地址和端口；

　　在非tracker节点执行检查集群的命令

　　建立devices

　　分别在3个mogstored节点的docroot说指定的目录下建立以dev+数字的目录，并将属主和属组修改为mogilefs

[root@node03 ~]# mkdir /data/mogilefs/dev3
[root@node03 ~]# chown -R mogilefs.mogilefs /data/mogilefs/dev3
[root@node03 ~]# ll -d /data/mogilefs/dev3
drwxr-xr-x 3 mogilefs mogilefs 37 Sep 16 21:02 /data/mogilefs/dev3
[root@node03 ~]#

　　添加device

　　查看device

　　提示：若是在指定节点的docroot所指定的目录下没有建立dev+设备编号的目录，上面是看不到对应设备的空间大小的；

　　建立domain

　　建立class

　　好了，到此咱们就能够上传文件了

　　上传文件，把/etc/passwd上传到files域中，使用html class

　　提示：mogupload这个工具是mogilefs的一个上传文件的工具，其中--domain用于指定域，就是把文件放到那个名称空间中；--class用于指定使用那个class策略来同步文件，默认不指定使用default；--key用于指定客户端访问文件的路径；--file用于指定要上传的文件路径；

　　查看上传文件的信息

　　提示：查看上传文件的信息须要用mogfileinfo这个工具，咱们只须要指定trancker，domain以及文件的key就能够看到，该文件的class，devcount,domain,fid,key,length,以及访问http地址；这里须要说明一下，咱们刚才在建立files域中的html这个class 咱们指定了mindevcount=3，表示只要使用这个class上传文件，文件至少有3个副本；而咱们如今看到的只有一个，这是由于在centos7上，Sys-syscall这个包的版本太高，致使mogilefs异常；

　　降级Sys-Syscall，解决centos7上安装mogilefs 后端mogstored 同步问题

　　安装perl编译环境

[root@node01 ~]# yum -y install make gcc unzip perl-DBD-MySQL perl perl-CPAN perl-YAML perl-Time-HiRes
Loaded plugins: fastestmirror
base                                                                                           | 3.6 kB  00:00:00     
epel                                                                                           | 4.7 kB  00:00:00     
extras                                                                                         | 2.9 kB  00:00:00     
mariadb                                                                                        | 2.9 kB  00:00:00     
updates                                                                                        | 2.9 kB  00:00:00     
(1/2): epel/x86_64/updateinfo                                                                  | 1.0 MB  00:00:00     
(2/2): epel/x86_64/primary_db                                                                  | 6.9 MB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
Package gcc-4.8.5-39.el7.x86_64 already installed and latest version
Package unzip-6.0-21.el7.x86_64 already installed and latest version
Package perl-DBD-MySQL-4.023-6.el7.x86_64 already installed and latest version
Package 4:perl-5.16.3-295.el7.x86_64 already installed and latest version
Package 4:perl-Time-HiRes-1.9725-3.el7.x86_64 already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package make.x86_64 1:3.82-23.el7 will be updated
---> Package make.x86_64 1:3.82-24.el7 will be an update
---> Package perl-CPAN.noarch 0:1.9800-295.el7 will be installed
--> Processing Dependency: perl(local::lib) for package: perl-CPAN-1.9800-295.el7.noarch
……省略部份内容……
Installed:
  perl-CPAN.noarch 0:1.9800-295.el7                           perl-YAML.noarch 0:0.84-5.el7                          

Dependency Installed:
  gdbm-devel.x86_64 0:1.10-8.el7                            libdb-devel.x86_64 0:5.3.21-25.el7                        
  perl-Digest-SHA.x86_64 1:5.85-4.el7                       perl-ExtUtils-Install.noarch 0:1.58-295.el7               
  perl-ExtUtils-MakeMaker.noarch 0:6.68-3.el7               perl-ExtUtils-Manifest.noarch 0:1.61-244.el7              
  perl-ExtUtils-ParseXS.noarch 1:3.18-3.el7                 perl-Test-Harness.noarch 0:3.28-3.el7                     
  perl-devel.x86_64 4:5.16.3-295.el7                        perl-local-lib.noarch 0:1.008010-4.el7                    
  pyparsing.noarch 0:1.5.6-9.el7                            systemtap-sdt-devel.x86_64 0:4.0-11.el7                   

Updated:
  make.x86_64 1:3.82-24.el7                                                                                           

Dependency Updated:
  libdb.x86_64 0:5.3.21-25.el7                           libdb-utils.x86_64 0:5.3.21-25.el7                          

Complete!
[root@node01 ~]#

　　提示：在mogstored节点上都要安装好perl环境依赖包；而后按照下面的步骤从新编译安装Sys-Syscall；而后重启mogstored便可生效；

　　下载低版本的Sys-Syscall源码包，下载地址：https://cpan.metacpan.org/authors/id/B/BR/BRADFITZ/Sys-Syscall-0.23.tar.gz；并解压，进入到源码目录

　　编译安装Sys-Syscall

　　验证

　　重启mogstored，mogilefsd

　　提示：在某一台tracker节点从新编译安装Sys-Syscall，而后重启mogilefsd和mogstored便可；我这里是在全部节点都编译安装了，而后再把全部节点的mogstored和全部的mogilefsd都重启了一遍，其实用不着；

　　上传文件看看对应的副本数量是否仍是只有一个？

　　提示：能够看到对应的devcount和咱们以前指定的class上的mindevcount就同样了；

　　验证：访问指定上述任一一个地址，看看访问到的文件内容是不是咱们上传到文件内容？

　　提示：能够看到咱们上传到文件内容在tracker给咱们返回的地址上可以访问到，而且里面的内容也是同样的；

　　验证：停掉node02上的mogstored，看看对应的tracker是否可以及时发现对应的storage不在线了呢？

　　提示：能够看到当node02上的mogstored挂掉之后，在检查集群，tracker就能及时的发现对应的设备不在线了，这个有点相似nginx对后端web服务器作健康状态监测的感受；也就说tracker在监视着注册在tracker上的storage节点；

　　到此，mogilefs在centos7上的安装部署，以及使用mogadm工具管理集群就完成了，对于mogilefs一般咱们上传文件是经过业务程序调用api实现，mogupload只是一个测试工具；对于查看文件，一般用户也不会使用mogfieinfo返回的连接去查看文件，一般会使用一个web反向代理（nginx）去代理用户的访问；好比用户要查看/images/test.jpg，而在tracker返回的对应资源的连接中，必定不是/images/test.jpg，这个时候咱们就须要用到web反向代理，让其反向代理可以帮助咱们去tracker取到对应uri的资源连接，从而帮助咱们去storage节点获取到对应的文件，而后再返回给用户；最后仍是建议，若是用到mogilefs，最好跑在centos6上，centos7系统环境较高，有些依赖可能成问题；在centos6上，我测试过，彻底没有上述的问题；咱们只须要安装perl-IO-AIO这一个依赖包就能够解决mogstored启动问题；centos6上须要用到的rpm包能够去个人网盘下载http://www.qiuhom.com/d/47e8f7ac56a445f1bfc6/；