目标
单机搭建mongodb分布式集群(副本集 + 分片集群),演示mongodb分布式集群的安装部署、简单操做。html
说明
在同一个vm启动由两个分片组成的分布式集群,每一个分片都是一个PSS(Primary-Secondary-Secondary)模式的数据副本集;
Config副本集采用PSS(Primary-Secondary-Secondary)模式。node
编号 | 实例类型 |
---|---|
1 | mongos |
2 | mongos |
3 | mongos |
4 | config |
5 | config |
6 | config |
7 | shard1 |
8 | shard1 |
9 | shard1 |
10 | shard2 |
11 | shard2 |
12 | shard2 |
内部鉴权
节点间鉴权采用keyfile方式实现鉴权,mongos与分片之间、副本集节点之间共享同一套keyfile文件。 官方说明linux
帐户设置
管理员帐户:admin/Admin@01,具备集群及全部库的管理权限
应用帐号:appuser/AppUser@01,具备appdb的owner权限算法
关于初始化权限
keyfile方式默认会开启鉴权,而针对初始化安装的场景,Mongodb提供了localhost-exception机制,
能够在首次安装时经过本机建立用户、角色,以及副本集初始操做。mongodb
官方地址:https://www.mongodb.com/download-center数据库
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.6.3.tgz
解压压缩文件,将bin目录拷贝到目标路径/opt/local/mongo-cluster,参考如下命令:架构
tar -xzvf mongodb-linux-x86_64-rhel70-3.6.3.tgz mkdir -p /opt/local/mongo-cluster cp -r mongodb-linux-x86_64-rhel70-3.6.3/bin /opt/local/mongo-cluster
cd /opt/local/mongo-cluster mkdir conf
A. mongod 配置文件 mongo_node.conf
mongo_node.conf 做为mongod实例共享的配置文件,内容以下:app
storage: engine: wiredTiger directoryPerDB: true journal: enabled: true systemLog: destination: file logAppend: true operationProfiling: slowOpThresholdMs: 10000 replication: oplogSizeMB: 10240 processManagement: fork: true net: http: enabled: false security: authorization: "enabled"
选项说明可参考这里分布式
B. mongos 配置文件 mongos.conf工具
systemLog: destination: file logAppend: true processManagement: fork: true net: http: enabled: false
cd /opt/local/mongo-cluster mkdir keyfile openssl rand -base64 756 > mongo.key chmod 400 mongo.key mv mongo.key keyfile
mongo.key 采用随机算法生成,用做节点内部通信的密钥文件
WORK_DIR=/opt/local/mongo-cluster mkdir -p $WORK_DIR/nodes/config/n1/data mkdir -p $WORK_DIR/nodes/config/n2/data mkdir -p $WORK_DIR/nodes/config/n3/data mkdir -p $WORK_DIR/nodes/shard1/n1/data mkdir -p $WORK_DIR/nodes/shard1/n2/data mkdir -p $WORK_DIR/nodes/shard1/n3/data mkdir -p $WORK_DIR/nodes/shard2/n1/data mkdir -p $WORK_DIR/nodes/shard2/n2/data mkdir -p $WORK_DIR/nodes/shard2/n3/data mkdir -p $WORK_DIR/nodes/mongos/n1 mkdir -p $WORK_DIR/nodes/mongos/n2 mkdir -p $WORK_DIR/nodes/mongos/n3
以config 节点1 为例,nodes/config/n1/data是数据目录,而pid文件、日志文件都存放于n1目录
以mongos 节点1 为例,nodes/mongos/n1 存放了pid文件和日志文件
按如下脚本启动3个Config实例
WORK_DIR=/opt/local/mongo-cluster KEYFILE=$WORK_DIR/keyfile/mongo.key CONFFILE=$WORK_DIR/conf/mongo_node.conf MONGOD=$WORK_DIR/bin/mongod $MONGOD --port 26001 --configsvr --replSet configReplSet --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/config/n1/data --pidfilepath $WORK_DIR/nodes/config/n1/db.pid --logpath $WORK_DIR/nodes/config/n1/db.log --config $CONFFILE $MONGOD --port 26002 --configsvr --replSet configReplSet --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/config/n2/data --pidfilepath $WORK_DIR/nodes/config/n2/db.pid --logpath $WORK_DIR/nodes/config/n2/db.log --config $CONFFILE $MONGOD --port 26003 --configsvr --replSet configReplSet --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/config/n3/data --pidfilepath $WORK_DIR/nodes/config/n3/db.pid --logpath $WORK_DIR/nodes/config/n3/db.log --config $CONFFILE
待成功启动后,输出日志以下:
about to fork child process, waiting until server is ready for connections. forked process: 4976 child process started successfully, parent exiting
此时经过ps 命令也能够看到3个启动的进程实例。
链接其中一个Config进程,执行副本集初始化
./bin/mongo --port 26001 --host 127.0.0.1 > MongoDB server version: 3.4.7 > cfg={ _id:"configReplSet", configsvr: true, members:[ {_id:0, host:'127.0.0.1:26001'}, {_id:1, host:'127.0.0.1:26002'}, {_id:2, host:'127.0.0.1:26003'} ]}; rs.initiate(cfg);
其中configsvr:true指明这是一个用于分片集群的Config副本集。
关于副本集配置可参考这里
按如下脚本启动Shard1的3个实例
WORK_DIR=/opt/local/mongo-cluster KEYFILE=$WORK_DIR/keyfile/mongo.key CONFFILE=$WORK_DIR/conf/mongo_node.conf MONGOD=$WORK_DIR/bin/mongod echo "start shard1 replicaset" $MONGOD --port 27001 --shardsvr --replSet shard1 --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/shard1/n1/data --pidfilepath $WORK_DIR/nodes/shard1/n1/db.pid --logpath $WORK_DIR/nodes/shard1/n1/db.log --config $CONFFILE $MONGOD --port 27002 --shardsvr --replSet shard1 --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/shard1/n2/data --pidfilepath $WORK_DIR/nodes/shard1/n2/db.pid --logpath $WORK_DIR/nodes/shard1/n2/db.log --config $CONFFILE $MONGOD --port 27003 --shardsvr --replSet shard1 --keyFile $KEYFILE --dbpath $WORK_DIR/nodes/shard1/n3/data --pidfilepath $WORK_DIR/nodes/shard1/n3/db.pid --logpath $WORK_DIR/nodes/shard1/n3/db.log --config $CONFFILE
待成功启动后,输出日志以下:
about to fork child process, waiting until server is ready for connections. forked process: 5976 child process started successfully, parent exiting
此时经过ps 命令也能够看到3个启动的Shard进程实例。
链接其中一个Shard进程,执行副本集初始化
./bin/mongo --port 27001 --host 127.0.0.1 > MongoDB server version: 3.4.7 > cfg={ _id:"shard1", members:[ {_id:0, host:'127.0.0.1:27001'}, {_id:1, host:'127.0.0.1:27002'}, {_id:2, host:'127.0.0.1:27003'} ]}; rs.initiate(cfg);
参考以上步骤,启动Shard2的3个实例进程,并初始化副本集。
执行如下脚本启动3个mongos进程
WORK_DIR=/opt/local/mongo-cluster KEYFILE=$WORK_DIR/keyfile/mongo.key CONFFILE=$WORK_DIR/conf/mongos.conf MONGOS=$WORK_DIR/bin/mongos echo "start mongos instances" $MONGOS --port=25001 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile $KEYFILE --pidfilepath $WORK_DIR/nodes/mongos/n1/db.pid --logpath $WORK_DIR/nodes/mongos/n1/db.log --config $CONFFILE $MONGOS --port 25002 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile $KEYFILE --pidfilepath $WORK_DIR/nodes/mongos/n2/db.pid --logpath $WORK_DIR/nodes/mongos/n2/db.log --config $CONFFILE $MONGOS --port 25003 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile $KEYFILE --pidfilepath $WORK_DIR/nodes/mongos/n3/db.pid --logpath $WORK_DIR/nodes/mongos/n3/db.log --config $CONFFILE
待成功启动后,经过ps命令看到mongos进程:
dbuser 7903 1 0 17:49 ? 00:00:00 /opt/local/mongo-cluster/bin/mongos --port=25001 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile /opt/local/mongo-cluster/keyfile/mongo.key --pidfilepath /opt/local/mongo-cluster/nodes/mongos/n1/db.pid --logpath /opt/local/mongo-cluster/nodes/mongos/n1/db.log --config /opt/local/mongo-cluster/conf/mongos.conf dbuser 7928 1 0 17:49 ? 00:00:00 /opt/local/mongo-cluster/bin/mongos --port 25002 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile /opt/local/mongo-cluster/keyfile/mongo.key --pidfilepath /opt/local/mongo-cluster/nodes/mongos/n2/db.pid --logpath /opt/local/mongo-cluster/nodes/mongos/n2/db.log --config /opt/local/mongo-cluster/conf/mongos.conf dbuser 7954 1 0 17:49 ? 00:00:00 /opt/local/mongo-cluster/bin/mongos --port 25003 --configdb configReplSet/127.0.0.1:26001,127.0.0.1:26002,127.0.0.1:26003 --keyFile /opt/local/mongo-cluster/keyfile/mongo.key --pidfilepath /opt/local/mongo-cluster/nodes/mongos/n3/db.pid --logpath /opt/local/mongo-cluster/nodes/mongos/n3/db.log --config /opt/local/mongo-cluster/conf/mongos.conf
接入其中一个mongos实例,执行添加分片操做:
./bin/mongo --port 25001 --host 127.0.0.1 mongos> MongoDB server version: 3.4.7 mongos> sh.addShard("shard1/127.0.0.1:27001") { "shardAdded" : "shard1", "ok" : 1 } mongos> sh.addShard("shard2/127.0.0.1:27004") { "shardAdded" : "shard2", "ok" : 1 }
至此,分布式集群架构启动完毕,但进一步操做须要先添加用户。
接入其中一个mongos实例,添加管理员用户
use admin db.createUser({ user:'admin',pwd:'Admin@01', roles:[ {role:'clusterAdmin',db:'admin'}, {role:'userAdminAnyDatabase',db:'admin'}, {role:'dbAdminAnyDatabase',db:'admin'}, {role:'readWriteAnyDatabase',db:'admin'} ]})
当前admin用户具备集群管理权限、全部数据库的操做权限。
须要注意的是,在第一次建立用户以后,localexception再也不有效,接下来的全部操做要求先经过鉴权。
use admin db.auth('admin','Admin@01')
检查集群状态
mongos> sh.status() --- Sharding Status --- sharding version: { "_id" : 1, "minCompatibleVersion" : 5, "currentVersion" : 6, "clusterId" : ObjectId("5aa39c3e915210dc501a1dc8") } shards: { "_id" : "shard1", "host" : "shard1/127.0.0.1:27001,127.0.0.1:27002,127.0.0.1:27003", "state" : 1 } { "_id" : "shard2", "host" : "shard2/127.0.0.1:27004,127.0.0.1:27005,127.0.0.1:27006", "state" : 1 } active mongoses: "3.4.7" : 3 autosplit: Currently enabled: yes
集群用户
分片集群中的访问都会经过mongos入口,而鉴权数据是存储在config副本集中的,即config实例中system.users数据库存储了集群用户及角色权限配置。mongos与shard实例则经过内部鉴权(keyfile机制)完成,所以shard实例上能够经过添加本地用户以方便操做管理。在一个副本集上,只须要在Primary节点上添加用户及权限,相关数据会自动同步到Secondary节点。
关于集群鉴权
在本案例中,咱们为两个分片副本集都添加了本地admin用户。
经过mongostat工具能够显示集群全部角色:
host insert query update delete getmore command dirty used flushes mapped vsize res faults qrw arw net_in net_out conn set repl time 127.0.0.1:27001 *0 *0 *0 *0 0 6|0 0.1% 0.1% 0 1.49G 44.0M n/a 0|0 0|0 429b 56.1k 25 shard1 PRI Mar 10 19:05:13.928 127.0.0.1:27002 *0 *0 *0 *0 0 7|0 0.1% 0.1% 0 1.43G 43.0M n/a 0|0 0|0 605b 55.9k 15 shard1 SEC Mar 10 19:05:13.942 127.0.0.1:27003 *0 *0 *0 *0 0 7|0 0.1% 0.1% 0 1.43G 43.0M n/a 0|0 0|0 605b 55.9k 15 shard1 SEC Mar 10 19:05:13.946 127.0.0.1:27004 *0 *0 *0 *0 0 6|0 0.1% 0.1% 0 1.48G 43.0M n/a 0|0 0|0 546b 55.8k 18 shard2 PRI Mar 10 19:05:13.939 127.0.0.1:27005 *0 *0 *0 *0 0 6|0 0.1% 0.1% 0 1.43G 42.0M n/a 0|0 0|0 540b 54.9k 15 shard2 SEC Mar 10 19:05:13.944 127.0.0.1:27006 *0 *0 *0 *0 0 6|0 0.1% 0.1% 0 1.46G 44.0M n/a 0|0 0|0 540b 54.9k 17 shard2 SEC Mar 10 19:05:13.936
在案例中,建立appuser用户、为数据库实例appdb启动分片。
use appdb db.createUser({user:'appuser',pwd:'AppUser@01',roles:[{role:'dbOwner',db:'appdb'}]}) sh.enableSharding("appdb")
建立集合book,为其执行分片初始化。
use appdb db.createCollection("book") db.device.ensureIndex({createTime:1}) sh.shardCollection("appdb.book", {bookId:"hashed"}, false, { numInitialChunks: 4} )
继续往device集合写入1000W条记录,观察chunks的分布状况
use appdb var cnt = 0; for(var i=0; i<1000; i++){ var dl = []; for(var j=0; j<100; j++){ dl.push({ "bookId" : "BBK-" + i + "-" + j, "type" : "Revision", "version" : "IricSoneVB0001", "title" : "Jackson's Life", "subCount" : 10, "location" : "China CN Shenzhen Futian District", "author" : { "name" : 50, "email" : "RichardFoo@yahoo.com", "gender" : "female" }, "createTime" : new Date() }); } cnt += dl.length; db.book.insertMany(dl); print("insert ", cnt); }
执行db.book.getShardDistribution(),输出以下:
Shard shard1 at shard1/127.0.0.1:27001,127.0.0.1:27002,127.0.0.1:27003 data : 13.41MiB docs : 49905 chunks : 2 estimated data per chunk : 6.7MiB estimated docs per chunk : 24952 Shard shard2 at shard2/127.0.0.1:27004,127.0.0.1:27005,127.0.0.1:27006 data : 13.46MiB docs : 50095 chunks : 2 estimated data per chunk : 6.73MiB estimated docs per chunk : 25047 Totals data : 26.87MiB docs : 100000 chunks : 4 Shard shard1 contains 49.9% data, 49.9% docs in cluster, avg obj size on shard : 281B Shard shard2 contains 50.09% data, 50.09% docs in cluster, avg obj size on shard : 281B