下载择操做系统对应的基础安装包
下载扩展包(不区分操做系统)html
http://www.oracle.com/technetwork/cn/server-storage/virtualbox/downloads/index.htmljava
按照提示安装便可node
1 先安装基础包linux
2 安装扩展包apache
打开虚拟机 -> 管理 -> 全局设定 -> 扩展 -> 点击右边的加号 -> 选择下载的扩展包文件,按照提示安装便可vim
http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1804.isocentos
从镜像列表里选择离本身近的镜像下载,我选择的是网易163源bash
CPU 2核,内存 1G,SWAP:2G,硬盘 40G(动态增加,非预先分配固定模式)网络
新建 -> 按照提示操做便可oracle
选中刚才建立的虚拟机 -> 设置 -> 存储 -> 点击+号,而后选择CentOS7 ISO文件
启动虚拟机,会显示图形化安装界面,按照提示操做便可,会提示一系列的设置,硬盘分区我选择的自动分区。
各类设置都完成后,点击安装,等待一段时间,安装须要一些时间,个人机器上,大概等了十几分钟:)
最好选择NAT网络模式
由于是安装的是最小化ISO,有些基础的包都没有
yum install gcc wget lrzsz vim
1 NAT网络虚拟机能够ping通主机,但主机ping不通虚拟机
采用了如下办法都不能解决(PS:之前是能够的)
1.关闭主机,虚拟机防火墙
2.从新安装VBox和虚拟机
折腾了几小时,未找到缘由,先暂时使用桥接模式,后面再看看能不能想到是什么缘由
使用3台虚拟机
1台master, ip地址:192.168.1.15
2台slave, slave1 ip地址:192.168.1.15, slave2 ip地址:192.168.1.16
实际环境中,namenode须要多分配内存,datanode须要多分配硬盘空间
选择安装java8
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
tar -xvzf jdk-8u181-linux-x64.tar.gz
export JAVA_HOME=/usr/local/src/jdk1.8.0_181 export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib export PATH=$PATH:$JAVA_HOME/bin
source ~/.bashrc
2.选择tar.gz仍是rpm?
这个看我的喜爱吧,我选择的tar.gz,须要单独配置java相关的环境变量
[root@localhost src]# tar -xvzf hadoop-1.2.1-bin.tar.gz
[root@localhost src]# cd hadoop-1.2.1 [root@localhost hadoop-1.2.1]# mkdir tmp
进入conf目录
[root@localhost hadoop-1.2.1]# cd conf [root@localhost conf]# pwd /usr/local/src/hadoop-1.2.1/conf
[root@localhost conf]# vim masters master
[root@localhost conf]# vim slaves slave1 slave2
vim core-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/src/hadoop-1.2.1/tmp</value> </property> <property> <name>fs.default.name</name> <value>hdfs://192.168.1.15:9000</value> </property> </configuration>
[root@localhost conf]# vim mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapred.job.tracker</name> <value>http://192.168.1.15:9001</value> </property> </configuration>
[root@localhost conf]# vim hdfs-site.xml <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>
6.配置hadoop-env.sh
[root@localhost conf]# vim hadoop-env.sh # 增长 export JAVA_HOME=/usr/local/src/jdk1.8.0_181
[root@localhost conf]# vim /etc/hosts 192.168.1.15 master 192.168.1.16 slave1 192.168.1.17 slave2
[root@localhost conf]# hostnamectl set-hostname master [root@localhost conf]# hostnamectl status Static hostname: master Icon name: computer-vm Chassis: vm Machine ID: 8751162d551a426393cd5e5c2fadf3d3 Boot ID: 4d3093f75e514da399ff522bea8b420f Virtualization: kvm Operating System: CentOS Linux 7 (Core) CPE OS Name: cpe:/o:centos:centos:7 Kernel: Linux 3.10.0-862.el7.x86_64 Architecture: x86-64
从master克隆一份(克隆以前,先退出虚拟机)
选中master虚拟机->点击鼠标右键->复制->设置虚拟名(勾选从新初始化全部网卡地址)->接下来的步骤按提示操做
hostnamectl set-hostname slave2
操做同slave1,只不过主机名设置为slave2
1.三台机分别生成rsa非对称秘钥
# master [wadeyu@master ~]$ su root Password: [root@master wadeyu]# ssh-keygen # slave1 [wadeyu@slave1 ~]$ su root Password: [root@slave1 wadeyu]# ssh-keygen # slave2 [wadeyu@slave2 ~]$ su root Password: [root@slave2 wadeyu]# ssh-keygen
2.保存公钥到~/.ssh/authorized_keys文件中
# master机器操做 [root@master wadeyu]# cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys 追加slave1和slave2的公钥到这个文件中 [root@master wadeyu]# scp slave1:~/.ssh/id_rsa.pub ~/slave1_id_rsa.pub [root@master wadeyu]# scp slave2:~/.ssh/id_rsa.pub ~/slave2_id_rsa.pub [root@master wadeyu]# cat ~/slave1_id_rsa.pub >> ~/.ssh/authorized_keys [root@master wadeyu]# cat ~/slave2_id_rsa.pub >> ~/.ssh/authorized_keys 复制文件~/.ssh/authorized_keys到slave1,slave2 [root@master wadeyu]# scp ~/.ssh/authorized_keys slave1:~/.ssh root@slave1's password: authorized_keys 100% 1179 458.2KB/s 00:00 [root@master wadeyu]# scp ~/.ssh/authorized_keys slave2:~/.ssh root@slave2's password: authorized_keys
为了减小系统配置对集群的影响,学习环境关闭防火墙和selinux
1.关闭防火墙
[root@master wadeyu]# systemctl stop firewalld [root@master wadeyu]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) Active: inactive (dead) since Sat 2018-09-01 11:26:29 CST; 5s ago Docs: man:firewalld(1) Process: 635 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (code=exited, status=0/SUCCESS) Main PID: 635 (code=exited, status=0/SUCCESS) Sep 01 10:23:16 master systemd[1]: Starting firewalld - dynamic firewall daemon... Sep 01 10:23:18 master systemd[1]: Started firewalld - dynamic firewall daemon. Sep 01 11:26:21 master systemd[1]: Stopping firewalld - dynamic firewall daemon... Sep 01 11:26:29 master systemd[1]: Stopped firewalld - dynamic firewall daemon.
[root@master wadeyu]# getenforce Enforcing [root@master wadeyu]# setenforce 0 [root@master wadeyu]# getenforce Permissive
master节点操做,进入hadoop/bin目录
[root@master wadeyu]# cd /usr/local/src/hadoop-1.2.1 hadoop-1.2.1/ hadoop-1.2.1-bin.tar.gz [root@master wadeyu]# cd /usr/local/src/hadoop-1.2.1 [root@master hadoop-1.2.1]# cd /usr/local/src/hadoop-1.2.1/bin [root@master bin]# ./hadoop namenode -format 18/09/01 11:37:07 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = master/192.168.1.15 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.8.0_181 ************************************************************/ 18/09/01 11:37:08 INFO util.GSet: Computing capacity for map BlocksMap 18/09/01 11:37:08 INFO util.GSet: VM type = 64-bit 18/09/01 11:37:08 INFO util.GSet: 2.0% max memory = 1013645312 18/09/01 11:37:08 INFO util.GSet: capacity = 2^21 = 2097152 entries 18/09/01 11:37:08 INFO util.GSet: recommended=2097152, actual=2097152 18/09/01 11:37:08 INFO namenode.FSNamesystem: fsOwner=root 18/09/01 11:37:08 INFO namenode.FSNamesystem: supergroup=supergroup 18/09/01 11:37:08 INFO namenode.FSNamesystem: isPermissionEnabled=true 18/09/01 11:37:08 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 18/09/01 11:37:08 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 18/09/01 11:37:08 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 18/09/01 11:37:08 INFO namenode.NameNode: Caching file names occuring more than 10 times 18/09/01 11:37:09 INFO common.Storage: Image file /usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds. 18/09/01 11:37:09 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits 18/09/01 11:37:09 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/src/hadoop-1.2.1/tmp/dfs/name/current/edits 18/09/01 11:37:09 INFO common.Storage: Storage directory /usr/local/src/hadoop-1.2.1/tmp/dfs/name has been successfully formatted. 18/09/01 11:37:09 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.15 ************************************************************/
2.启动全部节点
[root@master bin]# ./start-all.sh starting namenode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-wadeyu-namenode-master.out slave2: starting datanode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-slave2.out slave1: starting datanode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-slave1.out The authenticity of host 'master (192.168.1.15)' can't be established. ECDSA key fingerprint is SHA256:8DvdHBlcz1qInlLa9k2iYyd4Ip7auPhcb0mjHbEwZmo. ECDSA key fingerprint is MD5:9e:33:01:d2:fb:9c:dc:4f:40:30:90:fe:37:6e:1f:33. Are you sure you want to continue connecting (yes/no)? yes master: Warning: Permanently added 'master,192.168.1.15' (ECDSA) to the list of known hosts. master: starting secondarynamenode, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-master.out starting jobtracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-wadeyu-jobtracker-master.out slave1: starting tasktracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-slave1.out slave2: starting tasktracker, logging to /usr/local/src/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-slave2.out
3.查看集群状态
# master [root@master bin]# jps 2116 JobTracker 2232 Jps 1883 NameNode 2044 SecondaryNameNode # slave1 [root@master bin]# ssh slave1 Last login: Sat Sep 1 11:20:05 2018 from slave2 [root@slave1 ~]# jps 3936 Jps 1617 TaskTracker 1538 DataNode #slave2 [root@slave1 ~]# exit logout Connection to slave1 closed. [root@master bin]# ssh slave2 Last login: Sat Sep 1 11:20:24 2018 from slave1 [root@slave2 ~]# jps 3774 TaskTracker 3695 DataNode 3871 Jps
4.hadoop文件操做示例
# 查看/ [root@master bin]# ./hadoop fs -ls / Found 1 items drwxr-xr-x - root supergroup 0 2018-09-01 11:38 /usr # 上传文件 [root@master bin]# ./hadoop fs -put /etc/passwd / [root@master bin]# ./hadoop fs -ls / Found 2 items -rw-r--r-- 3 root supergroup 847 2018-09-01 11:44 /passwd drwxr-xr-x - root supergroup 0 2018-09-01 11:38 /usr # 查看文件内容 [root@master bin]# ./hadoop fs -cat /passwd root:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin operator:x:11:0:operator:/root:/sbin/nologin games:x:12:100:games:/usr/games:/sbin/nologin ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin nobody:x:99:99:Nobody:/:/sbin/nologin systemd-network:x:192:192:systemd Network Management:/:/sbin/nologin dbus:x:81:81:System message bus:/:/sbin/nologin polkitd:x:999:998:User for polkitd:/:/sbin/nologin sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin postfix:x:89:89::/var/spool/postfix:/sbin/nologin wadeyu:x:1000:1000:wadeyu:/home/wadeyu:/bin/bash
【0】八斗学院内部学习资料