企业实战——Hadoop大数据平台三个节点实现(1个master+2个slave)

先在server1上停掉刚才的运行文件

[hadoop@server1 hadoop]$ sbin/stop-dfs.sh [hadoop@server1 hadoop]$ cd /tmp/ [hadoop@server1 tmp]$ rm -fr *

现在开始做三个节点的集群 在server2上新建hadoop用户(server3与server2相同操作)

[root@server2 ~]# useradd hadoop [root@server2 ~]# id hadoop

在三个节点上均安装nfs服务并且开启服务共享hadoop

[root@server1 ~]# yum install -y nfs-utils [root@server1 ~]# systemctl start rpcbind

在server1(master节点)上配置nfs服务，设置共享目录，并且开启服务

[root@server1 ~]# systemctl start nfs-server [root@server1 ~]# vim /etc/exports /home/hadoop *(rw,anonuid=1000,anongid=1000) [root@server1 ~]# exportfs -rv

[root@server1 ~]# showmount -e

在server2和server3上建立用户并且挂载server1上的nfs共享目录到本地

[root@server2 ~]# mount 172.25.5.1:/home/hadoop/ /home/hadoop/ [root@server2 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/rhel-root 17811456 1139376 16672080 7% / devtmpfs 1011572 0 1011572 0% /dev tmpfs 1023808 0 1023808 0% /dev/shm tmpfs 1023808 8700 1015108 1% /run tmpfs 1023808 0 1023808 0% /sys/fs/cgroup /dev/sda1 1038336 132704 905632 13% /boot tmpfs 204764 0 204764 0% /run/user/0 172.25.5.1:/home/hadoop 17811456 2973184 14838272 17% /home/hadoop [root@server2 ~]# su - hadoop [hadoop@server2 ~]$ ls hadoop hadoop-3.2.1.tar.gz jdk1.8.0_181 hadoop-3.2.1 java jdk-8u181-linux-x64.tar.gz

[root@server3 ~]# mount 172.25.5.1:/home/hadoop/ /home/hadoop/ [root@server3 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/rhel-root 17811456 1139116 16672340 7% / devtmpfs 1011572 0 1011572 0% /dev tmpfs 1023808 0 1023808 0% /dev/shm tmpfs 1023808 8700 1015108 1% /run tmpfs 1023808 0 1023808 0% /sys/fs/cgroup /dev/sda1 1038336 132704 905632 13% /boot tmpfs 204764 0 204764 0% /run/user/0 172.25.5.1:/home/hadoop 17811456 2973184 14838272 17% /home/hadoop [root@server3 ~]# su - hadoop [hadoop@server3 ~]$ ls hadoop hadoop-3.2.1.tar.gz jdk1.8.0_181 hadoop-3.2.1 java jdk-8u181-linux-x64.tar.gz

在server1上修改相关的文件

[hadoop@server1 ~]# cd /home/hadoop/hadoop/etc/hadoop [hadoop@server1 hadoop]# vim hdfs-site.xml

[hadoop@server1 hadoop]# vim workers

由于家目录共享，因此三个节点上的公钥和私钥是一样的，因此可以互相免密连接 在server1上再次开启服务，发现datanode转移到了slave节点上面

[hadoop@server1 hadoop]# bin/hdfs namenode -format ##初始化

[hadoop@server1 hadoop]# sbin/start-dfs.sh [hadoop@server1 hadoop]$ jps

[hadoop@server2 hadoop]$ pwd /home/hadoop/hadoop/etc/hadoop [hadoop@server2 hadoop]$ cat workers 172.25.5.2 172.25.5.3 [hadoop@server2 hadoop]$ jps 11356 Jps 11277 DataNode

[hadoop@server3 hadoop]$ pwd /home/hadoop/hadoop/etc/hadoop [hadoop@server3 hadoop]$ cat workers 172.25.5.2 172.25.5.3 [hadoop@server3 hadoop]$ jps 11347 Jps 11269 DataNode

在master节点上再次上传一个文件并且在浏览器里面查看

[hadoop@server1 hadoop]$ pwd /home/hadoop/hadoop [hadoop@server1 hadoop]$ ls bin include libexec logs README.txt share etc lib LICENSE.txt NOTICE.txt sbin [hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir -p /user/hadoop [hadoop@server1 hadoop]$ bin/hdfs dfs -mkdir input [hadoop@server1 hadoop]$ bin/hdfs dfs -put etc/hadoop/*.xml input

[hadoop@server1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'

在浏览器查看