hadoop(六)rsync远程同步|xsync集群分发(完全分布式准备三)|8

前置环境准备：centos7克隆ip|机器名|映射关系|别名配置（hadoop完全分布式准备一）scp命令copy文件和配置(hadoop完全分布式准备二)

rsync远程同步工具

优点

rsync主要用于备份和镜像。具有速度快、避免复制相同内容和支持符号链接的优点。rsync和scp区别:用 rsync做文件的复制要比scp的速度快, rsync只对差异文件做更新。scp是把所有文件都复制过去。

基本语法

rsync -rvl $pdir/$fname $user(@hadoop$host: $pdir/$fname

命令 选项参数 要拷贝的文件路径/名称 目的用户@主机:目的路径名称选项参数说明选项 功能-r 递归-v 显示复制过程-l Copy符号链接

案例

将hadoop101机器上的/opt/software 同步到hadoop102的服务器上的对应目录下。

[shaozhiqi@hadoop101 ~]$ rsync -rvl /opt/software shaozhiqi@hadoop102:/opt/ shaozhiqi@hadoop102's password: \sending incremental file list software/hadoop-3.1.2.tar.gz software/jdk-8u211-linux-x64.tar.gz sent 527488768 bytes received 51 bytes 14066368.51 bytes/sec total size is 527424191 speedup is 1.00 [shaozhiqi@hadoop101 ~]$

查看102机器目录同步成功

[shaozhiqi@hadoop102 software]$ ll total 515068 -rw-rw-r--. 1 shaozhiqi shaozhiqi 332433589 Jun 23 04:55 hadoop-3.1.2.tar.gz -rw-rw-r--. 1 shaozhiqi shaozhiqi 194990602 Jun 23 04:56 jdk-8u211-linux-x64.tar.gz [shaozhiqi@hadoop102 software]$

xsync集群分发脚本

(1)需求:循环复制文件到所有节点的相同目录下(2)需求分析:在 home/shaozhiqi/bin这个目录下存放的脚本, shaozhiqi用户可以在系统任何地方直接执行。可用案例：如果我们修改了其中一台hadoop的某个xml的，需要在其他机器上都同步过去就可以这样做

需求实现在home/shao目录下创建bin目录,并在bin目录下 xsync创建文件,文件脚本实现

#!/bin/bash # $#：表示传递给脚本或函数的参数个数。 #1 获取输入参数个数，如果没有参数，直接退出 pcount=$# if((pcount==0)); then echo no args; exit; fi #2 获取文件名称 p1=$1 fname=`basename $p1` echo fname=$fname #3 获取上级目录到绝对路径 pdir=`cd -P $(dirname $p1); pwd` echo pdir=$pdir #4 获取当前用户名称 user=`whoami` #5循环 for (( host=103; host<105; host++)); do echo -------hadoop$host ----------------------- rsync -rvl $pdir/$fname $user@hadoop$host:$pdir done

basename shell 语法

[shaozhiqi@hadoop102 opt]$ basename module/hadoop-3.1.2/etc/hadoop/core-site.xml core-site.xml [shaozhiqi@hadoop102 opt]$ basename core-site.xml core-site.xml

dirname shell语法

[shaozhiqi@hadoop102 opt]$ dirname module/hadoop-3.1.2/etc/hadoop/core-site.xml module/hadoop-3.1.2/etc/hadoop [shaozhiqi@hadoop102 opt]$

whoami

[shaozhiqi@hadoop102 opt]$ whoami shaozhiqi [shaozhiqi@hadoop102 opt]$

赋权限

[shaozhiqi@hadoop102 bin]$ chmod 777 testxsync [shaozhiqi@hadoop102 bin]$ ll total 4 -rwxrwxrwx. 1 shaozhiqi shaozhiqi 520 Jun 23 05:48 testxsync

执行前查看103

[shaozhiqi@hadoop103 ~]$ ll total 0

执行前查看104

[shaozhiqi@hadoop104 ~]$ ll total 0

执行脚本将bin同步到103.104

[shaozhiqi@hadoop102 bin]$ testxsync bin/ fname=bin dirname: missing operand Try 'dirname --help' for more information. pdir=/home/shaozhiqi -------hadoop103 ----------------------- shaozhiqi@hadoop103's password: sending incremental file list bin/ bin/.testxsync.swo bin/.testxsync.swp bin/testxsync sent 25281 bytes received 73 bytes 2668.84 bytes/sec total size is 25052 speedup is 0.99 -------hadoop104 ----------------------- shaozhiqi@hadoop104's password: Permission denied, please try again. shaozhiqi@hadoop104's password: sending incremental file list bin/ bin/.testxsync.swo bin/.testxsync.swp bin/testxsync sent 25281 bytes received 73 bytes 2414.67 bytes/sec total size is 25052 speedup is 0.99 [shaozhiqi@hadoop102 bin]$

执行后查看103、104脚本执行成功

[shaozhiqi@hadoop103 ~]$ ll total 0 drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 23 06:13 bin [shaozhiqi@hadoop103 ~]$ [shaozhiqi@hadoop104 ~]$ ll total 0 drwxrwxr-x. 2 shaozhiqi shaozhiqi 64 Jun 23 06:13 bin [shaozhiqi@hadoop104 ~]$

说明我们[shaozhiqi@hadoop103 ~]$ mkdir 的文件都会在 /home/shaozhiqi/bin在 home/shaozhiqi/bin这个目录下存放的脚本, shaozhiqi用户可以在系统任何地方直接执行。如果有意外，导致无法识别，那就把它放在、usr/local/bin下

转载于:https://www.cnblogs.com/shaozhiqi/p/11534643.html

相关资源：hadoop运行模式（完全分布式）——xsync集群分发脚本