Linux CentOS 大数据集群的基础配置(2) 手动编译特定版本CHD

mac2024-02-23 44

开始之前确保你具备以下条件(劝退警告)： 1，使用VMware中的一台虚拟机。 2，虚拟机操作系统为CentOS 6.9 64位 basic server版本。 3，每台虚拟机4G及以上内存，40G及以上硬盘。 4，虚拟机可以正常的访问互联网。

此安装记录只适合上述版本，其它版本不兼容，仅供参考。严格地执行此安装记录，将会消耗你1小时左右的时间。

关闭防火墙

service iptables stop chkconfig iptables off

关闭selinux

vim /etc/selinux/config

把SELINUX=enforcing注释，添加SELINUX=disabled

卸载已安装的jdk

首先查询已安装的jdk

rpm -qa | grep java

把查询出来的这些全部卸载掉

rpm -e tzdata-java-2016j-1.el6.noarch java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el6_8.x86_64 java-1.7.0-openjdk-1.7.0.131-2.6.9.0.el6_8.x86_64

再次查询，看看是否完全卸载了

rpm -qa | grep java

ok，全部卸载完成

安装jdk1.7.0_71

首先创建两个目录

mkdir -p /export/servers mkdir -p /export/softwares

使用rz将jdk安装包上传到/export/softwares jdk1.7.0_71下载地址： https://www.oracle.com/technetwork/java/javase/archive-139210.html 如果没有rz命令，可以输入以下命令安装lrzsz，然后就能用rz命令了。

yum -y install lrzsz

安装lrzsz成功，现在切换到softwares目录，开始上传

cd /export/softwares/ rz

上传完成，把它解压到/export/servers/目录下

cd /export/softwares/ tar -zxvf jdk-7u71-linux-i586.tar.gz -C /export/servers/

配置环境变量：

vim /etc/profile.d/java.sh

输入以下内容保存退出

export JAVA_HOME=/export/servers/jdk1.7.0_71 export PATH=$PATH:$JAVA_HOME/bin

source一下profile，重新加载环境变量

source /etc/profile

输入如下命令，检查jdk1.7.0_71是否安装成功

java -version

安装maven3.0.5

在这里下载maven3.0.5 https://archive.apache.org/dist/maven/maven-3/

将maven的安装包上传到/export/softwares 然后解压maven的安装包到/export/servers

cd /export/softwares/ tar -zxvf apache-maven-3.0.5-bin.tar.gz -C ../servers/

然后配置maven的环境变量

vim /etc/profile.d/maven.sh

输入以下内容，保存退出

export MAVEN_HOME=/export/servers/apache-maven-3.0.5 export MAVEN_OPTS="-Xms4096m -Xmx4096m" export PATH=$PATH:$MAVEN_HOME/bin

让修改立刻生效

配置MAVEN仓库

上传本地仓库的压缩文件到/export/softwares/

把它解压到/export/servers/

cd /export/softwares/ unzip repository.zip -d ../servers/

修改maven的配置文件

cd /export/servers/apache-maven-3.0.5/conf vim settings.xml

找到修改为如下内容：再查找mirrors 修改为下图内容，换成阿里云的镜像保存退出，maven本地仓库和阿里源就配置好了。

安装findbugs1.3.9

下载地址： https://sourceforge.net/projects/findbugs/files/findbugs/1.3.9/ 下载完成后，把它上传到/export/softwares/目录下然后把它解压到/export/servers/目录下

cd /export/softwares/ tar -zxvf findbugs-1.3.9.tar.gz -C ../servers/

配置findbugs的环境变量

vim /etc/profile.d/findbugs.sh

输入以下内容保存退出

export FINDBUGS_HOME=/export/servers/findbugs-1.3.9 export PATH=$PATH:$FINDBUGS_HOME/bin

让修改立即生效

source /etc/profile

安装一些依赖包

yum install -y autoconf automake libtool cmake yum install -y ncurses-devel yum install -y openssl-devel yum install -y lzo-devel zlib-devel gcc gcc-c++ # bzip2压缩需要的依赖包 yum install -y bzip2-devel

安装protobuf2.5.0

下载地址： https://github.com/protocolbuffers/protobuf/releases

下载完成后上传到/export/softwares/目录下上传完成后，解压到/export/servers/目录下

cd /export/softwares/ tar -zxvf protobuf-2.5.0.tar.gz -C ../servers/

进入它的目录里

cd /export/servers/protobuf-2.5.0

执行如下命令

./configure

再执行如下命令

make && make install

protobuf安装完成。

安装snappy1.1.1

下载地址：官网http://google.github.io/snappy/

Snappy1.1.1下载链接，随时可能失效

下载完后上传到/export/servers/目录下上传完成后，解压到/export/servers/目录下

cd /export/softwares/ tar -zxvf snappy-1.1.1.tar.gz -C ../servers/

切换到解压的目录下

cd /export/servers/snappy-1.1.1/

执行如下命令

./configure

执行如下命令

make && make install

下载cdh源码准备编译

下载地址： http://archive.cloudera.com/cdh5/cdh/5/Hadoop-2.6.0-cdh5.14.0-src.tar.gz

下载完后上传到/export/softwares/目录下上传完成后，解压到/export/servers/目录下

cd /export/softwares/ tar -zxvf hadoop-2.6.0-cdh5.14.0-src.tar.gz -C ../servers/

进入它的目录

cd /export/servers/hadoop-2.6.0-cdh5.14.0

下面步骤需要联网

编译不支持snappy压缩（不执行这个）：

mvn package -Pdist,native -DskipTests –Dtar

编译支持snappy压缩(我们执行这个！)：

mvn package -DskipTests -Pdist,native -Dtar -Drequire.snappy -e -X

然而很遗憾，编译失败了，由于时间原因，这个记录被保留下来，以后有时间了，再去解决这个bug

[root@test01 hadoop-2.6.0-cdh5.14.0]# mvn package -DskipTests -Pdist,native -Dtar -D require.snappy -e -X Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800) Maven home: /export/servers/apache-maven-3.0.5 Java version: 1.7.0_71, vendor: Oracle Corporation Java home: /export/servers/jdk1.7.0_71/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "2.6.32-696.el6.x86_64", arch: "amd64", family: "unix" [INFO] Error stacktraces are turned on. [DEBUG] Reading global settings from /export/servers/apache-maven-3.0.5/conf/settings.xml [DEBUG] Reading user settings from /root/.m2/settings.xml [DEBUG] Using local repository at /root/.m2/repository [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10 for /root/.m2/repository [INFO] Scanning for projects... [DEBUG] Using connector WagonRepositoryConnector with priority 0 for https://repository.cloudera.com/artifactory/cloudera-repos Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom [DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated [DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated [DEBUG] Using connector WagonRepositoryConnector with priority 0 for http://repository.jboss.org/nexus/content/groups/public/ Downloading: http://repository.jboss.org/nexus/content/groups/public/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom [DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated [DEBUG] Using connector WagonRepositoryConnector with priority 0 for http://repo.maven.apache.org/maven2 Downloading: http://repo.maven.apache.org/maven2/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom [DEBUG] Writing resolution tracking file /root/.m2/repository/com/cloudera/cdh/cdh-root/5.14.0/cdh-root-5.14.0.pom.lastUpdated [ERROR] The build could not read 1 project -> [Help 1] org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated and 'parent.relativePath' points at wrong local POM @ line 19, column 11 at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:363) at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636) at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) [ERROR] [ERROR] The project org.apache.hadoop:hadoop-main:2.6.0-cdh5.14.0 (/export/servers/hadoop-2.6.0-cdh5.14.0/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated and 'parent.relativePath' points at wrong local POM @ line 19, column 11 -> [Help 2] org.apache.maven.model.resolution.UnresolvableModelException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:159) at org.apache.maven.model.building.DefaultModelBuilder.readParentExternally(DefaultModelBuilder.java:813) at org.apache.maven.model.building.DefaultModelBuilder.readParent(DefaultModelBuilder.java:664) at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:310) at org.apache.maven.model.building.DefaultModelBuilder.build(DefaultModelBuilder.java:232) at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:410) at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:379) at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:343) at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:636) at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:585) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:234) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:538) at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216) at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:193) at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveArtifact(DefaultRepositorySystem.java:286) at org.apache.maven.project.ProjectModelResolver.resolveModel(ProjectModelResolver.java:155) ... 22 more Caused by: org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.14.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): peer not authenticated at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:951) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:941) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:669) at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.maven.wagon.TransferFailedException: peer not authenticated at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:892) at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116) at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88) at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:601) ... 4 more Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:421) at org.apache.maven.wagon.providers.http.httpclient.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:126) at org.apache.maven.wagon.providers.http.httpclient.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:572) at org.apache.maven.wagon.shared.http4.ConfigurableSSLSocketFactoryDecorator.connectSocket(ConfigurableSSLSocketFactoryDecorator.java:64) at org.apache.maven.wagon.providers.http.httpclient.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180) at org.apache.maven.wagon.providers.http.httpclient.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294) at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645) at org.apache.maven.wagon.providers.http.httpclient.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480) at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.maven.wagon.providers.http.httpclient.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:746) at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:886) ... 8 more [ERROR] [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException [root@test01 hadoop-2.6.0-cdh5.14.0]#

编译完成之后我们需要的压缩包就在下面这个路径里面 /export/servers/hadoop-2.6.0-cdh5.14.0/hadoop-dist/target

最新回复(0)