CentOS 6.5编译安装hadoop-2.2.0

分类:CentOS教程 阅读:7809 次

这几天在琢磨Hadoop,首先是安装Hadoop,在安装过程中出现过不少问题,现在将整个过程总结一下,网络上已经有很多这方面的资料了,但我觉得还是有必要记述一下部分重要安装过程,方便以后发现与解决问题,也希望能给需要的朋友一些参考。

我所用系统是CentOS6.5 64bit,编译安装hadoop-2.2.0,hadoop配置为单节点。在ApacheHadoop下载的hadoop-2.2.0有两种版本:1)编译版本:hadoop-2.2.0.tar.gz,2)源代码版本:hadoop-2.2.0-src.tar.gz。对于编译版本,解压缩后进行配置就可以使用了,而对于源代码版本,先要编译,然后再进行配置。

我第一次装hadoop-2.2.0使用的是编译版本,但由于我的系统是64位,自带的native-hadooplibrary不能使用(自带的库适用于32位系统,但不适用于64位系统),总是出现下列提示:”WARNutil.NativeCodeLoader: Unable to load native-hadoop library for yourplatform... using builtin-java classes whereapplicable”(问题综述3.2),因此我采用源代码重新编译安装。

1编译hadoop-2.2.0源代码

1.1编译前环境准备

我编译源代码过程中参考的是这个篇博文:hadoop2.2.0centos编译安装详解,这其中需要安装的软件或者包有许多,可以分为两类:

  • yum安装:java,gcc,gcc-c++,make,lzo-devel,zlib-devel,gcc,autoconf,automake,libtool,ncurses-devel,openssl-devel。
  • 手动安装:Maven,ProtocolBuffer。

对于yum安装的软件或者依赖包,在CentOS 6.5中大部分可能都已经预先安装了,可以先先检查一下是否安装或者更新:yum info package,如果需要安装:yum -y install package,如果有可用更新:yum -y update package。

对于手动安装中的软件,需要先下载软件包,然后再安装,使用的具体版本是protobuf-2.5.0.tar.gz(http://download.csdn.net/detail/erli11/7408809,官方网站被wall了可以选择这个下载),apache-maven-3.0.5-bin.zip(mirror.bit.edu.cn/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.zip),protobuf需要采用源码安装;maven是编译版本,只需配置环境变量。但请注意:不要使用Maven3.1.1,因为Maven3.1.1与Maven3.0.x存在兼容性问题,不能成功下载插件,会出现问题maven-"ServiceUnavailable"。建议使用oschina的maven镜像,因为国外的某些网站可能会被Wall。这两个软件的安装都可参考上面的博客。

安装完上面所列出的软件或者依赖包后,需要配置系统环境变量,让系统能够找到软件所对应的命令,以下是我在/root/.bashrc中添加的配置:

  1. exportJAVA_HOME="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64"
  2. exportCLASSPATH=.:${JAVA_HOME}/lib/:${JAVA_HOME}/jre/lib/
  3. exportPATH=${JAVA_HOME}/bin:${JAVA_HOME}/jre/bin:$PATH
  4. exportMAVEN_HOME="/usr/local/src/apache-maven-3.0.5"
  5. exportPATH=$PATH:$MAVEN_HOME/bin
  6. exportPROTOBUF_HOME="/usr/local/protobuf"
  7. exportPATH=$PATH:$PROTOBUF_HOME/bin

在/root/.bashrc配置环境变量后需要使用命令source/root/.bashrc加载配置。检测java和maven是否安装成功,出现下列信息表示安装成功。

  1. [root@lls~]#java-version
  2. javaversion"1.7.0_71"
  3. OpenJDKRuntimeEnvironment(rhel-2.5.3.1.el6-x86_64u71-b14)
  4. OpenJDK64-BitServerVM(build24.65-b04,mixedmode)
  5. [root@lls~]#mvn-version
  6. ApacheMaven3.0.5(r01de14724cdef164cd33c7c8c2fe155faf9602da;2013-02-1921:51:28+0800)
  7. Mavenhome:/usr/local/src/apache-maven-3.0.5
  8. Javaversion:1.7.0_71,vendor:OracleCorporation
  9. Javahome:/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64/jre
  10. Defaultlocale:en_US,platformencoding:UTF-8
  11. OSname:"linux",version:"2.6.32-431.29.2.el6.x86_64",arch:"amd64",family:"unix"

1.2编译hadoop

下载hadoop-2.2.0(http://apache.fastbull.org/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz,官方已经不支持下载,可以在这里下载)源代码,hadoop-2.2.0的SourceCode压缩包解压出来的源代码有个bug需要patch后才能编译,参考:https://issues.apache.org/jira/browse/HADOOP-10110

一切准备就绪,开始编译:

  1. cd/home/xxx/softwares/hadoop/hadoop-2.2.0-src
  2. mvnpackage-Pdist,native-DskipTests-Dtar

需要等待一段时间,编译成功之后的结果如下(maven使用默认镜像):

  1. [INFO]ReactorSummary:
  2. [INFO]
  3. [INFO]ApacheHadoopMain................................SUCCESS[2.109s]
  4. [INFO]ApacheHadoopProjectPOM.........................SUCCESS[1.828s]
  5. [INFO]ApacheHadoopAnnotations.........................SUCCESS[5.266s]
  6. [INFO]ApacheHadoopAssemblies..........................SUCCESS[0.228s]
  7. [INFO]ApacheHadoopProjectDistPOM....................SUCCESS[2.184s]
  8. [INFO]ApacheHadoopMavenPlugins.......................SUCCESS[3.562s]
  9. [INFO]ApacheHadoopAuth................................SUCCESS[3.128s]
  10. [INFO]ApacheHadoopAuthExamples.......................SUCCESS[2.444s]
  11. [INFO]ApacheHadoopCommon..............................SUCCESS[1:17.748s]
  12. [INFO]ApacheHadoopNFS.................................SUCCESS[16.455s]
  13. [INFO]ApacheHadoopCommonProject......................SUCCESS[0.056s]
  14. [INFO]ApacheHadoopHDFS................................SUCCESS[2:18.736s]
  15. [INFO]ApacheHadoopHttpFS..............................SUCCESS[18.687s]
  16. [INFO]ApacheHadoopHDFSBookKeeperJournal.............SUCCESS[23.553s]
  17. [INFO]ApacheHadoopHDFS-NFS............................SUCCESS[3.453s]
  18. [INFO]ApacheHadoopHDFSProject........................SUCCESS[0.046s]
  19. [INFO]hadoop-yarn.......................................SUCCESS[48.652s]
  20. [INFO]hadoop-yarn-api...................................SUCCESS[44.591s]
  21. [INFO]hadoop-yarn-common................................SUCCESS[30.677s]
  22. [INFO]hadoop-yarn-server................................SUCCESS[0.096s]
  23. [INFO]hadoop-yarn-server-common.........................SUCCESS[9.340s]
  24. [INFO]hadoop-yarn-server-nodemanager....................SUCCESS[16.656s]
  25. [INFO]hadoop-yarn-server-web-proxy......................SUCCESS[3.115s]
  26. [INFO]hadoop-yarn-server-resourcemanager................SUCCESS[13.133s]
  27. [INFO]hadoop-yarn-server-tests..........................SUCCESS[0.614s]
  28. [INFO]hadoop-yarn-client................................SUCCESS[4.646s]
  29. [INFO]hadoop-yarn-applications..........................SUCCESS[0.100s]
  30. [INFO]hadoop-yarn-applications-distributedshell.........SUCCESS[2.815s]
  31. [INFO]hadoop-mapreduce-client...........................SUCCESS[0.096s]
  32. [INFO]hadoop-mapreduce-client-core......................SUCCESS[23.624s]
  33. [INFO]hadoop-yarn-applications-unmanaged-am-launcher....SUCCESS[2.056s]
  34. [INFO]hadoop-yarn-site..................................SUCCESS[0.099s]
  35. [INFO]hadoop-yarn-project...............................SUCCESS[11.009s]
  36. [INFO]hadoop-mapreduce-client-common....................SUCCESS[20.053s]
  37. [INFO]hadoop-mapreduce-client-shuffle...................SUCCESS[3.310s]
  38. [INFO]hadoop-mapreduce-client-app.......................SUCCESS[9.819s]
  39. [INFO]hadoop-mapreduce-client-hs........................SUCCESS[4.843s]
  40. [INFO]hadoop-mapreduce-client-jobclient.................SUCCESS[6.115s]
  41. [INFO]hadoop-mapreduce-client-hs-plugins................SUCCESS[1.682s]
  42. [INFO]ApacheHadoopMapReduceExamples..................SUCCESS[6.336s]
  43. [INFO]hadoop-mapreduce..................................SUCCESS[3.946s]
  44. [INFO]ApacheHadoopMapReduceStreaming.................SUCCESS[4.788s]
  45. [INFO]ApacheHadoopDistributedCopy....................SUCCESS[8.510s]
  46. [INFO]ApacheHadoopArchives............................SUCCESS[2.061s]
  47. [INFO]ApacheHadoopRumen...............................SUCCESS[7.269s]
  48. [INFO]ApacheHadoopGridmix.............................SUCCESS[4.815s]
  49. [INFO]ApacheHadoopDataJoin...........................SUCCESS[3.659s]
  50. [INFO]ApacheHadoopExtras..............................SUCCESS[3.132s]
  51. [INFO]ApacheHadoopPipes...............................SUCCESS[9.350s]
  52. [INFO]ApacheHadoopToolsDist..........................SUCCESS[1.850s]
  53. [INFO]ApacheHadoopTools...............................SUCCESS[0.023s]
  54. [INFO]ApacheHadoopDistribution........................SUCCESS[19.184s]
  55. [INFO]ApacheHadoopClient..............................SUCCESS[6.730s]
  56. [INFO]ApacheHadoopMini-Cluster........................SUCCESS[0.192s]
  57. [INFO]------------------------------------------------------------------------
  58. [INFO]BUILDSUCCESS
  59. [INFO]------------------------------------------------------------------------
  60. [INFO]Totaltime:10:40.193s
  61. [INFO]Finishedat:FriNov2114:43:06CST2014
  62. [INFO]FinalMemory:131M/471M
  63. [INFO]------------------------------------------------------------------------
编译后的文件为hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0。

2.单节点安装hadoop

以下采用single-node模式在CentOS6.5 64bits中安装hadoop-2.2.0。

2.1创建用户组及添加用户

  1. [root@llsDesktop]#groupaddhadoopgroup
  2. [root@llsDesktop]#useraddhadoopuser
  3. [root@llsDesktop]#passwdhadoopuserID-0217ef09-5d44-44dc-815a-f0e0569e0
  4. Changingpasswordforuserhadoopuser.
  5. Newpassword:
  6. Retypenewpassword:
  7. passwd:allauthenticationtokensupdatedsuccessfully.
  8. [root@llsDesktop]#usermod-ghadoopgrouphadoopuser

2.2.安装和配置SSH

Hadoop使用SSH的方式管理其节点,即使在single-node方式中也需要对其进行配置。否则会出现“connectionrefused on port 22”错误。在此之前,请确保您已经安装了SSH,如果没有,可以使用yum install openssh-server安装。

为hadoop用户产生一个SSH密钥,以后无需密码即可登录到hadoop节点:

注意:这步是切换到hadoopuser后执行的。

  1. [hadoopuser@lls~]$ssh-keygen-trsa
  2. Generatingpublic/privatersakeypair.
  3. Enterfileinwhichtosavethekey(/home/hadoopuser/.ssh/id_rsa):
  4. Enterpassphrase(emptyfornopassphrase):
  5. Entersamepassphraseagain:
  6. Youridentificationhasbeensavedin/home/hadoopuser/.ssh/id_rsa.
  7. Yourpublickeyhasbeensavedin/home/hadoopuser/.ssh/id_rsa.pub.
  8. Thekeyfingerprintis:
  9. 0b:6e:2f:89:a5:42:42:40:b2:69:fc:3f:4c:84:33:ebhadoopuser@lls.pc
  10. Thekey'srandomartimageis:
  11. +--[RSA2048]----+
  12. |o.|
  13. |+o.|
  14. |+o+.|
  15. |...=|
  16. |.o..S|
  17. |.o+....|
  18. |oEBo..|
  19. |.o.+.|
  20. |...|
  21. +-----------------+
使新创建的密钥能够访问本地主机:
  1. [hadoopuser@lls~]$cat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keys


2.3设置安装文件权限

我将hadoop安装在/usr/local中,将编译后的文件hadoop-2.2.0复制到/usr/local中,修改所有者:

  1. cp-R/home/xxx/softwares/hadoop/hadoop-2.2.0-src/hadoop-dist/target/hadoop-2.2.0/usr/local/
  2. cd/usr/local/
  3. mvhadoop-2.2.0/hadoop
  4. chown-Rhadoopuser:hadoopgrouphadoop/

2.4创建HDFS路径

  1. cd/usr/local/hadoop/
  2. mkdir-pdata/namenode
  3. mkdir-pdata/datanode
  4. mkdir-pdata/secondarydatanode

2.5配置hadoop-env.sh

在文件/usr/local/hadoop/etc/hadoop/hadoop-env.sh中,添加

  1. exportJAVA_HOME="/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71.x86_64"
  2. exportHADOOP_HOME="/usr/local/hadoop"
  3. exportHADOOP_MAPRED_HOME=$HADOOP_HOME
  4. exportHADOOP_COMMON_HOME=$HADOOP_HOME
  5. exportHADOOP_HDFS_HOME=$HADOOP_HOME
  6. exportYARN_HOME=$HADOOP_HOME
  7. exportHADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
  8. exportHADOOP_OPTS="$HADOOP_OPTS-Djava.library.path=/usr/local/hadoop/lib/native"
  9. exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

注意:要将原配置文件中下列的两行屏蔽或者直接删除,否则不能加载成功native-hadooplibrary

  1. exportJAVA_HOME=${JAVA_HOME}
  2. exportHADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

2.6配置core-site.xml

在文件/usr/local/hadoop/etc/hadoop/core-site.xml(在<configuration>标签内)添加:

  1. <property>
  2. <name>fs.default.name</name>
  3. <value>hdfs://localhost:9000</value>
  4. </property>

2.7配置hdfs-site.xml

在文件/usr/local/hadoop/etc/hadoop/hdfs-site.xml(<configuration>标签内)添加:

  1. <property>
  2. <name>dfs.replication</name>
  3. <value>1</value>
  4. </property>
  5. <property>
  6. <name>dfs.name.dir</name>
  7. <value>file:///usr/local/hadoop/data/namenode</value>
  8. </property>
  9. <property>
  10. <name>fs.checkpoint.dir</name>
  11. <value>file:///usr/local/hadoop/data/secondarynamenode</value>
  12. </property>
  13. <property>
  14. <name>dfs.data.dir</name>
  15. <value>file:///usr/local/hadoop/data/datanode</value>
  16. </property>


2.8配置yarn-site.xml

在文件/usr/local/hadoop/etc/hadoop/yarn-site.xml(<configuration>标签内)添加:

  1. <property>
  2. <name>yarn.nodemanager.aux-services</name>
  3. <value>mapreduce_shuffle</value>
  4. </property>
  5. <property>
  6. <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  7. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  8. </property>

2.9配置mapred-site.xml

创建mapred-site.xml文件:

  1. cp/usr/local/hadoop/etc/hadoop/mapred-site.xml.template/usr/local/hadoop/etc/hadoop/mapred-site.xml

在文件/usr/local/hadoop/etc/hadoop/mapred-site.xml(<configuration>标签内)添加:

  1. <property>
  2. <name>mapreduce.framework.name</name>
  3. <value>yarn</value>
  4. </property>
  5. <property>
  6. <name>mapreduce.job.tracker</name>
  7. <value>localhost:8021</value>
  8. </property>
  9. <property>
  10. <name>mapreduce.local.dir</name>
  11. <value>file:///usr/local/hadoop/data/mapreduce</value>
  12. </property>


2.10添加hadoop可执行路径

/home/hadoopuser/.bashrc添加hadoop可执行路径:

  1. echo"exportPATH=$PATH:/usr/local/hadoop/bin:/usr/local/hadoop/sbin">>/home/hadoopuser/.bashrc
  2. source~/.bashrc


2.11格式化HDFS


  1. [hadoopuser@llshadoop]$hdfsnamenode-format
  2. 14/11/2213:00:18INFOnamenode.NameNode:STARTUP_MSG:
  3. /************************************************************
  4. STARTUP_MSG:StartingNameNode
  5. STARTUP_MSG:host=lls.pc/127.0.0.1
  6. STARTUP_MSG:args=[-format]
  7. STARTUP_MSG:version=2.2.0
  8. STARTUP_MSG:classpath=/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/junit-4.8.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.42.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-httpclient-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/stax-api-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/common/lib/jets3t-0.6.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0-tests.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-2.2.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-el-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/yarn/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-io-2.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/hadoop-annotations-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/hamcrest-core-1.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/junit-4.10.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-site-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-io-2.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.10.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.2.0.jar:/usr/local/hadoop/contrib/capacity-scheduler/*.jar
  9. STARTUP_MSG:build=Unknown-rUnknown;compiledby'root'on2014-11-21T06:32Z
  10. STARTUP_MSG:java=1.7.0_71
  11. ************************************************************/
  12. 14/11/2213:00:18INFOnamenode.NameNode:registeredUNIXsignalhandlersfor[TERM,HUP,INT]
  13. Formattingusingclusterid:CID-bf1d252c-2710-45e6-af26-344debf86840
  14. 14/11/2213:00:19INFOnamenode.HostFileManager:readincludes:
  15. HostSet(
  16. )
  17. 14/11/2213:00:19INFOnamenode.HostFileManager:readexcludes:
  18. HostSet(
  19. )
  20. 14/11/2213:00:19INFOblockmanagement.DatanodeManager:dfs.block.invalidate.limit=1000
  21. 14/11/2213:00:19INFOutil.GSet:ComputingcapacityformapBlocksMap
  22. 14/11/2213:00:19INFOutil.GSet:VMtype=64-bit
  23. 14/11/2213:00:19INFOutil.GSet:2.0%maxmemory=889MB
  24. 14/11/2213:00:19INFOutil.GSet:capacity=2^21=2097152entries
  25. 14/11/2213:00:19INFOblockmanagement.BlockManager:dfs.block.access.token.enable=false
  26. 14/11/2213:00:19INFOblockmanagement.BlockManager:defaultReplication=1
  27. 14/11/2213:00:19INFOblockmanagement.BlockManager:maxReplication=512
  28. 14/11/2213:00:19INFOblockmanagement.BlockManager:minReplication=1
  29. 14/11/2213:00:19INFOblockmanagement.BlockManager:maxReplicationStreams=2
  30. 14/11/2213:00:19INFOblockmanagement.BlockManager:shouldCheckForEnoughRacks=false
  31. 14/11/2213:00:19INFOblockmanagement.BlockManager:replicationRecheckInterval=3000
  32. 14/11/2213:00:19INFOblockmanagement.BlockManager:encryptDataTransfer=false
  33. 14/11/2213:00:19INFOnamenode.FSNamesystem:fsOwner=hadoopuser(auth:SIMPLE)
  34. 14/11/2213:00:19INFOnamenode.FSNamesystem:supergroup=supergroup
  35. 14/11/2213:00:19INFOnamenode.FSNamesystem:isPermissionEnabled=true
  36. 14/11/2213:00:19INFOnamenode.FSNamesystem:HAEnabled:false
  37. 14/11/2213:00:19INFOnamenode.FSNamesystem:AppendEnabled:true
  38. 14/11/2213:00:19INFOutil.GSet:ComputingcapacityformapINodeMap
  39. 14/11/2213:00:19INFOutil.GSet:VMtype=64-bit
  40. 14/11/2213:00:19INFOutil.GSet:1.0%maxmemory=889MB
  41. 14/11/2213:00:19INFOutil.GSet:capacity=2^20=1048576entries
  42. 14/11/2213:00:19INFOnamenode.NameNode:Cachingfilenamesoccuringmorethan10times
  43. 14/11/2213:00:19INFOnamenode.FSNamesystem:dfs.namenode.safemode.threshold-pct=0.9990000128746033
  44. 14/11/2213:00:19INFOnamenode.FSNamesystem:dfs.namenode.safemode.min.datanodes=0
  45. 14/11/2213:00:19INFOnamenode.FSNamesystem:dfs.namenode.safemode.extension=30000
  46. 14/11/2213:00:19INFOnamenode.FSNamesystem:Retrycacheonnamenodeisenabled
  47. 14/11/2213:00:19INFOnamenode.FSNamesystem:Retrycachewilluse0.03oftotalheapandretrycacheentryexpirytimeis600000millis
  48. 14/11/2213:00:19INFOutil.GSet:ComputingcapacityformapNamenodeRetryCache
  49. 14/11/2213:00:19INFOutil.GSet:VMtype=64-bit
  50. 14/11/2213:00:19INFOutil.GSet:0.029999999329447746%maxmemory=889MB
  51. 14/11/2213:00:19INFOutil.GSet:capacity=2^15=32768entries
  52. Re-formatfilesysteminStorageDirectory/usr/local/hadoop/data/namenode?(YorN)y
  53. 14/11/2213:00:39INFOcommon.Storage:Storagedirectory/usr/local/hadoop/data/namenodehasbeensuccessfullyformatted.
  54. 14/11/2213:00:39INFOnamenode.FSImage:Savingimagefile/usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000usingnocompression
  55. 14/11/2213:00:39INFOnamenode.FSImage:Imagefile/usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000ofsize202bytessavedin0seconds.
  56. 14/11/2213:00:39INFOnamenode.NNStorageRetentionManager:Goingtoretain1imageswithtxid>=0
  57. 14/11/2213:00:39INFOutil.ExitUtil:Exitingwithstatus0
  58. 14/11/2213:00:39INFOnamenode.NameNode:SHUTDOWN_MSG:
  59. /************************************************************
  60. SHUTDOWN_MSG:ShuttingdownNameNodeatlls.pc/127.0.0.1
  61. ************************************************************/

2.12启动hadoop

  1. [root@llshadoop]#suhadoopuser
  2. [hadoopuser@llshadoop]$start-dfs.sh&&start-yarn.sh
  3. Startingnamenodeson[localhost]
  4. localhost:startingnamenode,loggingto/usr/local/hadoop/logs/hadoop-hadoopuser-namenode-lls.pc.out
  5. localhost:startingdatanode,loggingto/usr/local/hadoop/logs/hadoop-hadoopuser-datanode-lls.pc.out
  6. Startingsecondarynamenodes[0.0.0.0]
  7. 0.0.0.0:startingsecondarynamenode,loggingto/usr/local/hadoop/logs/hadoop-hadoopuser-secondarynamenode-lls.pc.out
  8. startingyarndaemons
  9. startingresourcemanager,loggingto/usr/local/hadoop/logs/yarn-hadoopuser-resourcemanager-lls.pc.out
  10. localhost:startingnodemanager,loggingto/usr/local/hadoop/logs/yarn-hadoopuser-nodemanager-lls.pc.out
  11. [root@llshadoop]#

2.13查看hadoop状态

查看hadoop守护进程:

  1. [hadoopuser@llsdata]$jps
  2. 13466Jps
  3. 18277ResourceManager
  4. 17952DataNode
  5. 18126SecondaryNameNode
  6. 18394NodeManager
  7. 17817NameNode

最左边的数字表示java进程的PID(启动hadoop时动态分配)DataNode,NameNode,NodeManager,SecondaryNameNode,ResourceManagerhadoop的守护进程。

HDFS内置了许多web服务,用户借助于这些服务,可以通过浏览器来查看HDFS的运行状况。可以通过浏览器查看更详细hadoop状态:

  • Cluster status http://localhost:8088/cluster

  • HDFS statushttp://localhost:50070/dfshealth.jsp

  • SecondaryNameNode statushttp://localhost:50090/status.jsp

2.14测试hadoop

以下是hadoop自带的计算pi的例子:

  1. [hadoopuser@llsdata]$hadoopjar/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jarpi10100
  2. NumberofMaps=10
  3. SamplesperMap=100
  4. WroteinputforMap#0
  5. WroteinputforMap#1
  6. WroteinputforMap#2
  7. WroteinputforMap#3
  8. WroteinputforMap#4
  9. WroteinputforMap#5
  10. WroteinputforMap#6
  11. WroteinputforMap#7
  12. WroteinputforMap#8
  13. WroteinputforMap#9
  14. StartingJob
  15. 14/11/2213:15:23INFOclient.RMProxy:ConnectingtoResourceManagerat/0.0.0.0:8032
  16. 14/11/2213:15:24INFOinput.FileInputFormat:Totalinputpathstoprocess:10
  17. 14/11/2213:15:24INFOmapreduce.JobSubmitter:numberofsplits:10
  18. 14/11/2213:15:24INFOConfiguration.deprecation:user.nameisdeprecated.Instead,usemapreduce.job.user.name
  19. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.jarisdeprecated.Instead,usemapreduce.job.jar
  20. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.map.tasks.speculative.executionisdeprecated.Instead,usemapreduce.map.speculative
  21. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.reduce.tasksisdeprecated.Instead,usemapreduce.job.reduces
  22. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.output.value.classisdeprecated.Instead,usemapreduce.job.output.value.class
  23. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.reduce.tasks.speculative.executionisdeprecated.Instead,usemapreduce.reduce.speculative
  24. 14/11/2213:15:24INFOConfiguration.deprecation:mapreduce.map.classisdeprecated.Instead,usemapreduce.job.map.class
  25. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.job.nameisdeprecated.Instead,usemapreduce.job.name
  26. 14/11/2213:15:24INFOConfiguration.deprecation:mapreduce.reduce.classisdeprecated.Instead,usemapreduce.job.reduce.class
  27. 14/11/2213:15:24INFOConfiguration.deprecation:mapreduce.inputformat.classisdeprecated.Instead,usemapreduce.job.inputformat.class
  28. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.input.dirisdeprecated.Instead,usemapreduce.input.fileinputformat.inputdir
  29. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.output.dirisdeprecated.Instead,usemapreduce.output.fileoutputformat.outputdir
  30. 14/11/2213:15:24INFOConfiguration.deprecation:mapreduce.outputformat.classisdeprecated.Instead,usemapreduce.job.outputformat.class
  31. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.map.tasksisdeprecated.Instead,usemapreduce.job.maps
  32. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.output.key.classisdeprecated.Instead,usemapreduce.job.output.key.class
  33. 14/11/2213:15:24INFOConfiguration.deprecation:mapred.working.dirisdeprecated.Instead,usemapreduce.job.working.dir
  34. 14/11/2213:15:24INFOmapreduce.JobSubmitter:Submittingtokensforjob:job_1416632942167_0003
  35. 14/11/2213:15:24INFOimpl.YarnClientImpl:Submittedapplicationapplication_1416632942167_0003toResourceManagerat/0.0.0.0:8032
  36. 14/11/2213:15:24INFOmapreduce.Job:Theurltotrackthejob:http://localhost:8088/proxy/application_1416632942167_0003/
  37. 14/11/2213:15:24INFOmapreduce.Job:Runningjob:job_1416632942167_0003
  38. 14/11/2213:15:31INFOmapreduce.Job:Jobjob_1416632942167_0003runninginubermode:false
  39. 14/11/2213:15:31INFOmapreduce.Job:map0%reduce0%
  40. 14/11/2213:15:54INFOmapreduce.Job:map10%reduce0%
  41. 14/11/2213:15:55INFOmapreduce.Job:map50%reduce0%
  42. 14/11/2213:15:56INFOmapreduce.Job:map60%reduce0%
  43. 14/11/2213:16:17INFOmapreduce.Job:map90%reduce0%
  44. 14/11/2213:16:18INFOmapreduce.Job:map100%reduce0%
  45. 14/11/2213:16:19INFOmapreduce.Job:map100%reduce100%
  46. 14/11/2213:16:19INFOmapreduce.Job:Jobjob_1416632942167_0003completedsuccessfully
  47. 14/11/2213:16:19INFOmapreduce.Job:Counters:43
  48. FileSystemCounters
  49. FILE:Numberofbytesread=226
  50. FILE:Numberofbyteswritten=879518
  51. FILE:Numberofreadoperations=0
  52. FILE:Numberoflargereadoperations=0
  53. FILE:Numberofwriteoperations=0
  54. HDFS:Numberofbytesread=2700
  55. HDFS:Numberofbyteswritten=215
  56. HDFS:Numberofreadoperations=43
  57. HDFS:Numberoflargereadoperations=0
  58. HDFS:Numberofwriteoperations=3
  59. JobCounters
  60. Launchedmaptasks=10
  61. Launchedreducetasks=1
  62. Data-localmaptasks=10
  63. Totaltimespentbyallmapsinoccupiedslots(ms)=215911
  64. Totaltimespentbyallreducesinoccupiedslots(ms)=20866
  65. Map-ReduceFramework
  66. Mapinputrecords=10
  67. Mapoutputrecords=20
  68. Mapoutputbytes=180
  69. Mapoutputmaterializedbytes=280
  70. Inputsplitbytes=1520
  71. Combineinputrecords=0
  72. Combineoutputrecords=0
  73. Reduceinputgroups=2
  74. Reduceshufflebytes=280
  75. Reduceinputrecords=20
  76. Reduceoutputrecords=0
  77. SpilledRecords=40
  78. ShuffledMaps=10
  79. FailedShuffles=0
  80. MergedMapoutputs=10
  81. GCtimeelapsed(ms)=3216
  82. CPUtimespent(ms)=6420
  83. Physicalmemory(bytes)snapshot=2573750272
  84. Virtualmemory(bytes)snapshot=10637529088
  85. Totalcommittedheapusage(bytes)=2063073280
  86. ShuffleErrors
  87. BAD_ID=0
  88. CONNECTION=0
  89. IO_ERROR=0
  90. WRONG_LENGTH=0
  91. WRONG_MAP=0
  92. WRONG_REDUCE=0
  93. FileInputFormatCounters
  94. BytesRead=1180
  95. FileOutputFormatCounters
  96. BytesWritten=97
  97. JobFinishedin55.969seconds
  98. EstimatedvalueofPiis3.14800000000000000000

出现上述结果,表示整个安装过程大功告成。

3.问题综述

3.1主机名映射错误

  1. STARTUP_MSG:host=java.net.UnknownHostException:lls.pc:lls.pc

解决方案:参考http://www.linuxidc.com/Linux/2012-03/55663.htm

3.2无法加载native-hadoop library

  1. WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourplatform...usingbuiltin-javaclasseswhereapplicable

解决方案:官网上下载的hadoop编译版本的native-hadooplibrary适用于32bit系统,不适用64bit系统,64bit系统需要自己手动编译;编译完成之后用64bitnative-hadooplibrary替代原来的库(64bitnative文件夹替代原来的文件夹),替代完成之后还需要谨慎地配置环境变量,可以参考我的配置或者下面的链接。

参考:http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos

参考:

[1]http://www.ercoppa.org/Linux-Compile-Hadoop-220-fix-Unable-to-load-native-hadoop-library.htm

[2]http://www.ercoppa.org/Linux-Install-Hadoop-220-on-Ubuntu-Linux-1304-Single-Node-Cluster.htm

[3]http://blog.csdn.net/w13770269691/article/details/16883663

[4]http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html

[5]http://tecadmin.net/steps-to-install-hadoop-on-centosrhel-6/

[6]http://alanxelsys.com/2014/02/01/hadoop-2-2-single-node-installation-on-centos-6-5/

[7]http://blog.csdn.net/zwj0403/article/details/16855555

[8]http://www.oschina.net/question/1177468_193584

[9]http://maven.oschina.net/help.html

[10]http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos

[11]https://issues.apache.org/jira/browse/HADOOP-10110

[12]http://www.linuxidc.com/Linux/2012-03/55663.htm