百度网盘下载地址（平台搭建全套安装包）：

链接：https://pan.baidu.com/s/11HyrxVKevgvJc-rI6bFLjA
提取码：sybk

基础环境准备

本文不做赘述，检查虚拟机网络是否可用，防火墙是否关闭，节点之间是否可以进行免密通信，JDK是否安装且可用

本文并不适合初学者，更多是记录和回顾，为方便复制粘贴快速部署hadoop完全分布式环境

设置静态IP

ping www.baidu.com				（测试网络连通性）

关闭防火墙

查看防火墙状态
systemctl status firewalld

暂时关闭防火墙
systemctl stop firewalld

关闭防火墙开机自启动
systemctl disable firewalld

防火墙开机自启动
systemctl enable firewalld

集群节点之间免密登录

ssh-keygen -t rsa

ssh-copy-id master 
ssh-copy-id slave1 
ssh-copy-id slave2

ssh master 
ssh slave1 
ssh slave2 

JDK1.8安装与环境变量配置

vim /etc/profile

export JAVA_HOME=/opt/module/jdk
export PATH=$PATH:$JAVA_HOME/bin

java -version 
javac 

解压Hadoop

tar -xzvf /opt/module/software/hadoop-3.1.3.tar.gz -C /opt/module/
mv /opt/module/hadoop-3.1.3 hadoop 

配置Hadoop配置

切换到/opt/module/hadoop/etc/hadoop目录下

cd /opt/module/hadoop/etc/hadoop

1. vim core-site.xml

 <configuration>
 
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/module/hadoop/tmp</value>
    </property>

    <property>	
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>

    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
    </property>

    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>

</configuration>

2. vim hdfs-site.xml

<configuration>

    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>master:50090</value>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/opt/module/hadoop/tmp/dfs/name</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/opt/module/hadoop/tmp/dfs/data</value>
    </property>
</configuration>

3. vim mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>

    <property>
       <name>yarn.app.mapreduce.am.env</name>
       <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
     </property>

     <property>
       <name>mapreduce.map.env</name>
       <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
     </property>

     <property>
       <name>mapreduce.reduce.env</name>
       <value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
     </property>

</configuration>

4. vim yarn-site.xml

<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>
</property>
</configuration>

5. hadoop-env.sh

export JAVA_HOME=/opt/module/jdk
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

6. yarn-env.sh

export JAVA_HOME=/opt/module/jdk
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

7. vim workers

master
slave1
slave2

配置环境变量

vim /etc/profile

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

使环境变量生效

source /etc/profile

分发安装包

scp -r /opt/module/hadoop slave1:/opt/module/
scp -r /opt/module/hadoop slave2:/opt/module/

格式化hdfs

hdfs namenode -format

启动集群

start-all.sh

使用jps查看java进程会看到 如下进程

在master上可看到

NameNode
SecondaryNameNode
ResourceManager
NodeManager
DataNode

在slave1上可看到

NodeManager
DataNode

在slave2上可看到

NodeManager
DataNode

Hadoop NameNode的Web UI默认端口为9870