一.安装docker1.配置镜像仓库(1)首先把原来的自带镜像移走或删除,我一般习惯删除[rootlocalhost /]# cd /etc/yum.repos.d/[rootlocalhost yum.repos.d]# lltotal40-rw-r--r--.1root root1664Oct232020CentOS-Base.repo -rw-r--r--.1root root1309Oct232020CentOS-CR.repo -rw-r--r--.1root root649Oct232020CentOS-Debuginfo.repo -rw-r--r--.1root root630Oct232020CentOS-Media.repo -rw-r--r--.1root root1331Oct232020CentOS-Sources.repo -rw-r--r--.1root root8515Oct232020CentOS-Vault.repo -rw-r--r--.1root root314Oct232020CentOS-fasttrack.repo -rw-r--r--.1root root616Oct232020CentOS-x86_64-kernel.repo[rootlocalhost yum.repos.d]# rm -rf *(2)关闭虚拟机的防火墙和selinux[rootlocalhost ~]# systemctl stop firewalld ##临时关闭[rootlocalhost ~]# systemctl disable firewalld --now ##永久关闭[rootlocalhost ~]# setenforce 0[rootlocalhost ~]# vim /etc/selinux/config# This file controls the state of SELinux on the system.# SELINUX can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - No SELinux policy is loaded.SELINUXdisabled# SELINUXTYPE can take one of three values:# targeted - Targeted processes are protected,# minimum - Modification of targeted policy. Only selected processes are protected.# mls - Multi Level Security protection.SELINUXTYPEtargeted(3)然后从阿里云拉取镜像curl-o/etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repowget-O/etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo(4)安装docker# step 1: 安装必要的一些系统工具sudoyuminstall-yyum-utils# Step 2: 添加软件源信息yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo# Step 3: 安装Dockersudoyuminstalldocker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin#Step 4: 启Docker服务 并设置开机自启sudosystemctl startdockersudosystemctlenabledocker--now(5)配置docker镜像加速器vim/etc/docker/daemon.json{dns:[8.8.8.8,8.8.4.4],registry-mirrors:[https://docker.rainbond.cc,https://docker.mirrors.sjtug.sjtu.edu.cn,https://docker.m.daocloud.io,https://docker.itelyou.cf,https://noohub.ru,https://docker.fxxk.dedyn.io,https://huecker.io,https://dockerhub.timeweb.cloud,https://registry.cn-hangzhou.aliyuncs.com,https://dockerproxy.com,https://docker.mirrors.ustc.edu.cn,https://docker.nju.edu.cn,https://xx4bwyg2.mirror.aliyuncs.com,https://f1361db2.m.daocloud.io,https://registry.docker-cn.com,https://hub-mirror.c.163.com],insecure-registries:[0.0.0.0/0]}(6)查看docker是否安装成功[rootdocker ~]# docker versionClient: Docker Engine - Community Version:26.1.4 API version:1.45Go version: go1.21.11 Git commit: 5650f9b Built: Wed Jun511:32:042024OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version:26.1.4 API version:1.45(minimum version1.24)Go version: go1.21.11 Git commit: de5c9cf Built: Wed Jun511:31:022024OS/Arch: linux/amd64 Experimental:falsecontainerd: Version:1.6.33 GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957 runc: Version:1.1.12 GitCommit: v1.1.12-0-g51d5e94 docker-init: Version:0.19.0 GitCommit: de40ad0二.搭建hadoop一个服务器起3容器,镜像用centos,这3个容器分别作为master,slave1,slave2节点1.先拉取centos镜像dockerpull centos:centos7.9.20092.编写docekrfile这里也可以不用写dockerfile直接起一个centons的基础镜像,但还要手动那些服务和下载手动拉取镜像仓库,我嫌麻烦,而且假如中途报错,还得从新做,所以我选择做一个镜像里面做好基础的配置FROM centos:centos7.9.2009 LABELwx17835414167RUNrm-rf/etc/yum.repos.d/* RUNcurl-o/etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo RUN yuminstall-ywgetRUNwget-O/etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo RUN yum clean allyuminstall-yvimyuminstall-ybash-*yuminstall-yopenssh-clients openssh-server RUN yuminstall-yiproutebashCMD[/bin/bash]3构建镜像dockerbuild-fDockerfile-tcentos:7.4 启动容器dockerrun-d\--privileged#--privileged 参数用于赋予容器特权模式,使容器拥有几乎和宿主机相同的权限 \-v/sys/fs/cgroup:/sys/fs/cgroup:ro#挂载cgroup机制 这样就能运行systemd \--namemaster\centos:7\/usr/sbin/init#初始化进程主要启动systemd服务我试过好多次,这个容器非常容易报权限的错误,加参数 -u root 也会报错,所以启动时一定要加–privileged 参数,要不然改主机名哪里就会报错,必须得把cgroup机制挂载进容器,不然sshd服务起不来,影响各个容器之间的通信像这样启动3个分别作为master,slave1,slave2,节点,后面我就直接说节点了[rootlocalhost ~]# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b4bb4ec638b4 centos:7/usr/sbin/init17hours ago Up17hours slave2 255b7c39ae99 centos:7/usr/sbin/init17hours ago Up17hours master aacb52fed314 centos:7/usr/sbin/init17hours ago Up17hours slave1[rootlocalhost ~]# docker exec -it -u root master /bin/bash ##进入容器,依此类推进入创建好的3的容器[root9bb49dececaf /]#5容器的基本配置(1)修改各个节点的主机名[root255b7c39ae99 ~]# vim /etc/hostnamemaster[root255b7c39ae99 ~]# bash###或者用###[root255b7c39ae99 ~]# hostnamectl set-hostname master(2)各个节点配置主机映射[rootmaster hadoop]# vim /etc/hosts172.17.0.3 master172.17.0.2 slave1172.17.0.4 slave2(3)给各个节点改密码[rootmaster /]# passwdChanging passwordforuser root. New password: BAD PASSWORD: The password is a palindrome Retype new password: passwd: all authentication tokens updated successfully.(4)给各个节点设置免密[rootmaster /]# ssh-keygen #一路回车Generating public/private rsa key pair. Enterfileinwhichto save the key(/root/.ssh/id_rsa): Created directory/root/.ssh.Enter passphrase(emptyforno passphrase): Enter same passphrase again: Your identification has been savedin/root/.ssh/id_rsa. Your public key has been savedin/root/.ssh/id_rsa.pub. The key fingerprint is: SHA256:xFT7Yo13H3mdXDsxbzGeNo8rpiVJVk8Z0wogPH5TRg rootb6065bf0fb3e The keys randomart image is: ---[RSA2048]----|oo.E.||o..o.o..||o .o.B.||.o* ^||S oo/||.oo.o*||..o. o||.o.||o..|----[SHA256]-----(5)将本地公钥复制到远程主机各个节点都执行[rootmaster /]# ssh-copy-id master/usr/bin/ssh-copy-id: INFO: Source of key(s)to be installed:/root/.ssh/id_rsa.pubThe authenticity ofhostmaster (172.17.0.3)cant be established. ECDSA key fingerprint is SHA256:zq/pAor6qOMIufHDozEd9mcF50JvG1ta/yBnoNdUJ4. ECDSA key fingerprint is MD5:0d:aa:e9:6a:cf:b8:87:63:f6:29:d0:f7:0d:dd:c0:6f. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys rootmasters password: Number of key(s)added:1Now try logging into the machine, with:ssh masterand check tomakesure that only the key(s)you wanted were added.[rootmaster /]# ssh-copy-id slave1/usr/bin/ssh-copy-id: INFO: Source of key(s)to be installed:/root/.ssh/id_rsa.pubThe authenticity ofhostslave1 (172.17.0.2)cant be established. ECDSA key fingerprint is SHA256:oRM14eX0t8/gBuEACQ6sqtfCwpi3fzIvdYGPFXl4bzM. ECDSA key fingerprint is MD5:43:51:89:72:0b:94:da:27:7a:77:e1:25:13:02:ff:5d. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys rootslave1s password: Number of key(s)added:1Now try logging into the machine, with:ssh slave1and check tomakesure that only the key(s)you wanted were added.[rootmaster /]# ssh-copy-id slave2/usr/bin/ssh-copy-id: INFO: Source of key(s)to be installed:/root/.ssh/id_rsa.pubThe authenticity ofhostslave2 (172.17.0.4)cant be established. ECDSA key fingerprint is SHA256:LShFFy1lFa1GDw8xixw4q9bck928ea89hnOzwYoY88o. ECDSA key fingerprint is MD5:94:0c:d3:68:79:2c:93:0b:10:f1:0c:90:a1:96:0a:5b. Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys rootslave2s password: Number of key(s)added:1Now try logging into the machine, with:ssh slave2and check tomakesure that only the key(s)you wanted were added.6jdk环境变量的配置(1)将包导入master节点[rootmaster ~]# docker cp jdk-8u371-linux-x64.tar.gz master:/root(2)解压jdk到/opt/module/路径下[rootmaster ~]# tar -zxvf jdk-8u371-linux-x64.tar.gz -C /opt/module/(3)配置环境变量[rootmaster ~]# vim /etc/profile #在这个配置文件添加下面3行JAVA_HOME/opt/module/jdk1.8.0_371PATH$PATH:$JAVA_HOME/binexportJAVA_HOMEPATH(4)使环境变量生效[rootmaster ~]# source /etc/profile(5)验证是否成功[rootmaster ~]# java -versionjavaversion1.8.0_371Java(TM)SE Runtime Environment(build1.8.0_371-b11)Java HotSpot(TM)64-Bit Server VM(build25.371-b11, mixed mode)(6)分发到其他节点,slave1和slave2[rootmaster ~]# scp -r /opt/module/jdk1.8.0_371/ slave1:/opt/module/[rootmaster ~]# scp -r /etc/profile slave1:/etc/profile在那两个节点执行source /etc/profile发现都配置好了jdk7hadoop环境变量的配置(1)导入节点[rootmaster~]# docker cp hadoop-3.2.1.tar.gz master:/root(2)解压[rootmaster ~]# tar -zxvf jdk-8u371-linux-x64.tar.gz -C /opt/module/[rootmaster ~]# vim /etc/profile #在这个配置文件添加下面几行exportHADOOP_HOME/opt/module/hadoop-3.2.1exportPATH$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATHexportHDFS_NAMENODE_USERrootexportHDFS_SECONDARYNAMENODE_USERrootexportYARN_RESOURCEMANAGER_USERrootexportYARN_NODEMANAGER_USERrootexportHDFS_JOURNALNODE_USERrootexportHDFS_ZKFC_USERroot(3)使环境变量生效[rootmaster ~]# source /etc/profile(4)验证是否成功[rootmaster ~]# hadoop versionHadoop3.2.1 Source code repository https://gitbox.apache.org/repos/asf/hadoop.git-rb3cbbb467e22ea829b3808f4b7b01d07e0bf3842 Compiled by rohithsharmaks on2019-09-10T15:56Z Compiled with protoc2.5.0 Fromsourcewith checksum 776eaf9eee9c0ffc370bcbc1888737 Thiscommandwas run using /opt/module/hadoop-3.2.1/share/hadoop/common/hadoop-common-3.2.1.jar(5)分发到其他节点,slave1和slave2[rootmaster ~]# scp -r /opt/module/hadoop-3.2.1/ slave1:/opt/module/[rootmaster ~]# scp -r /etc/profile slave1:/etc/profile在那两个节点执行source /etc/profile发现都配置好了hadoop8配置Hadoop相关的环境变量[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/core-site.xmlconfigurationpropertynamefs.defaultFS/namevaluehdfs://master:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/opt/module/hadoop-3.2.1/tmp/value/property/configuration[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/hdfs-site.xmlconfigurationpropertynamedfs.replication/namevalue3/value/propertypropertynamedfs.namenode.name.dir/namevalue/opt/module/hadoop-3.2.1/namenode/value/propertypropertynamedfs.datanode.data.dir/namevalue/opt/module/hadoop-3.2.1/datanode/value/property/configuration[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/mapred-site.xmlconfigurationpropertynamemapreduce.framework.name/namevalueyarn/value/property/configuration[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/yarn-site.xmlconfigurationpropertynameyarn.resourcemanager.hostname/namevaluemaster/value/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/property/configuration[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/workersmaster slave1 slave2[rootmaster hadoop]# vim /opt/module/hadoop-3.2.1/etc/hadoop/hadoop-env.shexportJAVA_HOME/opt/module/jdk1.8.0_371分发到slave1,slave2节点[rootmaster hadoop]# scp -r /opt/module/hadoop-3.2.1 slave1:/opt/module/[rootmaster hadoop]# scp -r /opt/module/hadoop-3.2.1 slave2:/opt/module/9.启动hadoop集群(1)初始化namenode[rootmaster hadoop]# hdfs namenode -format2025-10-1617:49:58,893 INFO namenode.FSDirectory: POSIX ACL inheritance enabled?true2025-10-1617:49:58,893 INFO namenode.FSDirectory: XAttrs enabled?true2025-10-1617:49:58,894 INFO namenode.NameNode: Cachingfilenames occurringmorethan10times2025-10-1617:49:58,898 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit:655362025-10-1617:49:58,900 INFO snapshot.SnapshotManager: SkipList is disabled2025-10-1617:49:58,904 INFO util.GSet: Computing capacityformap cachedBlocks2025-10-1617:49:58,905 INFO util.GSet: VMtype64-bit2025-10-1617:49:58,905 INFO util.GSet:0.25% max memory839.5MB2.1MB2025-10-1617:49:58,905 INFO util.GSet: capacity2^18262144entries2025-10-1617:49:58,914 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets102025-10-1617:49:58,914 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users102025-10-1617:49:58,914 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes1,5,252025-10-1617:49:58,918 INFO namenode.FSNamesystem: Retry cache on namenode is enabled2025-10-1617:49:58,920 INFO namenode.FSNamesystem: Retry cache will use0.03of total heap and retry cache entry expirytimeis600000millis2025-10-1617:49:58,922 INFO util.GSet: Computing capacityformap NameNodeRetryCache2025-10-1617:49:58,922 INFO util.GSet: VMtype64-bit2025-10-1617:49:58,923 INFO util.GSet:0.029999999329447746% max memory839.5MB257.9KB2025-10-1617:49:58,923 INFO util.GSet: capacity2^1532768entries2025-10-1617:49:58,950 INFO namenode.FSImage: Allocated new BlockPoolId: BP-354296932-172.17.0.3-17606369989422025-10-1617:49:58,964 INFO common.Storage: Storage directory /opt/module/hadoop-3.2.1/namenode has been successfully formatted.2025-10-1617:49:59,004 INFO namenode.FSImageFormatProtobuf: Saving imagefile/opt/module/hadoop-3.2.1/namenode/current/fsimage.ckpt_0000000000000000000 using no compression2025-10-1617:49:59,120 INFO namenode.FSImageFormatProtobuf: Imagefile/opt/module/hadoop-3.2.1/namenode/current/fsimage.ckpt_0000000000000000000 of size399bytes savedin0seconds.2025-10-1617:49:59,134 INFO namenode.NNStorageRetentionManager: Going to retain1images with txid02025-10-1617:49:59,138 INFO namenode.FSImage: FSImageSaver clean checkpoint:txid0when meet shutdown.2025-10-1617:49:59,138 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master/172.17.0.3 ************************************************************/(2)启动hdfs和yarn[rootmaster hadoop]# start-dfs.sh[rootmaster hadoop]# start-yarn.sh(3)查看是否搭建成功[rootmaster hadoop]# jps2753DataNode2615NameNode2956SecondaryNameNode3245ResourceManager7085Jps3391NodeManager出现这个几个进程就说明成功搭建