CentOS7部署HDP3.1.0.0-白红宇

CentOS7部署HDP3.1.0.0

阅读量：4621 次

发布时间：2019-06-09

本文共 6976 字，大约阅读时间需要 23 分钟。

Apache Ambari是一个基于Web的支持Apache Hadoop集群的供应、管理和监控的开源工具，Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。提供Web UI进行可视化的集群管理，简化了大数据平台的安装、使用难度。

主机配置规划

主机名	IP	配置	操作系统	角色
c1.heboan.com	10.1.1.121	2核/8G	CentOS Linux release 7.2.1511	Ambari server, mariadb
c2.heboan.com	10.1.1.120	2核/8G	CentOS Linux release 7.2.1511	agent
c3.heboan.com	10.1.1.120	2核/8G	CentOS Linux release 7.2.1511	agent

准备好软件包放到/root/rools/目录下

设置主机名

绑定主机名与ip的关系

10.1.1.120 c1.heboan.com10.1.1.121 c2.heboan.com10.1.1.122 c3.heboan.com

/etc/hosts

配置文件打开数

#vim /etc/security/limits.conf//末尾加上* soft nofile 32768* hard nofile 1048576* soft nproc 65536* hard nproc unlimited* hard memlock ulimited* soft memlock unlimited

/etc/security/limits.conf

关闭防火墙和selinux

ntp服务配置

设置swap

echo vm.swappiness = 10 >> /etc/sysctl.confsysctl -p

/etc/sysctl.conf

设置透明大页面

# echo never > /sys/kernel/mm/transparent_hugepage/defrag# echo never > /sys/kernel/mm/transparent_hugepage/enabled将如下脚本添加到/etc/rc.d/rc.local文件中if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi

设置透明大页面

安装jdk

tar xf jdk-8u131-linux-x64.tar.gz -C /usr/local/cd /usr/local/;ln -s jdk1.8.0_131 jdkvim /etc/profile.d/jdk.sh    export JAVA_HOME=/usr/local/jdk    export JAVA_BIN=/usr/local/jdk/bin    export PATH=$PATH:$JAVA_HOME/bin    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar    export JAVA_HOME JAVA_BIN PATH CLASSPATHsource /etc/profile.d/jdk.sh

/etc/profile.d/jdk.sh

安装jdbc驱动

mkdir /usr/share/javamv mysql-connector-java-5.1.22.jar /usr/share/java/mysql-connector-java.jarchmod +x /usr/share/java/mysql-connector-java.jar

配置ssh免密互信

ssh-keygenssh-copy-id -i ~/.ssh/id_rsa.pub c1.heboan.comssh-copy-id -i ~/.ssh/id_rsa.pub c2.heboan.comssh-copy-id -i ~/.ssh/id_rsa.pub c3.heboan.comcd .ssh/scp id_rsa id_rsa.pub c2.heboan.com:/root/.ssh/scp id_rsa id_rsa.pub c3.heboan.com:/root/.ssh/

ssh配置

搭建本地yum源（c1.heboan.com）

安装httpd服务

yum install -y httpdsystemctl start httpd

配置仓库

yum install -y createrepotar xf ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/tar xf HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/tar xf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/tar xf HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/cd /var/www/html/ambari/;createrepo .cd /var/www/html/HDP;createrepo .cd /var/www/html/HDP-UTILS/;createrepo .cd /var/www/html/HDP-GPL;createrepo .#/etc/yum.repos.d/am.repo[amrepo]name = am_repobaseurl = http://c1.heboan.com/ambarienable = truegpgcheck = false#/etc/yum.repos.d/hdp.repo[hdprepo]name = hdp_repobaseurl = http://c1.heboan.com/HDPenable = truegpgcheck = false#/etc/yum.repos.d/hdp-utils.repo[hdp-utils-repo]name = hdp-utils_repobaseurl = http://c1.heboan.com/HDP-UTILSenable = truegpgcheck = false#/etc/yum.repos.d/hdp-gpl.repo[hdp-gpl-repo]name = hdp-gpl_repobaseurl = http://c1.heboan.com/HDP-GPLenable = truegpgcheck = falseyum repolist#然后也am.repo, hdp.repo, hdp-utils.repo,hdp-gpl.repo 传到其他机器节点

yum配置

配置数据库MariaDB(c1.heboan.com)

安装数据库

yum install -y mariadb mariadb-server mariadb-devel

修改字符集

# vim /etc/my.cnf.d/server.cnf...[mysqld]lower_case_table_names=1character-set-server = utf8collation-server = utf8_general_ci

启动数据库并且初始化配置

systemctl start mariadbsystemctl enable mariadbmysql_secure_installation

创建库并账号授权

> create database ambari;> grant all privileges on *.* to  'bigdata'@'%' identified by 'Bigdata@123';> flush privileges;

部署ambari（c1.heoan.com）

安装ambari

yum -y install ambari-server

导入数据库

> use ambari> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

配置ambari

[root@c1 ~]# ambari-server setupUsing python  /usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is 'disabled'Customize user account for ambari-server daemon [y/n] (n)? Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK...Do you want to change Oracle JDK [y/n] (n)? y[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8[2] Custom JDK==============================================================================Enter choice (1): 2WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.Path to JAVA_HOME: /usr/local/jdk  Validating JDK on Ambari Server...done.Check JDK version for Ambari Server...JDK version found: 8Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.Checking GPL software agreement...Completing setup...Configuring database...Enter advanced database configuration [y/n] (n)? yConfiguring database...==============================================================================Choose one of the following options:[1] - PostgreSQL (Embedded)[2] - Oracle[3] - MySQL / MariaDB[4] - PostgreSQL[5] - Microsoft SQL Server (Tech Preview)[6] - SQL Anywhere[7] - BDB==============================================================================Enter choice (3): 3Hostname (localhost): Port (3306): Database name (ambari): Username (bigdata): bigdataEnter Database Password (123456789): Configuring ambari database...Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? yConfiguring remote database connection properties...WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [y/n] (y)? yExtracting system views...ambari-admin-2.7.3.0.139.jar....Ambari repo file doesn't contain latest json url, skipping repoinfos modificationAdjusting ambari-server permissions and ownership...Ambari Server 'setup' completed successfully.

配置

启动ambari-server

[root@c1 ~]# ambari-server startUsing python  /usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at /var/lib/ambari-server/resources...Ambari database consistency check started...Server PID at: /var/run/ambari-server/ambari-server.pidServer out at: /var/log/ambari-server/ambari-server.outServer log at: /var/log/ambari-server/ambari-server.logWaiting for server start............................................Server started listening on 8080DB configs consistency check: no errors and warnings were found.Ambari Server 'start' completed successfully.

浏览器访问http://10.1.1.120:8080, 账号密码：admin/admin

启动向导创建集群

配置集群名字

配置本地仓库

配置主机和私钥

配置主机

根据实际需求选择安装的组件

服务节点分布

选择安装slaves and clients

配置密码

配置hive 元数据存储数据库，选择新的mysql

[root@c1 ~]# ambari-server setup --jdbc-db=mysql  --jdbc-driver=/usr/share/java/mysql-connector-java.jar