hadoop2.7.7 +hbase2.0.6 + apache-phoenix-5.0.0-HBase-2.0 安装使用
1、安装环境准备
  • 1.1）、准备机器说明
主机名(机器) ip 说明
master 192.168.2.10 主节点（安装hadoop、hbase、phoenix、zookeeper）
node1 192.168.2.20 节点1（安装hadoop节点）
node2 192.168.2.30 节点2（安装hadoop节点）

三台机器均关闭防火墙并关闭ipv6

  • 1.2）、三台机器均匀配置host
192.168.2.10 master
192.168.2.20 node1
192.168.2.30 node2
  • 1.3）、master机器设置免登录node1、node2（在master上执行以下命令）
ssh-keygen -t rsa (一路默认回车即可)
ssh-copy-id hadoop-master
ssh-copy-id hadoop-node1
ssh-copy-id hadoop-node2
  • 1.4）、master机器增加以下环境变量配置/etc/profile（第二步开始逐一安装）
export JAVA_HOME=/software/jdk1.8.0_151
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

export HADOOP_HOME=/software/hadoop-2.7.7
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

export HBASE_HOME=/software/hbase-2.0.6
export PATH=$HBASE_HOME/bin:$PATH

export PHOENIX_HOME=/software/apache-phoenix-5.0.0-HBase-2.0-bin
export PHOENIX_CLASSPATH=$PHOENIX_HOME
export PATH=$PATH:$PHOENIX_HOME/bin

在master机器上安装jdk、hadoop、hbase、phoenix，安装包的解压目录均为/software

  • 1.5）、node1、node2机器增加以下环境变量配置/etc/profile
export JAVA_HOME=/software/jdk1.8.0_151
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

export HADOOP_HOME=/software/hadoop-2.7.7
export PATH=${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

在两个node节点上安装jdk、hadoop，安装包的解压目录均为/software

2、安装hadoop2.7.7

解压安装包hadoop-2.7.7.tar.gz到目录/software，与高版本的不同，此版本需修改/software/hadoop-2.7.7/etc/hadoop目录下的配置文件core-site.xml、hdfs-site.xml、hadoop-env.sh、slaves、yarn-env.sh共五个文件并同步到node1、node2一，两个节点

首先将安装包同步到节点上并解压

scp jdk-8u151-linux-x64.tar.gz  hadoop-2.7.7.tar.gz  root@node1:/software
scp jdk-8u151-linux-x64.tar.gz  hadoop-2.7.7.tar.gz  root@node2:/software
  • 2.1）、core-site.xml配置文件新增以下配置
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://master:8020</value>
        </property>

        <property>
                <name>dfs.namenode.name.dir</name>
                <value>/software/hadoop-2.7.7/tmp</value>
        </property>
</configuration>
  • 2.2）、hadoop-env.sh配置文件新增jdk路径配置
export JAVA_HOME=/software/jdk1.8.0_151
  • 2.3）、hdfs-site.xml配置文件新增以下配置
<configuration>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>node1:50090</value>
        </property>
        <property>
                <name>dfs.replication</name>
                <value>2</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/var/data/hadoop/hdfs/name</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/var/data/hadoop/hdfs/data</value>
        </property>
        <property>
                <name>dfs.namenode.http-address</name>
                <value>master:8084</value>
        </property>
</configuration>
  • 2.4）、slaves配置文件新增节点配置
master
node1
node2
  • 2.5）、yarn-env.sh配置文件新增jdk路径配置
export JAVA_HOME=/software/jdk1.8.0_151
  • 2.6）、同步配置文件到node1、node2
scp -r core-site.xml hadoop-env.sh hdfs-site.xml slaves yarn-env.sh  root@node1:/software/hadoop-2.7.7/etc/hadoop
scp -r core-site.xml hadoop-env.sh hdfs-site.xml slaves yarn-env.sh  root@node2:/software/hadoop-2.7.7/etc/hadoop
  • 2.7）、在master上执行格式化命令完成格式化
hdfs namenode -format
  • 2.8）、启动hadoop集群
启动
start-all.sh

jps可查看相关进程情况
停止
stop-all.sh

重启nodemanager
yarn  --daemon start nodemanager

重启datanode
hdfs --daemon start datanode

master上查看节点状态
yarn node -list

master查看hdfs状态
hdfs dfsadmin -report

查看hdfs占用空间
hadoop fs -du -h /

启动完成后，可访问地址http://192.168.2.10:8084/dfshealth.html#tab-overview查看详情

3、master安装zookeeper

zookeeper简单安装单机版的作为演示，解压安装包apache-zookeeper-3.6.1-bin.tar.gz到目录/software，拷贝配置文件启动即可

[root@master conf]# cd /software/apache-zookeeper-3.6.1-bin/conf
[root@master conf]# cp zoo_sample.cfg zoo.cfg 
[root@master conf]# cd ../bin/
[root@master bin]# ./zkServer.sh start
4、master安装hbase2.0.6

下载安装包hbase-2.0.6-bin.tar.gz解压到目录/software，并配置/software/hbase-2.0.6/conf中的配置文件hbase-env.sh、hbase-site.xml

  • 4.1）、hbase-env.sh配置jdk环境变量并配置zk不使用内部zk（使用外部zookeeper）
export JAVA_HOME=/software/jdk1.8.0_151
export HBASE_MANAGES_ZK=false
  • 4.2）、hbase-site.xml配置文件新增以下配置
<configuration>
 <property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
  </property>
  <property>
    <name>hbase.tmp.dir</name>
    <value>./tmp</value>
  </property>
  <property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
  </property>

  <property>
    <name>hbase.rootdir</name>
    <value>hdfs://master:8020/hbase</value>
  </property>

  <property>
    <name>hbase.master.maxclockskew</name>
    <value>150000</value>
  </property>
  <property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/data/zookeeper</value>
  </property>

  <property>
    <name>hbase.zookeeper.quorum</name>
    <value>master:2181</value>
  </property>
</configuration>
  • 4.3）、启动hbase
启动
start-hbase.sh

jps可查看hbase相关进程
[root@master bin]# jps
3600 HMaster
2258 QuorumPeerMain
3724 HRegionServer


关闭hbase
stop-hbase.sh

若stop-hbase.sh关闭不了直接使用以下两句
hbase-daemon.sh stop master
hbase-daemon.sh stop regionserver
5、master安装phoenix5.0.0

解压安装包apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz到目录/software，环境变量第一步已配置，将两个jar包拷贝到hbase的lib目录中，命令如下

cd /software/apache-phoenix-5.0.0-HBase-2.0-bin
cp phoenix-5.0.0-HBase-2.0-server.jar phoenix-core-5.0.0-HBase-2.0.jar /software/hbase-2.0.6/lib/

拷贝jar包到hbase后需重启hbase

  • 5.1）、修改phoenix执行脚本权限
cd /software/apache-phoenix-5.0.0-HBase-2.0-bin/bin
chmod 777 psql.py sqlline.py
  • 5.2）、master上使用phoenix操作hbase

[root@master ~]# sqlline.py master:2181

Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:master:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:master:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/software/apache-phoenix-5.0.0-HBase-2.0-bin/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/software/hadoop-2.7.7/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
20/10/18 22:18:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 5.0)
Driver: PhoenixEmbeddedDriver (version 5.0)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:master:2181> 
0: jdbc:phoenix:master:2181> !tables
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+-+
| TABLE_CAT  | TABLE_SCHEM  | TABLE_NAME  |  TABLE_TYPE   | REMARKS  | TYPE_NAME  | SELF_REFERENCING_COL_NAME  | REF_GENERATION  | |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+-+
|            | SYSTEM       | CATALOG     | SYSTEM TABLE  |          |            |                            |                 | |
|            | SYSTEM       | FUNCTION    | SYSTEM TABLE  |          |            |                            |                 | |
|            | SYSTEM       | LOG         | SYSTEM TABLE  |          |            |                            |                 | |
|            | SYSTEM       | SEQUENCE    | SYSTEM TABLE  |          |            |                            |                 | |
|            | SYSTEM       | STATS       | SYSTEM TABLE  |          |            |                            |                 | |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+-+
0: jdbc:phoenix:master:2181> create table test (mykey integer not null primary key, mycolumn varchar);
No rows affected (0.871 seconds)
0: jdbc:phoenix:master:2181> select * from test;
+--------+-----------+
| MYKEY  | MYCOLUMN  |
+--------+-----------+
+--------+-----------+
No rows selected (0.081 seconds)
0: jdbc:phoenix:master:2181> upsert into test values (1,'Hello');
1 row affected (0.045 seconds)
0: jdbc:phoenix:master:2181> upsert into test values (2,'World!');
1 row affected (0.009 seconds)
0: jdbc:phoenix:master:2181> select * from test;
+--------+-----------+
| MYKEY  | MYCOLUMN  |
+--------+-----------+
| 1      | Hello     |
| 2      | World!    |
+--------+-----------+
2 rows selected (0.025 seconds)
0: jdbc:phoenix:master:2181> !exit
Closing: org.apache.phoenix.jdbc.PhoenixConnection
[root@master ~]#
赞赏(Donation)
微信(Wechat Pay)

donation-wechatpay

相关博客