此主题介绍Sqoop的安装、配置及基础使用。java
为某企业作的培训,完整文档见:http://gudaoxuri.github.io/bd-lab/ html
官网:http://sqoop.apache.org/ 官方文档:http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html
Sqoop有两大版本,Sqoop稳定,Sqoop2目前问题比较多,如下使用Sqoop |
wget http://mirror.bit.edu.cn/apache/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz rm -rf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha ./frameworks/sqoop
export SQOOP_HOME=/opt/workspaces/frameworks/sqoop
source ~/.profile
export HADOOP_COMMON_HOME=/opt/workspaces/frameworks/hadoop export HADOOP_MAPRED_HOME=/opt/workspaces/frameworks/hadoop export HIVE_HOME=/opt/workspaces/frameworks/hive
wget -P ./frameworks/sqoop/lib http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.36/mysql-connector-java-5.1.36.jar
./frameworks/sqoop/bin/sqoop import --connect jdbc:mysql://<host>:<port>/hive \ --username hive --password hive \ --table ROLES \ --where 1=1 \ --hive-import --hive-table hive_role # 如何要启用增量导入须要加上以下参数 --incremental lastmodified --check-column <source field> --last-value ''
增量的字段必须是timestamp 或date/datetime |