elephant-bird学习笔记

时间 2019-11-06

原文原文链接

elephant-bird是Twitter的开源项目，项目的地址为 https://github.com/twitter/elephant-birdgit

该项目是Twitter为LZO,thrift,protocol buffer相关的hadoop InputFormats, OutputFormats, Writables, Pig加载函数, Hive SerDe, HBase二级索引等编写的库github

mvn clean install -U -Dprotobuf.version=2.5.0 -DskipTests=true

mvn package的时候须要签名apache

gpg --gen-key

以及须要安装apache Thrift和Protocol Buffersbash

使用elephant-bird来建hive表的类型对应关系函数

CREATE EXTERNAL TABLE `xxxx`(
	  `ts` string COMMENT 'from deserializer', 
	  `schema` string COMMENT 'from deserializer', 
	  `test_string` string COMMENT 'from deserializer', 
	  `test_long` bigint COMMENT 'from deserializer', 
	  `test_int` int COMMENT 'from deserializer', 
	  `test_short` smallint COMMENT 'from deserializer', 
	  `test_double` double COMMENT 'from deserializer', 
	  `test_byte` tinyint COMMENT 'from deserializer', 
	  `test_bool` boolean COMMENT 'from deserializer', 
	  `test_list` array<string> COMMENT 'from deserializer', 
	  `test_set` array<bigint> COMMENT 'from deserializer', 
	  `test_map` map<string,int> COMMENT 'from deserializer')
	COMMENT 'test_all_type'
	PARTITIONED BY ( 
	  `ds` string COMMENT '日期分区')
	ROW FORMAT SERDE 
	  'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' 
	WITH SERDEPROPERTIES ( 
	  'serialization.class'='com.xxx.xxx.xxx', 
	  'serialization.format'='org.apache.thrift.protocol.TCompactProtocol') 
	STORED AS INPUTFORMAT 
	  'org.apache.hadoop.mapred.SequenceFileInputFormat' 
	OUTPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat'
	LOCATION
	  'hdfs://xxxxxxx'
	TBLPROPERTIES (