Hadoop+HBase平台上安装snappy

在测试Apache Phoenix过程当中，遇到了许多问题，首当其冲的就是缺乏snappy压缩包。没办法，hbase-0.98.6.1没有自带，就只能本身安装了。首先固然仍是上网搜一搜，看有没有写的比较好的安装步骤。结果发现，千篇一概，都是要下载编译一个貌似是hadoop-snappy的包。仔细看了看过程，还挺吓人的，中间出现了各类错误，甚至还对gcc的版本有要求。一气之下，就不想装了（实际上是比较懒了嘿嘿）。

而后某一天忽然看到了这篇博客：How to install snappy with HBase 0.94.x，做者说他摸索了3天才找出来这个不走弯路的办法，且仅需5分钟就能搞定(方法很简单，但我仍是花了大概5个小时的时间，在Ubuntu上安装东西，你懂的。。。)

文章中介绍了一种安装在HBase上安装snappy的方法，很简单，因而决定一试，现将这篇文中讲到的方法翻译过来与你们分享：

一、检查snappy压缩包是否安装

命令为：bin/hbase org.apache.hadoop.hbase.util.CompressionTest file:///tmp/test.txt snappy

若是显示信息为：

12/12/03 10:30:02 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:30:02 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
     at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
     at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
     at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
     at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
     at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
     at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
     at org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)

 则说明snappy压缩包没有安装；

二、下载snappy-*.tar.gz压缩包(只要和hbase版本兼容就能够，个人是snappy-1.1.1.tar.gz)，解压；

三、进入snappy目录，进行编译，两条命令：

      ./configure

       make

四、make完以后会产生一个libsnappy.so文件(这就是咱们所须要的库！！！)，正常状况下出如今当前目录./libs/libsnappy.so，可是不少时候不按套路出牌，跑到别的文件夹下了，若是make没有出错，能够在根目录search一下，确定能找到这个文件；

五、将生成的这个libsnappy.so拷贝到HBase的lib/native/Linux-ARCH目录下，ARCH表明 amd64 或 i386-32，注意，对于amd64的HBase可能没有这个目录，此时，须要手动建立：

     mkdir /opt/hbase-0.98.6.1/lib/native/Linux-amd64-64

六、若是仍是不肯定HBase在哪里查找lib，那么能够修改log4j文件中的日志级别(log level)进行调试；

七、从新运行第1步中的命令，如今看到的信息应该为：

12/12/03 10:34:35 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
12/12/03 10:34:35 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32C not available. 
12/12/03 10:34:35 DEBUG util.FSUtils: Creating file:file:/tmp/test.txtwith permission:rwxrwxrwx
12/12/03 10:34:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
12/12/03 10:34:35 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:34:35 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 10:34:35 WARN snappy.LoadSnappy: Snappy native library not loaded
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
    at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:123)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
    at org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:264)
    at org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:739)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:127)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:118)
    at org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:101)
    at org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:394)
    at org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:108)
    at org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:138)

八、能够看到，snappy已经能够找到了，可是尚未加载(not loaded)。想加载的话，还须要拷贝hadoop的本地库到与libsnappy.so同一个路径下，hadoop的本地库路径为：

      hadoop-1.2.1/lib/native/Linux-ARCH/libhadoop.so；

       若是这个路径下没有，能够根据所使用的hadoop版本到 https://archive.apache.org/dist/hadoop/core/ 下载相应的tar.gz包，解压以后就能找到所须要的文件了；

九、再次运行测试命令(第1步中的命令)，能够获得：

12/12/03 10:37:48 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32 not available.
12/12/03 10:37:48 INFO util.ChecksumType: Checksum can use java.util.zip.CRC32
12/12/03 10:37:48 INFO util.ChecksumType: org.apache.hadoop.util.PureJavaCrc32C not available. 
12/12/03 10:37:48 DEBUG util.FSUtils: Creating file:file:/tmp/test.txtwith permission:rwxrwxrwx
12/12/03 10:37:48 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/12/03 10:37:48 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:37:48 WARN snappy.LoadSnappy: Snappy native library is available
12/12/03 10:37:48 INFO snappy.LoadSnappy: Snappy native library loaded
12/12/03 10:37:48 INFO compress.CodecPool: Got brand-new compressor
12/12/03 10:37:48 DEBUG hfile.HFileWriterV2: Initialized with CacheConfig:disabled
12/12/03 10:37:49 WARN metrics.SchemaConfigured: Could not determine table and column family of the HFile path file:/tmp/test.txt. Expecting at least 5 path components.
12/12/03 10:37:49 INFO compress.CodecPool: Got brand-new decompressor
SUCCESS

      看到SUCCESS，说明安装成功，snappy压缩包可使用，搞定。

ps：建议你们能够去看下这篇文章

网址：http://www.spaggiari.org/index.php/hbase/how-to-install-snappy-with#.VREJ__Sl8tM

 

注意：若是在make过程当中出错，在从新make时必定要把以前的snappy文件夹删除，从新解压出一个新的文件夹，切记