Data Lake Analytics，大数据的ETL神器！

时间 2019-12-11

标签 data lake analytics 数据 etl 神器繁體版

原文原文链接

0. Data Lake Analytics（简称DLA）介绍

数据湖（Data Lake）是时下大数据行业热门的概念：https://en.wikipedia.org/wiki/Data_lake。基于数据湖作分析，能够不用作任何ETL、数据搬迁等前置过程，实现跨各类异构数据源进行大数据关联分析，从而极大的节省成本和提高用户体验。关于Data Lake的概念。html

终于，阿里云如今也有了本身的数据湖分析产品：https://www.aliyun.com/product/datalakeanalytics
能够点击申请使用（目前公测阶段还属于邀测模式），体验本教程分析OTS数据之旅。
产品文档：https://help.aliyun.com/product/70174.htmlmysql

1. ETL介绍

ETL（https://en.wikipedia.org/wiki/Extract,_transform,_load）就是Extract、Transfrom、Load即抽取、转换、加载，是传统数仓和大数据的重要工具。sql

抽取：就是从源系统抽取须要的数据，这些源系统是同构或异构的：好比Excel表格、XML文件、关系型数据库。
转换：源系统的数据按照分析目的，转换成目标系统要求的格式，或者作数据清洗和数据加工。
加载：把转换后的数据装载到目标数据库，做为联机分析、数据挖掘、数据展现的基础。数据库

整个ETL过程就像是在源系统和目标系统之间构建一个管道，数据在这个管道里源源不断的流动。微信

2. DLA与ETL

Data Placement Optimization（数据摆放优化）是目前云平台上的业务系统的主流架构方向和思路。架构师们会从读写性能、稳定性、强一致性、成本、易用性、开发效率等方面来考量不一样存储引擎给业务上带来的好处，从而实现整个业务系统的完美的平衡状态。架构

而这种跨异构数据源之间的数据搬迁，却不是一件容易的事情。不少ELT工具基本上属于框架级别，须要本身开发很多的辅助工具；同时表达能力也较弱，没法知足不少场景；另外对异构数据源的抽象和兼容性也不是那么完美。框架

反观DLA，不管从哪方面来看，DLA都完美的契合ETL的需求场景。下图是DLA的简易架构图，DLA一开始就是基于“MPP计算引擎+存储计算分离+弹性高可用+异构数据集源”等架构原则来设计的，支持各类异构数据源读写是DLA的核心目标！less

经过链接异构数据源来执行select + join + subQuery等逻辑实现Extract，经过Filter+ Project + Aggregation + Sort + Functions等实现数据流转换和映射Transform，而经过insert实现Load，下面是一个例子：异步

--基本格式
insert into target_table (col1, col2, col3, ....)  --须要导入的列以及列的顺序
select c1, c2, c3, ....                            --须要与导入列的类型兼容，顺序要确认清楚
from ...                         --能够是任何你想要查询的数据目标
where ...

--下面是一个例子
insert into target_table (id, name, age)  
select s1.pk1, s2.name, s1.age            
from source_table1 s1
join source_table2 s2
on s1.sid = s2.sid
where s1.xxx = 'yyy'

下面咱们就尝试往不一样的数据源导入数据吧。工具

3. 实际测试（以TableStore:为例）

准备DLA帐号（已有测试帐号）
- 测试集群：上海region；
- 帐号帐号：DLA测试帐号；
准备两个来源表（两个TPC-H的OSS表，customer和nation），用来作join和数据查询；
准备一个TableStore(https://help.aliyun.com/document_detail/27280.html)的目标表；
执行导入SQL，写入数据后校验结果；

a）两个来源表定义：

mysql> show create database tpch_50x_text;
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Database      | Create Database                                                                                                                                                        |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tpch_50x_text | CREATE DATABASE `tpch_50x_text`
WITH DBPROPERTIES (
    catalog = 'hive',
    location = 'oss://${您的bucket}/datasets/tpch/50x/text_date/'
)
COMMENT '' |
+---------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)

mysql> show tables;
+------------+
| Table_Name |
+------------+
| customer   |
| nation     |
+------------+
2 rows in set (0.03 sec)

mysql> show create table customer;
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table    | Create Table                                                                                                                                                                                                                                                                                                                                                                           |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| customer | CREATE EXTERNAL TABLE `tpch_50x_text`.`customer` (
    `c_custkey` int,
    `c_name` string,
    `c_address` string,
    `c_nationkey` int,
    `c_phone` string,
    `c_acctbal` double,
    `c_mktsegment` string,
    `c_comment` string
)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '|'
STORED AS `TEXTFILE`
LOCATION 'oss://${您的bucket}/datasets/tpch/50x/text_date/customer_text' |
+----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.90 sec)

mysql> show create table nation;
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table      | Create Table                                                                                                                                                                                                                                    |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| nation     | CREATE EXTERNAL TABLE `tpch_50x_text`.`nation` (
    `n_nationkey` int,
    `n_name` string,
    `n_regionkey` int,
    `n_comment` string
)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY '|'
STORED AS `TEXTFILE`
LOCATION 'oss://${您的bucket}/datasets/tpch/50x/text_date/nation_text' |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.73 sec)

b）准备TableStore的库和表

## 建库
mysql> show create database etl_ots_test;
+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Database     | Create Database                                                                                                                                                                |
+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| etl_ots_test | CREATE DATABASE `etl_ots_test`
WITH DBPROPERTIES (
    catalog = 'ots',
    location = 'https://${您的instance}.cn-shanghai.ots-internal.aliyuncs.com',
    instance = '${您的instance}'
)
COMMENT '' |
+--------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)

## 使用库
mysql> use etl_ots_test;
Database changed

## 建表
mysql> show create table test_insert;
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table       | Create Table                                                                                                                                                                          |
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test_insert | CREATE EXTERNAL TABLE `test_insert` (
    `id1_int` int NOT NULL COMMENT '客户id主键',
    `c_address` varchar(20) NULL COMMENT '客户的地址',
    `c_acctbal` double NULL COMMENT '客户的account balance',
    PRIMARY KEY (`id1_int`)
)
COMMENT ''             |
+-------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.03 sec)

如下是实际数据的截图：

c）开始导入数据，确保导入字段顺序和类型兼容性：

## 检查数据，都是空的
mysql> select * from etl_ots_test.test_insert;
Empty set (0.31 sec)

mysql> use tpch_50x_text;
Database changed

## 查询下nation数据，其中CANADA的nationkey是3，后续要找这个数据
mysql> select n_nationkey, n_name from nation;
+-------------+----------------+
| n_nationkey | n_name         |
+-------------+----------------+
|           0 | ALGERIA        |
|           1 | ARGENTINA      |
|           2 | BRAZIL         |
|           3 | CANADA         |
|           4 | EGYPT          |
|           5 | ETHIOPIA       |
|           6 | FRANCE         |
|           7 | GERMANY        |
|           8 | INDIA          |
|           9 | INDONESIA      |
|          10 | IRAN           |
|          11 | IRAQ           |
|          12 | JAPAN          |
|          13 | JORDAN         |
|          14 | KENYA          |
|          15 | MOROCCO        |
|          16 | MOZAMBIQUE     |
|          17 | PERU           |
|          18 | CHINA          |
|          19 | ROMANIA        |
|          20 | SAUDI ARABIA   |
|          21 | VIETNAM        |
|          22 | RUSSIA         |
|          23 | UNITED KINGDOM |
|          24 | UNITED STATES  |
+-------------+----------------+
25 rows in set (0.37 sec)

## 查询下customer数据，咱们只关注nationkey=3以及c_mktsegment='BUILDING'的数据
mysql> select count(*) from customer where c_nationkey = 3 and c_mktsegment = 'BUILDING';
+----------+
| count(*) |
+----------+
|    60350 |
+----------+
1 row in set (0.66 sec)

## 查询下customer数据，咱们只关注nationkey=3以及c_mktsegment='BUILDING'的数据
mysql> select * from customer where c_nationkey = 3 and c_mktsegment = 'BUILDING' order by c_custkey limit 3;
+-----------+--------------------+-------------------------+-------------+-----------------+-----------+--------------+----------------------------------------------------------------------------------------------------+
| c_custkey | c_name             | c_address               | c_nationkey | c_phone         | c_acctbal | c_mktsegment | c_comment                                                                                          |
+-----------+--------------------+-------------------------+-------------+-----------------+-----------+--------------+----------------------------------------------------------------------------------------------------+
|        13 | Customer#000000013 | nsXQu0oVjD7PM659uC3SRSp |           3 | 13-761-547-5974 |   3857.34 | BUILDING     | ounts sleep carefully after the close frays. carefully bold notornis use ironic requests. blithely |
|        27 | Customer#000000027 | IS8GIyxpBrLpMT0u7       |           3 | 13-137-193-2709 |   5679.84 | BUILDING     |  about the carefully ironic pinto beans. accoun                                                    |
|        40 | Customer#000000040 | gOnGWAyhSV1ofv          |           3 | 13-652-915-8939 |    1335.3 | BUILDING     | rges impress after the slyly ironic courts. foxes are. blithely                                    |
+-----------+--------------------+-------------------------+-------------+-----------------+-----------+--------------+----------------------------------------------------------------------------------------------------+
3 rows in set (0.78 sec)

导入以前咱们想清楚需求：把国家是'CANADA'的，客户的market segmentation为'BUILDING'的客户找到，而后对c_custkey排序，选择前10条数据，而后选择他们的c_custkey、c_address、c_acctbal三列，清晰到OTS的test_insert表中，以备后续使用。

##先查询下数据，看看有几条数据
mysql> select c.c_custkey, c.c_address, c.c_acctbal 
    -> from tpch_50x_text.customer c
    -> join tpch_50x_text.nation n 
    -> on c.c_nationkey = n.n_nationkey
    -> where n.n_name = 'CANADA' 
    -> and c.c_mktsegment = 'BUILDING' 
    -> order by c.c_custkey
    -> limit 10;
+-----------+--------------------------------+-----------+
| c_custkey | c_address                      | c_acctbal |
+-----------+--------------------------------+-----------+
|        13 | nsXQu0oVjD7PM659uC3SRSp        |   3857.34 |
|        27 | IS8GIyxpBrLpMT0u7              |   5679.84 |
|        40 | gOnGWAyhSV1ofv                 |    1335.3 |
|        64 | MbCeGY20kaKK3oalJD,OT          |   -646.64 |
|       255 | I8Wz9sJBZTnEFG08lhcbfTZq3S     |   3196.07 |
|       430 | s2yfPEGGOqHfgkVSs5Rs6 qh,SuVmR |   7905.17 |
|       726 | 4w7DOLtN9Hy,xzZMR              |   6253.81 |
|       905 | f iyVEgCU2lZZPCebx5bGp5        |   -600.73 |
|      1312 | f5zgMB4MHLMSHaX0tDduHAmVd4     |    9459.5 |
|      1358 | t23gsl4TdVXqTZha DioEHIq5w7y   |   5149.23 |
+-----------+--------------------------------+-----------+
10 rows in set (1.09 sec)

##开始导入
mysql> insert into etl_ots_test.test_insert (id1_int ,c_address, c_acctbal)
    -> select c.c_custkey, c.c_address, c.c_acctbal 
    -> from tpch_50x_text.customer c
    -> join tpch_50x_text.nation n 
    -> on c.c_nationkey = n.n_nationkey
    -> where n.n_name = 'CANADA' 
    -> and c.c_mktsegment = 'BUILDING' 
    -> order by c.c_custkey
    -> limit 10;
+------+
| rows |
+------+
|   10 |
+------+
1 row in set (2.14 sec)

## 验证结果，没有问题：
mysql> select * from etl_ots_test.test_insert;
+---------+--------------------------------+-----------+
| id1_int | c_address                      | c_acctbal |
+---------+--------------------------------+-----------+
|      13 | nsXQu0oVjD7PM659uC3SRSp        |   3857.34 |
|      27 | IS8GIyxpBrLpMT0u7              |   5679.84 |
|      40 | gOnGWAyhSV1ofv                 |    1335.3 |
|      64 | MbCeGY20kaKK3oalJD,OT          |   -646.64 |
|     255 | I8Wz9sJBZTnEFG08lhcbfTZq3S     |   3196.07 |
|     430 | s2yfPEGGOqHfgkVSs5Rs6 qh,SuVmR |   7905.17 |
|     726 | 4w7DOLtN9Hy,xzZMR              |   6253.81 |
|     905 | f iyVEgCU2lZZPCebx5bGp5        |   -600.73 |
|    1312 | f5zgMB4MHLMSHaX0tDduHAmVd4     |    9459.5 |
|    1358 | t23gsl4TdVXqTZha DioEHIq5w7y   |   5149.23 |
+---------+--------------------------------+-----------+
10 rows in set (0.27 sec)

d）注意点：

虽然有ETL工具快速导入导出，但也有些问题须要注意的，好比：

若是导入任务时间太长，请走异步模式，不然链接断开可能会影响任务正常运行；
TableStore目前的insert是根据主键覆盖，主键不会去重判断的，请务必不能对你正常的数据表作插入；
目前DLA和TableStore的事务能力还不够，可能会出现中断，已导入的数据不会清楚，须要自行清理；
列的个数和列的类型，须要本身对齐保障，不然会报错；

4. 其余数据源导入

整个过程是否是很简单？是否是想要导入其余场景的数据源？对DLA而言，底层任何数据源都以相同方式处理，只要确保其余数据源的库、表在DLA中正常建立，就能够正常的读写，实现ETL啦！赶忙试试吧！

其余相关的文档：

使用Data Lake Analytics从OSS清洗数据到AnalyticDB：https://yq.aliyun.com/articles/623401
DLA相关技术文档：https://yq.aliyun.com/search?q=DLA&type=ARTICLE
Data Lake Analytics使用场景：https://help.aliyun.com/document_detail/70380.html
OLAP on TableStore——基于Data Lake Analytics的Serverless SQL大数据分析https://yq.aliyun.com/articles/618501
使用Data Lake Analytics 分析OSS数据：https://help.aliyun.com/document_detail/70387.html
Data Lake Analytics数据库的链接方式：https://help.aliyun.com/document_detail/71074.htm
Data Lake Analytics分析RDS数据：https://yq.aliyun.com/articles/629046

原文连接更多技术干货请关注阿里云云栖社区微信号：yunqiinsight