[AI] 深度数据 - Data

Data Engineering

---

Data Pipeline

Introduction

[DE] How to learn Big Data【了解大数据】

[DE] Pipeline for Data Engineering【工做流案例示范】

[DE] ML on Big data: MLlib【大数据的机器学习方案】

 

DE基础（厦大）

 

[Spark] 00 - Install Hadoop & Spark【ing】

• • [Hadoop] HDFS
  • [Hadoop] MapReduce
  • [Hadoop] Yarn & k8s
    

[Spark] 01 - What is Spark【RDD原理和方法】

[Spark] 02 - Practice PySpark【实践编程】

• • [PySpark] Build R&D environment
  • [PySpark] RDD programming on a large file【notebook demo】
  • /* to do, 优化？ */

[Spark] 03 - Spark SQL【具备了SQL操做的便捷性】

• • [PySpark] Spark SQL on a large file【notebook demo】
  • [Hadoop] HBase【分布式稀疏大表】

[Spark] 04 - What is Spark Streaming

[Spark] 05 - Apache Kafka

[Spark] 06 - Structured Streaming【对应 DataFrame】

 

AWS基础

[Full-stack] 一切皆在云上 - AWS【AWS基础服务】

[AWS] 01 - What is Amazon EMR【EMR简介】

[AWS] 02 - Pipeline on EMR【基础了解】

 

/* important */

 

 

 

Data Science

---

Local Data Processing

"矩阵"计算

[Code] 大蛇之数据工程【语法驱动】

[Code] 变态之人键合一【需求驱动】

[Pandas] 01 - A guy based on NumPy【如何高性能】

[Pandas] 02 - Tutorial of NumPy【NumPy常见用法】

 

"表格"处理

[Pandas] 03 - DataFrame【读入并处理表格】

[Pandas] 04 - Efficient I/O【从数据库加载到arr, df, EArray】

• • [MySQL] 01- Basic sql
  • [MySQL] 02- Optimisation solutions

 

"特征"工程

[Feature] Preprocessing tutorial【伟哥的特征工程步骤讲解】

[Feature] Feature engineering【特征工程大纲】

• • [Scikit-learn] 4.3 Preprocessing data【概念夯实】
  • [Feature] Compare the effect of different scalers【去量纲】
  • [Feature] Feature selection
  • [Feature] Feature selection - Embedded topic
  • [Converge] Feature Selection in training of Deep Learning【感性理解】

[Feature] Build pipeline【展现Pipeline大概思路过程】

[Feature] Final pipeline: custom transformers【本章总结】

 

"机器"学习

[AI] 深度数学 - Bayes【Scikit-learn Cookbook】

[Distributed ML] Yi WANG's talk【王益大佬】

 

数据"可视化"

[Matplotlib] Data Representation

[Tableau] Tableau for BI

 

Kaggle经验谈

[Kaggle] Online Notebooks【模块化代码】

[Kaggle] How to kaggle?【方法导论】 

[Kaggle] How to handle big data?【方法进阶】

 

 

 

Cloud Data Processing

Introduction

[ML] Pyspark ML tutorial for beginners【房价预测之"常规分析套路"】

 

ML-Features

[ML] Load and preview large scale data【保证特征完整性】

[Link] https://spark.apache.org/docs/2.4.4/ml-guide.html

• • [ML] Feature Transformers
  • [ML] Feature Selectors

[ML] Pipeline in Distributed ML Library【Pipline"套路”】  

[ML] Online learning【Pipline做为 “在线学习” 的 “数据源”】  

 

GPU ML

[GPU] Install H2O.ai

[GPU] Machine Learning on C++

[Spark] Spark 3.0 Accelerator Aware Scheduling - GPU

 

Distributed ML

[ML] LIBSVM Data: Classification, Regression, and Multi-label【三种方案时效对比】

[ML] Machine Learning in the Common Infrastructure ecosystem【架构了解】

 

 

 

Big Data Algorithms

本篇章终极形态，开发/优化一个大数据分布式算法。

https://github.com/apache/spark/tree/master/examples/src/main/python/ml

 

https://spark.apache.org/mllib/

http://stanford.edu/~rezab/

http://stanford.edu/~rezab/slides/

Distributed Computing with Spark, Reza Zadeh 20140623

Reza Zadeh, Scalable Machine Learning

Apache Spark™ ML and Distributed Learning (1/5) （databrick）

Module 4: Creating Distributed Algorithms

 

stanford.edu: Chapter 12 Large-Scale Machine Learning

<Large Scale Machine Learning with Python>

 

Processing Big Data in Main Memory and on GPU，2016年硕士论文

[Spark News] Spark + GPU are the next generation technology

 

Spark大数据互联网项目实战推荐系统（全套）

Spark项目实战：爱奇艺用户行为实时分析系统

/* implement */