[Feature] Final pipeline: custom transformers

有视频：https://www.youtube.com/watch?v=BFaadIqWlAg

有代码：https://github.com/jem1031/pandas-pipelines-custom-transformers

 

 

幼儿级模型

---

1、模型训练

简单的preprocessing后，仅使用一个“属性”作预测，看看结果如何？

#%%
import pandas as pd
import numpy as np
import os

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline

# SET UP

# Read in data
# source: https://data.seattle.gov/Permitting/Special-Events-Permits/dm95-f8w5
data_folder = '../data/'
data_file = 'Special_Events_Permits_2016.csv'
data_file_path = os.path.join(data_folder, data_file)
print("debug: data_file_path is {}".format(data_file_path))
df = pd.read_csv(data_file_path)

# Set aside 25% as test data
df_train, df_test = train_test_split(df, random_state=4321)

# Take a look
df_train.head()

#%%
# SIMPLE MODEL

# Binarize string feature
y_train = np.where(df_train.permit_status == 'Complete', 1, 0)
y_test  = np.where(df_test.permit_status == 'Complete', 1, 0)

print(y_train[:5])
print(y_test[:5])

# Missing value，且只使用这一列作出此次模型训练的特征！
X_train_1 = df_train[['attendance']].fillna(value=0)
X_test_1  = df_test[['attendance']].fillna(value=0)

print(X_train_1[:5])
print(X_test_1[:5])

#%%
# Fit model
model_1 = LogisticRegression(random_state=5678)
model_1.fit(X_train_1, y_train)

 

2、模型评估

评估指标 ROC AUC

(1) 得到二值化的分类结果； 

(2) 得到分类的几率数值。

y_pred_train_1 = model_1.predict(X_train_1)
print("y_pred_train_1 is {}".format(y_pred_train_1))
p_pred_train_1 = model_1.predict_proba(X_train_1)[:, 1]
print("p_pred_train_1 is {}".format(p_pred_train_1))

# Evaluate model
# baseline: always predict the average
p_baseline_test = [y_train.mean()]*len(y_test)
auc_baseline = roc_auc_score(y_test, p_baseline_test)
print(auc_baseline)  # 0.5

#######################################################

y_pred_test_1 = model_1.predict(X_test_1)
print("y_pred_test_1 is {}".format(y_pred_test_1))
p_pred_test_1 = model_1.predict_proba(X_test_1)[:, 1]
print("p_pred_test_1 is {}".format(p_pred_test_1))

# Evaluate model
auc_test_1 = roc_auc_score(y_test, p_pred_test_1)
print(auc_test_1)  # 0.576553672316

 

Ref: 机器学习评价指标 ROC与AUC 的理解和python实现

以FPR为横坐标，TPR为纵坐标，那么ROC曲线就是改变各类阈值后获得的全部坐标点 (FPR,TPR) 的连线，画出来以下。

红线是随机乱猜状况下的 ROC，曲线越靠左上角，分类器越佳。

AUC（Area Under Curve）就是ROC曲线下的面积。

既然已经这么多评价标准，为何还要使用ROC和AUC呢？

由于ROC曲线有个很好的特性：当测试集中的正负样本的分布变化的时候，ROC曲线可以保持不变。

 

评估指标 R2

决定系数R2 Score ，衡量模型预测能力好坏（真实和预测的 相关程度百分比）

预测数据和真实数据越接近，R2越大，固然最大值是 1；模型的R2 值为0，还不如直接用平均值(均值模型)来预测效果好。

 

Ref: 【从零开始学机器学习12】MSE、RMSE、R2_score

既然不一样数据集的量纲不一样，很难经过上面的三种方式去比较，那么不妨找一个第三者做为参照，根据参照计算 R方值，就能够比较模型的好坏了。

R2_score < 0 ：分子大于分母，训练模型产生的偏差比使用均值产生的还要大，也就是训练模型反而不如直接去均值效果好。出现这种状况，一般是模型自己不是线性关系的，而咱们误使用了线性模型，致使偏差很大。

评估指标 Residual

方差越大，模型越不稳定； 

import numpy as np
from sklearn.datasets import load_boston
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, ConstantKernel as CK
from sklearn.model_selection import cross_val_predict

boston = load_boston()
boston_X = boston.data
boston_y = boston.target
train_set = np.random.choice([True, False], len(boston_y),p=[.75, .25])
# 这里得到布尔index，方便从数据集中pick up所需数据

mixed_kernel = kernel = CK(1.0, (1e-4, 1e4)) * RBF(10, (1e-4, 1e4))
gpr = GaussianProcessRegressor(alpha=5, n_restarts_optimizer=20, kernel = mixed_kernel) 
gpr.fit(boston_X[train_set], boston_y[train_set])
test_preds = gpr.predict(boston_X[~train_set]

View Code

from matplotlib import pyplot as plt
f, ax = plt.subplots(figsize=(10, 7), nrows=3)
f.tight_layout()


ax[0].plot(range(len(test_preds)), test_preds,           label='Predicted Values')
ax[0].plot(range(len(test_preds)), boston_y[~train_set], label='Actual Values')
ax[0].set_title("Predicted vs Actuals")
# ax[0].legend(loc='best')

# 参差图 residual
residual = test_preds - boston_y[~train_set]

ax[1].plot(range(len(test_preds)), residual)
ax[1].set_title("Plotted Residuals")

ax[2].hist(residual)
ax[2].set_title("Histogram of Residuals")

Result:

 

 

 

 

 

模型改进

---

初探数据

1、数据清理时，须要考虑的内容

Ref: [Pandas] 03 - DataFrame

• • 看看某列，瞧瞧某行【第一步】
  • 可视化一列数据【第一步】
  • 分组统计【第三步】
  • 重采样【第三步】

 

Ref: [Feature] Preprocessing tutorial

• • 特征统计分布【第一步】
  • 空数据【第二步】
  • 特征间线性关系【第一步】

 

2、空数据太多怎么办？

能够考虑放弃这个特征。

park_cts = df_train.event_location_park.value_counts(dropna=False)
print(park_cts)
# NaN                                    364
# Magnuson Park                            8
# Gas Works Park                           5
# Occidental Park                          3
# Greenlake Park                           2
# Volunteer Park                           2
# Seattle Center                           1
# Seward Park                              1
# Anchor Park                              1
# Madison Park                             1
# OTHER                                    1
# Myrtle Edwards Park                      1
# Martin Luther King Jr Memorial Park      1
# Hamilton Viewpoint Park                  1
# Ballard Commons Park                     1
# Lake Union Park                          1
# Judkins Park                             1
# Bell Street Park                         1
# Comments:
# - about 90% missing values
# - could be new values in test data
# - Note: there are 400+ parks in Seattle

View Code

 

3、数据太多且分散怎么办？

相似高频特征，可分组归类，resampling。

org_cts = df_train.organization.value_counts(dropna=False)

Red Carpet Valet                                             44
Seattle Sounders FC                                          19
Butler Valet                                                 15
Seafair                                                       9
Fuel Sports Eats and Beats                                    6
CBS Seattle                                                   5
Pro-Motion Events, Inc.                                       5
Madison Park Business Association                             4
Rejuvenation                                                  4
Fremont Arts Council                                          4
The U District Partnership                                    4
Seattle Department of Transportation                          4
University of Washington Rowing                               4
Upper Left                                                    3
Seattle Symphony                                              3
Argosy Cruises                                                3
The Corson Building                                           3
Waterways Cruises                                             3
Run for Good Racing Co./5 Focus                               3
Seattle Symphony/Benaroya Hall                                3
West Seattle Junction Association                             3
University of Washington Husky Marching Band                  3
Pro-Motion Events, Inc                                        2
Northwest Yacht Brokers Association                           2
Seattle Yacht Club                                            2
Café Campagne                                                 2
HONK! Fest West                                               2
Umoja Fest                                                    2
Ethiopians in Seattle                                         2
Emerald City Pet Rescue                                       2
                                                             ..
Fizz Events, LLC                                              1
Wing Luke Museum of the Asian Pacific American Experience     1
Independent Event Solutions                                   1
Vulcan Inc.                                                   1
City of Seattle/Animal Shelter                                1
GO LONG SR520 Floating Bridge Run                             1
The Queen AnneCamber of Commerce                              1
Greenwood Knights                                             1
Alki Art Fair                                                 1
Fizz Events LLC                                               1
Sea Deli, Inc                                                 1
Rotary Foundation of West Seattle                             1
Seattle Buddhist Church                                       1
TUNE                                                          1
AMERICAN CANCER SOCIETY, INC.                                 1
CWD Group, Inc.                                               1
Beacon Arts                                                   1
Southwest Seattle Historical Society                          1
Northwest Museum of Legends and Lore                          1
magnolia chamber of commerce                                  1
Ram Racing                                                    1
Seattle Events A Non-Profit Corporation                       1
Sound Transit                                                 1
Piranha Blonde Interactive                                    1
City of Seattle Parks and Recreation Department               1
El Centro de La Raza                                          1
Northwest Hope and Healing Foundation                         1
Orswell Events                                                1
Lifelong                                                      1
NaN                                                           1
Name: organization, Length: 245, dtype: int64

Result

 

4、极端值outlier太多怎么办？

”泰尔森估算“是其中的一个策略，但这属于ML estimator的选择范畴。

具体参见：[AI] 深度数学 - Bayes

 

 

清理数据

1、特征名字统一格式

# Switch column names to lower_case_with_underscores
def standardize_name(cname):
    cname = re.sub(r'[-\.]', ' ', cname)
    cname = cname.strip().lower()
    cname = re.sub(r'\s+', '_', cname)
    return cname

print(df_raw.columns)
df_raw.columns = df_raw.columns.map(standardize_name)
print(df_raw.columns)

Index(['Application Date', 'Permit Status', 'Permit Type', 'Event Category',
       'Event Sub-Category', 'Name of Event', 'Year-Month-App.',
       'Event Start Date', 'Event End Date', 'Event Location - Park',
       'Event Location - Neighborhood', 'Council District', 'Precinct',
       'Organization', 'Attendance'],
      dtype='object')
Index(['application_date', 'permit_status', 'permit_type', 'event_category',
       'event_sub_category', 'name_of_event', 'year_month_app',
       'event_start_date', 'event_end_date', 'event_location_park',
       'event_location_neighborhood', 'council_district', 'precinct',
       'organization', 'attendance'],
      dtype='object')

Result

 

2、分割数据

按照时间分割，比较常见的方式。

# Filter to 2016 events
df_raw['event_start_date1'] = pd.to_datetime(df_raw.event_start_date)

df = df_raw[np.logical_and(df_raw.event_start_date1 >= '2016-01-01',
                           df_raw.event_start_date1 <= '2016-12-31')]
df = df.drop('event_start_date1', axis=1)


# Export data
data_file = 'Special_Events_Permits_2016.csv'
df.to_csv(data_folder + data_file, index=False)

 

 

特征选择

能够本身添加一些随机特征做为noise，做为特征选择的上手练习。

• • [Feature] Feature selection
  • [Feature] Feature selection - Embedded topic

 

 

工做流模型

1、FeatureUnion 组织 Transform

>>> from sklearn.pipeline import FeatureUnion
>>> feature_union = FeatureUnion([
... ('fill_avg',  Imputer(strategy='mean')),
... ('fill_mid',  Imputer(strategy='median')),
... ('fill_freq', Imputer(strategy='most_frequent'))
... ])

>>> X_train = feature_union.fit_transform(X_train_raw)
>>> X_test  = feature_union.transform(X_test_raw)

 

2、构建自定义 Transform

一个表格中有不少特征，"定性特征" 和 "定量特征" 能够按照以下的思路分开且并行的解决。

# Preprocessing with a Pipeline
pipeline = Pipeline([

    ('features', DFFeatureUnion([
        ('categoricals', Pipeline([
            ('extract', ColumnExtractor(CAT_FEATS)),
            ('dummy',   DummyTransformer())
        ])),
        ('numerics', Pipeline([
            ('extract',   ColumnExtractor(NUM_FEATS)),
            ('zero_fill', ZeroFillTransformer()),
            ('log',       Log1pTransformer())
        ]))
    ])),
    ('scale', DFStandardScaler())
])

固定的套路是：继承TransformerMixin后，实现 fit 和 tranform 方法。

class DummyTransformer(TransformerMixin):

    def __init__(self):
        self.dv = None

    def fit(self, X, y=None):
        # assumes all columns of X are strings
        Xdict = X.to_dict('records')
        self.dv = DictVectorizer(sparse=False)
        self.dv.fit(Xdict)
        return self

    def transform(self, X):
        # assumes X is a DataFrame
        Xdict = X.to_dict('records')
        Xt   = self.dv.transform(Xdict)
        cols = self.dv.get_feature_names()
        Xdum = pd.DataFrame(Xt, index=X.index, columns=cols)
        # drop column indicating NaNs
        nan_cols = [c for c in cols if '=' not in c]
        Xdum = Xdum.drop(nan_cols, axis=1)
        return Xdum

 

知识点

处理 "定性特征" 的套路。

Ref: pandas.DataFrame.to_dict()的使用详解

Ref: 特征提高之特征抽取----DictVectorizer

 

3、特征联合 Feature Union

由于默认是用numpy做为参数格式，但这里都是dataframe结构，稍微自定义下便可。

class DFFeatureUnion(TransformerMixin):
    # FeatureUnion but for pandas DataFrames

    def __init__(self, transformer_list):
        self.transformer_list = transformer_list

    def fit(self, X, y=None):
        # 执行完，却不须要结果 for (_, t) in self.transformer_list:
            t.fit(X, y)
        return self

    def transform(self, X):
        # 执行完，须要结果；由于结果还要被用来作reduce操做
        Xts = [t.transform(X) for _, t in self.transformer_list]
        Xunion = reduce(lambda X1, X2: pd.merge(X1, X2, left_index=True, right_index=True), Xts)
        return Xunion

 

4、训练模型并测试

可见，测试结果好了一些。

pipeline.fit(df_train)
X_train_2 = pipeline.transform(df_train)
X_test_2  = pipeline.transform(df_test)

# Fit model
model_2 = LogisticRegression(random_state=5678)
model_2.fit(X_train_2, y_train)
y_pred_train_2 = model_2.predict(X_train_2)
p_pred_train_2 = model_2.predict_proba(X_train_2)[:, 1]

# Evaluate model
p_pred_test_2 = model_2.predict_proba(X_test_2)[:, 1]
auc_test_2 = roc_auc_score(y_test, p_pred_test_2)
print(auc_test_2)  # 0.70508474576

 

 

过拟合

更多的特征致使过拟合，以下，反而性能降低了。

# Preprocessing with a Pipeline
pipeline3 = Pipeline([
    ('features', DFFeatureUnion([
        ('dates', Pipeline([
            ('extract',  ColumnExtractor(DATE_FEATS)),　　# 考虑日期相关特征　　
            ('to_date',  DateFormatter()),
            ('diffs',    DateDiffer()),
            ('mid_fill', DFImputer(strategy='median'))
        ])),
        ('categoricals', Pipeline([
            ('extract',  ColumnExtractor(CAT_FEATS)),
            ('dummy',    DummyTransformer())
        ])),
        ('multi_labels', Pipeline([
            ('extract',     ColumnExtractor(MULTI_FEATS)),
            ('multi_dummy', MultiEncoder(sep=';'))
        ])),
        ('numerics', Pipeline([
            ('extract',   ColumnExtractor(NUM_FEATS)),
            ('zero_fill', ZeroFillTransformer()),
            ('log',       Log1pTransformer())
        ]))
    ])),
    ('scale', DFStandardScaler())
])

pipeline3.fit(df_train)
X_train_3 = pipeline3.transform(df_train)
X_test_3  = pipeline3.transform(df_test)

# Fit model
model_3 = LogisticRegression(random_state=5678)
model_3.fit(X_train_3, y_train)
y_pred_train_3 = model_3.predict(X_train_3)
p_pred_train_3 = model_3.predict_proba(X_train_3)[:, 1]

# Evaluate model
p_pred_test_3 = model_3.predict_proba(X_test_3)[:, 1]
auc_test_3 = roc_auc_score(y_test, p_pred_test_3)
print(auc_test_3)  # 0.680790960452  # too many features -> starting to overfit

 

End.