数据挖掘入门——以二手车价格预测为例
做者：张杰html
数据挖掘的步骤
- Data Analysis
- Feature Engineering
- Feature Selection
- Model Building
- Model Deployment
1. Data Analysis
对于数据分析部分，须要探索如下几个点： 1)Missing Valuespython
2)All The Numerical Variables算法
3)Distribution of the Numerical Variables网络
4)Categorical Variables架构
5)Cardinality of Categorical Variablesapp
6)Outliersdom
Relationship between independent and dependent feature(SalePrice)机器学习
2. Feature Engineering
数据和特征决定了机器学习的上限，而模型和算法只是逼近这个上限而已。那特征工程究竟是什么呢？顾名思义，其本质是一项工程活动，目的是最大限度地从原始数据中提取特征以供算法和模型使用。函数
特征工程中可能有如下问题：工具
- 不属于同一量纲：即特征的规格不同，不可以放在一块儿比较；
- 定性特征不能直接使用：某些机器学习算法和模型只能接受定量特征的输入，那么须要将定性特征转换为定量特征。最简单的方式是为每一种定性值指定一个定量值，可是这种方式过于灵活，增长了调参的工做。一般使用哑编码的方式将定性特征转换为定量特征：假设有N种特征，当原始特征值为第i种定性值时，第i个扩展特征为1，其余扩展特征赋值为0。哑编码的方式相比直接指定的方式，不用增长调参的工做，对于线性模型来讲，使用哑编码的特征可达到非线性的效果；
- 存在缺失值：缺失值须要补充；
- 信息利用率低：不一样的机器学习算法和模型对数据中信息的利用是不一样的，以前提到在线性模型中，使用对定性特征哑编码能够达到非线性的效果。相似地，对定量变量多项式化，或者进行其余的转换，都能达到非线性的效果。
特别地，须要重点关注缺失值的处理，异常值的处理，数据归一化，数据编码等关键问题。
3. Feature Selection
特征选择意味着选择那些在咱们的模型上能够提升性能的特征变量。可使用了一些机器学习和统计方法来选择最有相关的特征来改进模型性能。
4. Model Building
模型部分一般能够选择机器学习模型，也能够选择使用深度学习模型，特别地，不少时候模型集成每每有着出人意料的效果。
赛题分析
赛题数据
赛题以预测二手车的交易价格为任务，该数据来自某交易平台的二手车交易记录，总数据量超过40w，包含31列变量信息，其中15列为匿名变量。为了保证比赛的公平性，将会从中抽取15万条做为训练集，5万条做为测试集，同时会对name、model、brand和regionCode等信息进行脱敏。 数据连接：[https://tianchi.aliyun.com/competition/entrance/231784/introduction]
评测标准
评价标准为MAE(Mean Absolute Error)。
导入基本模块
# 基础工具
import numpy as np
import pandas as pd
import warnings
import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.special import jn
from IPython.display import display, clear_output
import time
from tqdm import tqdm
import itertools
warnings.filterwarnings('ignore')
%matplotlib inline
## 模型预测的
from sklearn import linear_model
from sklearn import preprocessing
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor,GradientBoostingRegressor
## 数据降维处理的
from sklearn.decomposition import PCA,FastICA,FactorAnalysis,SparsePCA
## 参数搜索和评价的
from sklearn.model_selection import GridSearchCV,cross_val_score,StratifiedKFold,train_test_split
from sklearn.metrics import mean_squared_error, mean_absolute_error
import scipy.signal as signal
数据分析与特征工程
def reduce_mem_usage(df):
""" iterate through all the columns of a dataframe and modify the data type
to reduce memory usage.
"""
start_mem = df.memory_usage().sum()
print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))
for col in df.columns:
col_type = df[col].dtype
if col_type != object:
c_min = df[col].min()
c_max = df[col].max()
if str(col_type)[:3] == 'int':
if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:
df[col] = df[col].astype(np.int8)
elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
df[col] = df[col].astype(np.int16)
elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
df[col] = df[col].astype(np.int32)
elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
df[col] = df[col].astype(np.int64)
else:
if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
df[col] = df[col].astype(np.float16)
elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
df[col] = df[col].astype(np.float32)
else:
df[col] = df[col].astype(np.float64)
else:
df[col] = df[col].astype('category')
end_mem = df.memory_usage().sum()
print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))
return df
Train_data = reduce_mem_usage(pd.read_csv('used_car_train_20200313.csv', sep=' '))
Test_data = reduce_mem_usage(pd.read_csv('used_car_testB_20200421.csv', sep=' '))
## 输出数据的大小信息
print('Train data shape:',Train_data.shape)
print('TestA data shape:',Test_data.shape)
#合并数据集 concat_data = pd.concat([Train_data,Test_data]) concat_data.isnull().sum()
这里咱们发现bodyType，fuelType和gearbox缺失比较多，model缺失一行，price因为是输出，这里不用额外处理。
分析V系列匿名特征和非V系列特征
对于匿名变量来讲，只有数值信息，须要更多关注到，这里把变量分为匿名变量和非匿名变量，单独进行分析
concat_data.columns
首先提取非匿名变量进行分析，随机抽样10行数据.
concat_data[['bodyType', 'brand', 'creatDate', 'fuelType', 'gearbox',
'kilometer', 'model', 'name', 'notRepairedDamage', 'offerType', 'power',
'regDate', 'regionCode', 'seller']].sample(10)
concat_data[['bodyType', 'brand', 'creatDate', 'fuelType', 'gearbox',
'kilometer', 'model', 'name', 'notRepairedDamage', 'offerType', 'power',
'regDate', 'regionCode', 'seller']].describe()
这里发现了列名为notRepairedDamage的取值中包含了"-"的异常值，这里使用众数进行替换
concat_data['notRepairedDamage'].value_counts()
concat_data['notRepairedDamage'] = concat_data['notRepairedDamage'].replace('-',0).astype('float16')
接着继续分析匿名变量
concat_data[['v_0', 'v_1', 'v_2', 'v_3',
'v_4', 'v_5', 'v_6', 'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12',
'v_13', 'v_14']].sample(10)
concat_data[['v_0', 'v_1', 'v_2', 'v_3',
'v_4', 'v_5', 'v_6', 'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12',
'v_13', 'v_14']].describe()
对于缺失值，先简单的使用众数进行填充。 填充完后，数据中再也不含有缺失值。
concat_data = concat_data.fillna(concat_data.mode().iloc[0,:])
print('concat_data shape:',concat_data.shape)
concat_data.isnull().sum()
对离散型数值进行独热编码
对于每个特征，若是它有m个可能值，那么通过独热编码后，就变成了m个二元特征（如成绩这个特征有好，中，差变成one-hot就是100, 010, 001）。而且，这些特征互斥，每次只有一个激活。所以，数据会变成稀疏的。 这样作的好处主要有：
-
解决了分类器很差处理属性数据的问题
-
在必定程度上也起到了扩充特征的做用
参考连接[https://www.cnblogs.com/zongfa/p/9305657.html]
能够经过df.value_counts().plot.bar来绘制数值分布状况
def plot_discrete_bar(data) :
cnt = data.value_counts()
p1 = plt.bar(cnt.index, height=list(cnt) , width=0.8)
for x,y in zip(cnt.index,list(cnt)):
plt.text(x+0.05,y+0.05,'%.2f' %y, ha='center',va='bottom')
clo_list = ['bodyType','fuelType','gearbox','notRepairedDamage']
i = 1
fig = plt.figure(figsize=(8,8))
for col in clo_list:
plt.subplot(2,2,i)
plot_discrete_bar(concat_data[col])
i = i + 1
对类别较少的特征采用one-hot编码，编码后特征由31个变为50个。
one_hot_list = ['gearbox','notRepairedDamage','bodyType','fuelType']
for col in one_hot_list:
one_hot = pd.get_dummies(concat_data[col])
one_hot.columns = [col+'_'+str(i) for i in range(len(one_hot.columns))]
concat_data = pd.concat([concat_data,one_hot],axis=1)
这里发现seller和offerType虽然应该是二分类的取值，可是分布状况都偏向于一种取值结果,能够直接删掉
concat_data['seller'].value_counts()
concat_data['offerType'].value_counts()
concat_data.drop(['offerType','seller'],axis=1,inplace=True)
对于匿名变量来讲，但愿能更多的使用到这里的数值信息，经过选取若干个非匿名变量和匿名变量进行加法和乘法的数值操做，扩展数据的特征
for i in ['v_' +str(t) for t in range(14)]:
for j in ['v_' +str(k) for k in range(int(i[2:])+1,15)]:
concat_data[str(i)+'+'+str(j)] = concat_data[str(i)]+concat_data[str(j)]
for i in ['model','brand', 'bodyType', 'fuelType','gearbox', 'power', 'kilometer', 'notRepairedDamage', 'regionCode']:
for j in ['v_' +str(i) for i in range(14)]:
concat_data[str(i)+'*'+str(j)] = concat_data[i]*concat_data[j]
concat_data.shape
对日期数据处理
日期数据一样是很重要的数据，有着具体的实际含义。这里咱们先提取出日期中的年月日，再具体分析每一个日期的数据。
# 设置日期的格式，例如20160404，设为2016-04-04，其中月份从1-12
def date_proc(x):
m = int(x[4:6])
if m == 0:
m = 1
return x[:4] + '-' + str(m) + '-' + x[6:]
#定义日期提取函数
def date_transform(df,fea_col):
for f in tqdm(fea_col):
df[f] = pd.to_datetime(df[f].astype('str').apply(date_proc))
df[f + '_year'] = df[f].dt.year
df[f + '_month'] = df[f].dt.month
df[f + '_day'] = df[f].dt.day
df[f + '_dayofweek'] = df[f].dt.dayofweek
return (df)
#提取日期信息 date_cols = ['regDate', 'creatDate'] concat_data = date_transform(concat_data,date_cols)
继续使用日期的数据，构造其余特征。分析var=data['creatDate'] - data['regDate']的含义，var表明了车辆注册日期和建立交易的日期之间相差的天数，能够从侧面反映出汽车使用的时长，通常来讲价格与使用时间成反比 不过要注意，数据里有时间出错的格式，因此咱们须要 errors='coerce'
data = concat_data.copy()
# 统计使用天数
data['used_time1'] = (pd.to_datetime(data['creatDate'], format='%Y%m%d', errors='coerce') -
pd.to_datetime(data['regDate'], format='%Y%m%d', errors='coerce')).dt.days
data['used_time2'] = (pd.datetime.now() - pd.to_datetime(data['regDate'], format='%Y%m%d', errors='coerce')).dt.days
data['used_time3'] = (pd.datetime.now() - pd.to_datetime(data['creatDate'], format='%Y%m%d', errors='coerce') ).dt.days
#分桶操做,划分到区间内
def cut_group(df,cols,num_bins=50):
for col in cols:
all_range = int(df[col].max()-df[col].min())
# print(all_range)
bin = [i*all_range/num_bins for i in range(all_range)]
df[col+'_bin'] = pd.cut(df[col], bin, labels=False) # 使用cut方法进行分箱
return df
#分桶操做
cut_cols = ['used_time1','used_time2','used_time3']
data = cut_group(data,cut_cols,50)
#分桶操做
data = cut_group(data,['kilometer'],10)
对年份和月份进行处理，继续使用独热编码
data['creatDate_year'].value_counts()
data['creatDate_month'].value_counts() # data['regDate_year'].value_counts()
# 对类别较少的特征采用one-hot编码
one_hot_list = ['creatDate_year','creatDate_month','regDate_month','regDate_year']
for col in one_hot_list:
one_hot = pd.get_dummies(data[col])
one_hot.columns = [col+'_'+str(i) for i in range(len(one_hot.columns))]
data = pd.concat([data,one_hot],axis=1)
# 删除无用的SaleID data.drop(['SaleID'],axis=1,inplace=True)
增长特征数量
增长特征的数量能够从数理统计的角度出发，经过选取一些变量的数理特性来增长数据维度
# count编码
def count_coding(df,fea_col):
for f in fea_col:
df[f + '_count'] = df[f].map(df[f].value_counts())
return(df)
#count编码
count_list = ['model', 'brand', 'regionCode','bodyType','fuelType','name','regDate_year', 'regDate_month', 'regDate_day',
'regDate_dayofweek' , 'creatDate_month','creatDate_day', 'creatDate_dayofweek','kilometer']
data = count_coding(data,count_list)
绘制热力图来分析匿名变量和price的相关性，其中v_0，v_8，v_12相关性较高
temp = Train_data[['v_0', 'v_1', 'v_2', 'v_3',
'v_4', 'v_5', 'v_6', 'v_7', 'v_8', 'v_9', 'v_10', 'v_11', 'v_12',
'v_13', 'v_14','price']]
# Zoomed heatmap, correlation matrix
sns.set(rc={'figure.figsize':(8,6)})
correlation_matrix = temp.corr()
k = 8 #number of variables for heatmap
cols = correlation_matrix.nlargest(k, 'price')['price'].index
cm = np.corrcoef(temp[cols].values.T)
sns.set(font_scale=1.25)
hm = sns.heatmap(cm, cbar=True, annot=True, square=True, fmt='.2f', annot_kws={'size': 10}, yticklabels=cols.values, xticklabels=cols.values)
plt.show()
#定义交叉特征统计
def cross_cat_num(df,num_col,cat_col):
for f1 in tqdm(cat_col): # 对类别特征遍历
g = df.groupby(f1, as_index=False)
for f2 in tqdm(num_col): # 对数值特征遍历
feat = g[f2].agg({
'{}_{}_max'.format(f1, f2): 'max',
'{}_{}_min'.format(f1, f2): 'min',
'{}_{}_median'.format(f1, f2): 'median',
})
df = df.merge(feat, on=f1, how='left')
return(df)
# 用数值特征对类别特征作统计刻画，挑了几个跟price相关性最高的匿名特征 cross_cat = ['model', 'brand','regDate_year'] cross_num = ['v_0','v_3', 'v_4', 'v_8', 'v_12','power'] data = cross_cat_num(data,cross_num,cross_cat)#一阶交叉
划分数据集
## 选择特征列
numerical_cols = data.columns
feature_cols = [col for col in numerical_cols if col not in ['price']]
## 提早特征列，标签列构造训练样本和测试样本
X_data = data.iloc[:len(Train_data),:][feature_cols]
Y_data = Train_data['price']
X_test = data.iloc[len(Train_data):,:][feature_cols]
print("X_data: ",X_data.shape)
print("X_test: ",X_test.shape)
平均数编码：针对高基数定性特征（类别特征）的数据预处理
定性特征的基数（cardinality）指的是这个定性特征全部可能的不一样值的数量。在高基数（high cardinality）的定性特征面前，这些数据预处理的方法每每得不到使人满意的结果。
高基数定性特征的例子：IP地址、电子邮件域名、城市名、家庭住址、街道、产品号码。
主要缘由：
- LabelEncoder编码高基数定性特征，虽然只须要一列，可是每一个天然数都具备不一样的重要意义，对于y而言线性不可分。使用简单模型，容易欠拟合（underfit），没法彻底捕获不一样类别之间的区别；使用复杂模型，容易在其余地方过拟合（overfit）。
- OneHotEncoder编码高基数定性特征，必然产生上万列的稀疏矩阵，易消耗大量内存和训练时间，除非算法自己有相关优化（例：SVM）。
所以，咱们能够尝试使用平均数编码（mean encoding）的编码方法，在贝叶斯的架构下，利用所要预测的应变量（target variable），有监督地肯定最适合这个定性特征的编码方式。在Kaggle的数据竞赛中，这也是一种常见的提升分数的手段。参考连接[https://blog.csdn.net/juzexia/article/details/78581462]
import numpy as np
import pandas as pd
from sklearn.model_selection import StratifiedKFold,KFold
from itertools import product
class MeanEncoder:
def __init__(self, categorical_features, n_splits=10, target_type='classification', prior_weight_func=None):
"""
:param categorical_features: list of str, the name of the categorical columns to encode
:param n_splits: the number of splits used in mean encoding
:param target_type: str, 'regression' or 'classification'
:param prior_weight_func:
a function that takes in the number of observations, and outputs prior weight
when a dict is passed, the default exponential decay function will be used:
k: the number of observations needed for the posterior to be weighted equally as the prior
f: larger f --> smaller slope
"""
self.categorical_features = categorical_features
self.n_splits = n_splits
self.learned_stats = {}
if target_type == 'classification':
self.target_type = target_type
self.target_values = []
else:
self.target_type = 'regression'
self.target_values = None
if isinstance(prior_weight_func, dict):
self.prior_weight_func = eval('lambda x: 1 / (1 + np.exp((x - k) / f))', dict(prior_weight_func, np=np))
elif callable(prior_weight_func):
self.prior_weight_func = prior_weight_func
else:
self.prior_weight_func = lambda x: 1 / (1 + np.exp((x - 2) / 1))
@staticmethod
def mean_encode_subroutine(X_train, y_train, X_test, variable, target, prior_weight_func):
X_train = X_train[[variable]].copy()
X_test = X_test[[variable]].copy()
if target is not None:
nf_name = '{}_pred_{}'.format(variable, target)
X_train['pred_temp'] = (y_train == target).astype(int) # classification
else:
nf_name = '{}_pred'.format(variable)
X_train['pred_temp'] = y_train # regression
prior = X_train['pred_temp'].mean()
col_avg_y = X_train.groupby(by=variable, axis=0)['pred_temp'].agg({'mean': 'mean', 'beta': 'size'})
col_avg_y['beta'] = prior_weight_func(col_avg_y['beta'])
col_avg_y[nf_name] = col_avg_y['beta'] * prior + (1 - col_avg_y['beta']) * col_avg_y['mean']
col_avg_y.drop(['beta', 'mean'], axis=1, inplace=True)
nf_train = X_train.join(col_avg_y, on=variable)[nf_name].values
nf_test = X_test.join(col_avg_y, on=variable).fillna(prior, inplace=False)[nf_name].values
return nf_train, nf_test, prior, col_avg_y
def fit_transform(self, X, y):
"""
:param X: pandas DataFrame, n_samples * n_features
:param y: pandas Series or numpy array, n_samples
:return X_new: the transformed pandas DataFrame containing mean-encoded categorical features
"""
X_new = X.copy()
if self.target_type == 'classification':
skf = StratifiedKFold(self.n_splits)
else:
skf = KFold(self.n_splits)
if self.target_type == 'classification':
self.target_values = sorted(set(y))
self.learned_stats = {'{}_pred_{}'.format(variable, target): [] for variable, target in
product(self.categorical_features, self.target_values)}
for variable, target in product(self.categorical_features, self.target_values):
nf_name = '{}_pred_{}'.format(variable, target)
X_new.loc[:, nf_name] = np.nan
for large_ind, small_ind in skf.split(y, y):
nf_large, nf_small, prior, col_avg_y = MeanEncoder.mean_encode_subroutine(
X_new.iloc[large_ind], y.iloc[large_ind], X_new.iloc[small_ind], variable, target, self.prior_weight_func)
X_new.iloc[small_ind, -1] = nf_small
self.learned_stats[nf_name].append((prior, col_avg_y))
else:
self.learned_stats = {'{}_pred'.format(variable): [] for variable in self.categorical_features}
for variable in self.categorical_features:
nf_name = '{}_pred'.format(variable)
X_new.loc[:, nf_name] = np.nan
for large_ind, small_ind in skf.split(y, y):
nf_large, nf_small, prior, col_avg_y = MeanEncoder.mean_encode_subroutine(
X_new.iloc[large_ind], y.iloc[large_ind], X_new.iloc[small_ind], variable, None, self.prior_weight_func)
X_new.iloc[small_ind, -1] = nf_small
self.learned_stats[nf_name].append((prior, col_avg_y))
return X_new
def transform(self, X):
"""
:param X: pandas DataFrame, n_samples * n_features
:return X_new: the transformed pandas DataFrame containing mean-encoded categorical features
"""
X_new = X.copy()
if self.target_type == 'classification':
for variable, target in product(self.categorical_features, self.target_values):
nf_name = '{}_pred_{}'.format(variable, target)
X_new[nf_name] = 0
for prior, col_avg_y in self.learned_stats[nf_name]:
X_new[nf_name] += X_new[[variable]].join(col_avg_y, on=variable).fillna(prior, inplace=False)[
nf_name]
X_new[nf_name] /= self.n_splits
else:
for variable in self.categorical_features:
nf_name = '{}_pred'.format(variable)
X_new[nf_name] = 0
for prior, col_avg_y in self.learned_stats[nf_name]:
X_new[nf_name] += X_new[[variable]].join(col_avg_y, on=variable).fillna(prior, inplace=False)[
nf_name]
X_new[nf_name] /= self.n_splits
return X_new
# 高基数定性特征：name汽车交易名称，brand汽车品牌，regionCode地区编码 class_list = ['model','brand','name','regionCode']+date_cols # date_cols = ['regDate', 'creatDate'] MeanEnocodeFeature = class_list # 声明须要平均数编码的特征 ME = MeanEncoder(MeanEnocodeFeature,target_type='regression') # 声明平均数编码的类 X_data = ME.fit_transform(X_data,Y_data) # 对训练数据集的X和y进行拟合 X_test = ME.transform(X_test)#对测试集进行编码
X_data['price'] = Train_data['price']
from sklearn.model_selection import KFold
# target encoding目标编码，回归场景相对来讲作目标编码的选择更多，不只能够作均值编码，还能够作标准差编码、中位数编码等
enc_cols = []
stats_default_dict = {
'max': X_data['price'].max(),
'min': X_data['price'].min(),
'median': X_data['price'].median(),
'mean': X_data['price'].mean(),
'sum': X_data['price'].sum(),
'std': X_data['price'].std(),
'skew': X_data['price'].skew(),
'kurt': X_data['price'].kurt(),
'mad': X_data['price'].mad()
}
### 暂且选择这三种编码
enc_stats = ['max','min','mean']
skf = KFold(n_splits=10, shuffle=True, random_state=42)
for f in tqdm(['regionCode','brand','regDate_year','creatDate_year','kilometer','model']):
enc_dict = {}
for stat in enc_stats:
enc_dict['{}_target_{}'.format(f, stat)] = stat
X_data['{}_target_{}'.format(f, stat)] = 0
X_test['{}_target_{}'.format(f, stat)] = 0
enc_cols.append('{}_target_{}'.format(f, stat))
for i, (trn_idx, val_idx) in enumerate(skf.split(X_data, Y_data)):
trn_x, val_x = X_data.iloc[trn_idx].reset_index(drop=True), X_data.iloc[val_idx].reset_index(drop=True)
enc_df = trn_x.groupby(f, as_index=False)['price'].agg(enc_dict)
val_x = val_x[[f]].merge(enc_df, on=f, how='left')
test_x = X_test[[f]].merge(enc_df, on=f, how='left')
for stat in enc_stats:
val_x['{}_target_{}'.format(f, stat)] = val_x['{}_target_{}'.format(f, stat)].fillna(stats_default_dict[stat])
test_x['{}_target_{}'.format(f, stat)] = test_x['{}_target_{}'.format(f, stat)].fillna(stats_default_dict[stat])
X_data.loc[val_idx, '{}_target_{}'.format(f, stat)] = val_x['{}_target_{}'.format(f, stat)].values
X_test['{}_target_{}'.format(f, stat)] += test_x['{}_target_{}'.format(f, stat)].values / skf.n_splits
drop_list = ['regDate', 'creatDate','brand_power_min', 'regDate_year_power_min'] x_train = X_data.drop(drop_list+['price'],axis=1) x_test = X_test.drop(drop_list,axis=1) x_train.shape
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
使用MinMaxScaler处理数据，再使用PCA下降维度
from sklearn.preprocessing import MinMaxScaler #特征归一化 min_max_scaler = MinMaxScaler() min_max_scaler.fit(pd.concat([x_train,x_test]).values) all_data = min_max_scaler.transform(pd.concat([x_train,x_test]).values)
print(all_data.shape) from sklearn import decomposition pca = decomposition.PCA(n_components=400) all_pca = pca.fit_transform(all_data) X_pca = all_pca[:len(x_train)] test = all_pca[len(x_train):] y = Train_data['price'].values print(all_pca.shape)
模型选择
这里以keras搭建基础的神经网络模型,模型结构选取全链接神经网络。
from keras.layers import Conv1D, Activation, MaxPool1D, Flatten, Dense
from keras.layers import Input, Dense, Concatenate, Reshape, Dropout, merge, Add
def NN_model(input_dim):
init = keras.initializers.glorot_uniform(seed=1)
model = keras.models.Sequential()
model.add(Dense(units=300, input_dim=input_dim, kernel_initializer=init, activation='softplus'))
#model.add(Dropout(0.2))
model.add(Dense(units=300, kernel_initializer=init, activation='softplus'))
#model.add(Dropout(0.2))
model.add(Dense(units=64, kernel_initializer=init, activation='softplus'))
model.add(Dense(units=32, kernel_initializer=init, activation='softplus'))
model.add(Dense(units=8, kernel_initializer=init, activation='softplus'))
model.add(Dense(units=1))
return model
from keras.callbacks import Callback, EarlyStopping
class Metric(Callback):
def __init__(self, model, callbacks, data):
super().__init__()
self.model = model
self.callbacks = callbacks
self.data = data
def on_train_begin(self, logs=None):
for callback in self.callbacks:
callback.on_train_begin(logs)
def on_train_end(self, logs=None):
for callback in self.callbacks:
callback.on_train_end(logs)
def on_epoch_end(self, batch, logs=None):
X_train, y_train = self.data[0][0], self.data[0][1]
y_pred3 = self.model.predict(X_train)
y_pred = np.zeros((len(y_pred3), ))
y_true = np.zeros((len(y_pred3), ))
for i in range(len(y_pred3)):
y_pred[i] = y_pred3[i]
for i in range(len(y_pred3)):
y_true[i] = y_train[i]
trn_s = mean_absolute_error(y_true, y_pred)
logs['trn_score'] = trn_s
X_val, y_val = self.data[1][0], self.data[1][1]
y_pred3 = self.model.predict(X_val)
y_pred = np.zeros((len(y_pred3), ))
y_true = np.zeros((len(y_pred3), ))
for i in range(len(y_pred3)):
y_pred[i] = y_pred3[i]
for i in range(len(y_pred3)):
y_true[i] = y_val[i]
val_s = mean_absolute_error(y_true, y_pred)
logs['val_score'] = val_s
print('trn_score', trn_s, 'val_score', val_s)
for callback in self.callbacks:
callback.on_epoch_end(batch, logs)
import keras.backend as K
from keras.callbacks import LearningRateScheduler
def scheduler(epoch):
# 每隔20个epoch，学习率减少为原来的0.5
if epoch % 20 == 0 and epoch != 0:
lr = K.get_value(model.optimizer.lr)
K.set_value(model.optimizer.lr, lr * 0.5)
print("lr changed to {}".format(lr * 0.5))
return K.get_value(model.optimizer.lr)
reduce_lr = LearningRateScheduler(scheduler)
#model.fit(train_x, train_y, batch_size=32, epochs=5, callbacks=[reduce_lr])
n_splits = 5
kf = KFold(n_splits=n_splits, shuffle=True)
import keras
b_size = 2000
max_epochs = 145
oof_pred = np.zeros((len(X_pca), ))
sub = pd.read_csv('used_car_testB_20200421.csv',sep = ' ')[['SaleID']].copy()
sub['price'] = 0
avg_mae = 0
for fold, (trn_idx, val_idx) in enumerate(kf.split(X_pca, y)):
print('fold:', fold)
X_train, y_train = X_pca[trn_idx], y[trn_idx]
X_val, y_val = X_pca[val_idx], y[val_idx]
model = NN_model(X_train.shape[1])
simple_adam = keras.optimizers.Adam(lr = 0.01)
model.compile(loss='mae', optimizer=simple_adam,metrics=['mae'])
es = EarlyStopping(monitor='val_score', patience=10, verbose=0, mode='min', restore_best_weights=True,)
es.set_model(model)
metric = Metric(model, [es], [(X_train, y_train), (X_val, y_val)])
model.fit(X_train, y_train, batch_size=b_size, epochs=max_epochs,
validation_data = [X_val, y_val],
callbacks=[reduce_lr], shuffle=True, verbose=0)
y_pred3 = model.predict(X_val)
y_pred = np.zeros((len(y_pred3), ))
sub['price'] += model.predict(test).reshape(-1,)/n_splits
for i in range(len(y_pred3)):
y_pred[i] = y_pred3[i]
oof_pred[val_idx] = y_pred
val_mae = mean_absolute_error(y[val_idx], y_pred)
avg_mae += val_mae/n_splits
print()
print('val_mae is:{}'.format(val_mae))
print()
mean_absolute_error(y, oof_pred)