用TensorFlow实现lasso回归和岭回归算法的示例_Python

也有些正则方法可以限制回归算法输出结果中系数的影响，其中最常用的两种正则方法是lasso回归和岭回归。

lasso回归和岭回归算法跟常规线性回归算法极其相似，有一点不同的是，在公式中增加正则项来限制斜率（或者净斜率）。这样做的主要原因是限制特征对因变量的影响，通过增加一个依赖斜率A的损失函数实现。

对于lasso回归算法，在损失函数上增加一项：斜率A的某个给定倍数。我们使用TensorFlow的逻辑操作，但没有这些操作相关的梯度，而是使用阶跃函数的连续估计，也称作连续阶跃函数，其会在截止点跳跃扩大。一会就可以看到如何使用lasso回归算法。

对于岭回归算法，增加一个L2范数，即斜率系数的L2正则。

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

									# LASSO and Ridge Regression

									# lasso回归和岭回归

									# 

									# This function shows how to use TensorFlow to solve LASSO or 

									# Ridge regression for 

									# y = Ax + b

									# 

									# We will use the iris data, specifically: 

									#  y = Sepal Length 

									#  x = Petal Width

									# import required libraries

									import matplotlib.pyplot as plt

									import sys

									import numpy as np

									import tensorflow as tf

									from sklearn import datasets

									from tensorflow.python.framework import ops

									# Specify 'Ridge' or 'LASSO'

									regression_type = 'LASSO'

									# clear out old graph

									ops.reset_default_graph()

									# Create graph

									sess = tf.Session()

									###

									# Load iris data

									###

									# iris.data = [(Sepal Length, Sepal Width, Petal Length, Petal Width)]

									iris = datasets.load_iris()

									x_vals = np.array([x[3] for x in iris.data])

									y_vals = np.array([y[0] for y in iris.data])

									###

									# Model Parameters

									###

									# Declare batch size

									batch_size = 50

									# Initialize placeholders

									x_data = tf.placeholder(shape=[None, 1], dtype=tf.float32)

									y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)

									# make results reproducible

									seed = 13

									np.random.seed(seed)

									tf.set_random_seed(seed)

									# Create variables for linear regression

									A = tf.Variable(tf.random_normal(shape=[1,1]))

									b = tf.Variable(tf.random_normal(shape=[1,1]))

									# Declare model operations

									model_output = tf.add(tf.matmul(x_data, A), b)

									###

									# Loss Functions

									###

									# Select appropriate loss function based on regression type

									if regression_type == 'LASSO':

									  # Declare Lasso loss function

									  # 增加损失函数，其为改良过的连续阶跃函数，lasso回归的截止点设为0.9。

									  # 这意味着限制斜率系数不超过0.9

									  # Lasso Loss = L2_Loss + heavyside_step,

									  # Where heavyside_step ~ 0 if A < constant, otherwise ~ 99

									  lasso_param = tf.constant(0.9)

									  heavyside_step = tf.truediv(1., tf.add(1., tf.exp(tf.multiply(-50., tf.subtract(A, lasso_param)))))

									  regularization_param = tf.multiply(heavyside_step, 99.)

									  loss = tf.add(tf.reduce_mean(tf.square(y_target - model_output)), regularization_param)

									elif regression_type == 'Ridge':

									  # Declare the Ridge loss function

									  # Ridge loss = L2_loss + L2 norm of slope

									  ridge_param = tf.constant(1.)

									  ridge_loss = tf.reduce_mean(tf.square(A))

									  loss = tf.expand_dims(tf.add(tf.reduce_mean(tf.square(y_target - model_output)), tf.multiply(ridge_param, ridge_loss)), 0)

									else:

									  print('Invalid regression_type parameter value',file=sys.stderr)

									###

									# Optimizer

									###

									# Declare optimizer

									my_opt = tf.train.GradientDescentOptimizer(0.001)

									train_step = my_opt.minimize(loss)

									###

									# Run regression

									###

									# Initialize variables

									init = tf.global_variables_initializer()

									sess.run(init)

									# Training loop

									loss_vec = []

									for i in range(1500):

									  rand_index = np.random.choice(len(x_vals), size=batch_size)

									  rand_x = np.transpose([x_vals[rand_index]])

									  rand_y = np.transpose([y_vals[rand_index]])

									  sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})

									  temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})

									  loss_vec.append(temp_loss[0])

									  if (i+1)%300==0:

									    print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b)))

									    print('Loss = ' + str(temp_loss))

									    print('\n')

									###

									# Extract regression results

									###

									# Get the optimal coefficients

									[slope] = sess.run(A)

									[y_intercept] = sess.run(b)

									# Get best fit line

									best_fit = []

									for i in x_vals:

									 best_fit.append(slope*i+y_intercept)

									###

									# Plot results

									###

									# Plot regression line against data points

									plt.plot(x_vals, y_vals, 'o', label='Data Points')

									plt.plot(x_vals, best_fit, 'r-', label='Best fit line', linewidth=3)

									plt.legend(loc='upper left')

									plt.title('Sepal Length vs Pedal Width')

									plt.xlabel('Pedal Width')

									plt.ylabel('Sepal Length')

									plt.show()

									# Plot loss over time

									plt.plot(loss_vec, 'k-')

									plt.title(regression_type + ' Loss per Generation')

									plt.xlabel('Generation')

									plt.ylabel('Loss')

									plt.show()

输出结果：

Step #300 A = [[ 0.77170753]] b = [[ 1.82499862]]
Loss = [[ 10.26473045]]
Step #600 A = [[ 0.75908542]] b = [[ 3.2220633]]
Loss = [[ 3.06292033]]
Step #900 A = [[ 0.74843585]] b = [[ 3.9975822]]
Loss = [[ 1.23220456]]
Step #1200 A = [[ 0.73752165]] b = [[ 4.42974091]]
Loss = [[ 0.57872057]]
Step #1500 A = [[ 0.72942668]] b = [[ 4.67253113]]
Loss = [[ 0.40874988]]

用TensorFlow实现lasso回归和岭回归算法的示例