Example 2:SVM

This example try to show how to use our PLQ Composite Decomposition tool to decompose an SVM Loss function

Given a loss function \(L(\beta) = \frac{1}{2}||\beta||^2 + C\sum_{i=1}^n max(0, 1-y_i\beta^Tx_i)\), we can decompose it into a PLQ composite function.

from plqcom import PLQLoss, plq_to_rehloss, affine_transformation
import numpy as np
from rehline import ReHLine

1. Data Generation

First, we generate a classification dataset with 10000 samples and 5 features. \(\mathbf{Y}=sgn(\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon})\) where \(\mathbf{X} \in \mathbb{R}^{10000 \times 1}\), \(\mathbf{Y} \in \mathbb{R}^{10000}\), and \(\beta \in \mathbb{R}^{5}\) The true parameter \(\mathbf{\beta}\) is randomly generated.

n, d, C = 1000, 3, 0.5
np.random.seed(1024)
X = np.random.randn(1000, 3)
beta = np.random.randn(3)
y = np.sign(X.dot(beta) + np.random.randn(n))

Check the first 10 samples

X_head = X[:10]
y_head = y[:10]
X_head, y_head

(array([[ 2.12444863,  0.25264613,  1.45417876],
        [ 0.56923979,  0.45822365, -0.80933344],
        [ 0.86407349,  0.20170137, -1.87529904],
        [-0.56850693, -0.06510141,  0.80681666],
        [-0.5778176 ,  0.57306064, -0.33667496],
        [ 0.29700734, -0.37480416,  0.15510474],
        [ 0.70485719,  0.8452178 , -0.65818079],
        [ 0.56810558,  0.51538125, -0.61564998],
        [ 0.92611427, -1.28591285,  1.43014026],
        [-0.4254975 , -0.40257712,  0.60410409]]),
 array([ 1., -1., -1., -1., -1.,  1., -1., -1.,  1.,  1.]))

2.Create and Decompose the PLQ Loss

# Create a PLQLoss object
plqloss = PLQLoss(quad_coef={'a': np.array([0., 0.]), 'b': np.array([0., 1.]), 'c': np.array([0., 0.])},
                  cutpoints=np.array([0]))
# Decompose the SVM loss into PLQ composite loss
rehloss = plq_to_rehloss(plqloss)

print(rehloss.relu_coef, rehloss.relu_intercept)
print(rehloss.rehu_cut, rehloss.rehu_coef, rehloss.rehu_intercept)

[[1.]] [[-0.]]
[] [] []

3. Broadcast to all samples

# Broadcast the PLQ composite loss to all samples
rehloss = affine_transformation(rehloss, n=X.shape[0], c=C, p=-y, q=1)

print(rehloss.relu_coef.shape)
print("First ten sample relu coefficients: %s" % rehloss.relu_coef[0][:10])
print("First ten sample relu intercepts: %s" % rehloss.relu_intercept[0][:10])

(1, 1000)
First ten sample relu coefficients: [-0.5  0.5  0.5  0.5  0.5 -0.5  0.5  0.5 -0.5 -0.5]
First ten sample relu intercepts: [0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]

4. Use the ReHLine to solve the problem

clf = ReHLine()
clf.U, clf.V = rehloss.relu_coef, rehloss.relu_intercept
clf.fit(X=X)
print('sol privided by rehline: %s' % clf.coef_)
print(clf.decision_function([[.1, .2, .3]]))

sol privided by rehline: [ 0.74100049 -0.00617326  2.66988444]
[0.87383073]