Example 2: SVM

This notebook demonstrates two ways to fit a linear SVM with rehline:

1. Low-level API — decompose the hinge loss with plqcom (steps 1–3), then pass coefficients to ReHLine.

2. Scikit-learn style — call plq_Ridge_Classifier with the built-in svm loss (step 4b).

ERM objective:

\[\min_{\boldsymbol{\beta}} \;\tfrac{1}{2}\|\boldsymbol{\beta}\|_2^2 + C\sum_{i=1}^{n}\max(0,\,1-y_i\mathbf{x}_i^\top\boldsymbol{\beta}).\]

Here C = 0.5 is the ReHLine penalty weight (not the c argument of affine_transformation).

from plqcom import PLQLoss, plq_to_rehloss, affine_transformation
import numpy as np
from rehline import ReHLine

1. Data Generation

Synthetic labels with \(n{=}1000\), \(d{=}3\): \(\mathbf{y}=\mathrm{sgn}(\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\epsilon})\), \(\mathbf{X}\in\mathbb{R}^{n\times d}\).

n, d, C = 1000, 3, 0.5
np.random.seed(1024)
X = np.random.randn(1000, 3)
beta = np.random.randn(3)
y = np.sign(X.dot(beta) + np.random.randn(n))

Preview the first 10 samples:

X_head = X[:10]
y_head = y[:10]
X_head, y_head

(array([[ 2.12444863,  0.25264613,  1.45417876],
        [ 0.56923979,  0.45822365, -0.80933344],
        [ 0.86407349,  0.20170137, -1.87529904],
        [-0.56850693, -0.06510141,  0.80681666],
        [-0.5778176 ,  0.57306064, -0.33667496],
        [ 0.29700734, -0.37480416,  0.15510474],
        [ 0.70485719,  0.8452178 , -0.65818079],
        [ 0.56810558,  0.51538125, -0.61564998],
        [ 0.92611427, -1.28591285,  1.43014026],
        [-0.4254975 , -0.40257712,  0.60410409]]),
 array([ 1., -1., -1., -1., -1.,  1., -1., -1.,  1.,  1.]))

2. Create and Decompose the PLQ Loss

# Create a PLQLoss object
plqloss = PLQLoss(quad_coef={'a': np.array([0., 0.]), 'b': np.array([0., 1.]), 'c': np.array([0., 0.])},
                  cutpoints=np.array([0]))
# Decompose the SVM loss into PLQ composite loss
rehloss = plq_to_rehloss(plqloss)

print(rehloss.relu_coef, rehloss.relu_intercept)
print(rehloss.rehu_cut, rehloss.rehu_coef, rehloss.rehu_intercept)

[[1.]] [[-0.]]
[] [] []

3. Broadcast to All Samples

Per-sample hinge: \(L_i(z_i)=L(1-y_i z_i)\), i.e. \(p_i=-y_i\), \(q_i=1\). Use c=1 below; ERM strength is set by ReHLine(C=C) in step 4.

# Broadcast: c=1 (uniform weights); ERM strength via ReHLine(C=C) in step 4
rehloss = affine_transformation(rehloss, n=X.shape[0], c=1, p=-y, q=1)

print(rehloss.relu_coef.shape)
print("First ten sample relu coefficients: %s" % rehloss.relu_coef[0][:10])
print("First ten sample relu intercepts: %s" % rehloss.relu_intercept[0][:10])

(1, 1000)
First ten sample relu coefficients: [-0.5  0.5  0.5  0.5  0.5 -0.5  0.5  0.5 -0.5 -0.5]
First ten sample relu intercepts: [0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5]

4. Solve with ReHLine

rehline \(\geq\) 0.1.0 supports two calling styles:

• 4a. Low-level API — after plqcom decomposition (steps 1–3), pass rehloss coefficients via _U, _V, etc. Use this for custom PLQ composites.

• 4b. Scikit-learn style — for built-in losses (SVM, hinge, Huber, MSE, …), call plq_Ridge_Classifier / plq_Ridge_Regressor with fit(X, y) directly.

4a. Low-level API (plqcom Decomposition)

clf = ReHLine(C=C)
clf._U, clf._V = rehloss.relu_coef, rehloss.relu_intercept
clf.fit(X=X)
print('sol provided by rehline: %s' % clf.coef_)
print(clf.decision_function([[.1, .2, .3]]))

sol provided by rehline: [ 0.74100049 -0.00617326  2.66988444]
[0.87383073]

4b. Scikit-learn Style (Built-in SVM Loss)

Skip plqcom steps 1–3 for standard losses. Set fit_intercept=False to match the low-level setup above (no intercept).

from rehline import plq_Ridge_Classifier

clf_sk = plq_Ridge_Classifier(loss={'name': 'svm'}, C=C, fit_intercept=False)
clf_sk.fit(X, y)
print('sol provided by plq_Ridge_Classifier: %s' % clf_sk.coef_)
print(clf_sk.decision_function([[.1, .2, .3]]))