SVM with Linear and Polynomial Kernels throw Accord.ConvergenceException

[ChristosManosEmpora]

Library version 3.0.2
I am using SequentialMinimalOptimization for binary classification learning. I tried both Linear and Polynomial Kernels with no success:

For example the SVM with Polynomial Kernel crashes with values:

this.Tolerance = 1e-4;
this.Complexity = 0.001;
this.Degree = 4;
this.Constant = 1; 

while the SVM with Linear Kernel crashes with values:

this.Tolerance = 1e-4;
this.Complexity = 0.316227766016838
this.Constant = 0; 

The data I used are from the Data Science London + Scikit-learn in Kaggle. I have attached the 2 files (train.csv and trainLabels.csv) necessary for the training.

Data Science London and Scikit-learn.zip

The following method will prepare the labelled data for training:

public static int[] LoadTrainingLabelsDataSet(string TrainLabelsLink)
{
    var lines = File.ReadAllLines(TrainLabelsLink).Select(a => a.Split(';'));
    var csv = from line in lines
              select (line[0].Split(',')).ToArray();
    int NoRows = csv.Count();
    int[] labels = new int[NoRows];
    int i = 0;
    foreach (string[] row in csv)
    {
        foreach (string value in row)
        {
            labels[i] = Convert.ToInt32(value) == 0 ? -1 : 1;
        }
        i++;
    }
    return labels;
}

The error I get is the following:

An exception of type 'Accord.ConvergenceException' occurred in Accord.MachineLearning.dll but was not handled in user code

Additional information: Convergence could not be attained. Please reduce the cost of misclassification errors by reducing the complexity parameter C or try a different kernel function.

I had the impression that error was already addressed in a previous version.
Is there something I am missing (maybe some boundaries i am crossing) or is it a bug?

[mar14]

I've same problem. Any ideas?

[cesarsouza]

Hello,

Sorry to ask, but have you tried to reduce the cost of misclassification by reducing the value for the complexity parameter as indicated in the exception?

The exception is not exactly a crash, but an indication that some of the parameters might indeed be a bit off.

In any case I will setup some more unit tests with the provided data. If you are using Linear kernels, you might also want to try the other learning algorithms, such as LinearDualCoordinateDescent, which might actually be faster and more stable than SMO in this case.

Best regards,
Cesar

[mar14]

I have specific parameters which I must use and I know that algorithm with that parameters and data should work because I check that in other machine learning tool.

Best regards.
Marcin

[cesarsouza]

Hi Marcin,

Thanks for the answer. Could you please share which other machine learning tool you are referencing to, so I could investigate?

Regards,
Cesar

[mar14]

I use Weka 3.8. Algorithm: SVM Polynomial Kernel with parameters: C=0.001, exponent 3, constant 1. My data:
DaneDostosowane.txt

[ChristosManosEmpora]

Thanks for the reply. Yes I have tried different complexity values in a grid search (i.e. C logarithmic range {0.0001-1}). Depending on the data set I use the error limit varies and in some cases I get that error throughout the entire range.

I was wondering if there is a way to indicate that limit and avoid the exception.

Regards,
Christos

[cesarsouza]

Well, given that the SVM has no closed form solution, it is impossible to know whether the learning algorithm with converge or diverge before training starts. Sometimes the complexity constant can be too tight for non-linear separable problems which would certainly lead to a convergence exception. This is by design, and cannot be handled otherwise, unfortunately.

[wup11111]

Hi Cesar,

I used the particle swarm optimization algorithm to find the optimal classifier parameters on a particular dataset. There was no problem when Gaussian is used as the kernel function, but the same problem was encountered when using polynomial kernel or linear kernel. Is there a way to skip the exception and continue with parameter optimization?

Best regards.
Wu Peng