Multiclass SVM and DTW System.AggregateException

[vutle]

Hi cesarouza,
I tried to use multiclass SVM with DTW with the following code but have the following error:

System.AggregateException: One or more errors occurred. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
at Accord.MachineLearning.VectorMachines.MulticlassSupportVectorMachine3.<>c__DisplayClass21_0.<distance>b__1(Int32 i, ParallelLoopState state, Tuple3 partial)
at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.b__1()
at System.Threading.Tasks.Task.InnerInvoke()
...

---

The code I used is given below. The data for the sequences is attached.
The data Shapes.csv contains (SequenceID,ClassID,ClassName,X,Y). There are only 3 classes (Circle, Triangle and Rectangle) with lots of sequences.

The code works fine when I have a few sequences. If I have this many sequences, I get that described error.

Do you know what is causing this?

  double[][] inputs = new double[Sequences.Count][];     //X,Y values sequences from attached file.
        int[] outputs = new int[Sequences.Count];  //Class 0,1,2
        int dimension = 2;
        int ClassCount = 3;

        for (int i = 0; i < Sequences.Count; i++)
        {
            outputs[i] = Sequences[i].ClassID;
            double[][] sequence = Preprocess(Sequences[i].ToArray());
            inputs[i] = Matrix.Concatenate(sequence); 
        }

              var gkernel = new  DynamicTimeWarping(length: dimension);

        svm = new MulticlassSupportVectorMachine <DynamicTimeWarping>(0, gkernel, ClassCount);

        // Create the learning algorithm to teach the multiple class classifier
        var teacher = new MulticlassSupportVectorLearning<DynamicTimeWarping>()
        {
            Learner = (param) => new SequentialMinimalOptimization<DynamicTimeWarping>()
            {
                Kernel =  new  DynamicTimeWarping(dimension),
            }
        };

        this.svm = teacher.Learn(inputs, outputs);

        int[] predicted = this.svm.Decide(inputs);  //This one runs ok

        try
        {
            for (int i = 0; i < Sequences.Count; i++)
            {

                double[][] sequence = Preprocess(Sequences[i].ToArray());
                double[] seq = Matrix.Concatenate(sequence);

                int outClass = this.svm.Decide(seq); //Error at this point here.



                Console.WriteLine(outClass);
            }
        }
        catch (Exception ex)
        {
            
            Console.WriteLine(ex.ToString());
        }

Shapes.zip

[mmehrle]

for (int i = 0; i <= Sequences.Count; i++)

…

On Mar 4, 2017, at 4:33 AM, vutle ***@***.***> wrote: for (int i = 0; i < Sequences.Count; i++)

[vutle]

Hi mmehrle,
Thanks for trying to help. But your loop will cause an out of range exception.
The error is inside Accord, not my loop. As I stated in the bug report, the code runs fine if I have a few sequences.

[vutle]

Hi cesarouza,
The error occurs at line 321 locks[j].Enter(ref taken); in MulticlassSupportVectorMachine.cs

Where j = sharedVectors[i] is greater than lock.Length (e.g. j = 66 when locks.Length = 36.

[cesarsouza]

Hello there,

Well, first of all, sorry for the super long delay in replying. I couldn't answer the issue at the time and I couldn't manage to get back into it under a reasonable amount of time...

Second, many thanks for reporting the issue! Indeed, this issue happened due to a problem with the function evaluation cache of the MulticlassSupportVectorMachine. I will be committing a fix in a few minutes.

Here is a slightly simplified version of the code above (but without the Preprocessing part), that I had added as a unit test to the framework, and that can serve as an example in case others would be interested in using the DynamicTimeWarping kernel and are looking for ideas:

var instances = CsvReader.FromText(Resources.Shapes, hasHeaders: false).Select(x => new
{
    InstanceId = int.Parse(x[0]),
    ClassId = int.Parse(x[1]),
    X = Double.Parse(x[3]),
    Y = Double.Parse(x[4])
});

var Sequences = (from a in instances
                    group a by a.InstanceId into g
                    select new
                    {
                        InstanceId = g.Key,
                        ClassID = g.First().ClassId,
                        Values = Enumerable.Zip(g.Select(x => x.X), g.Select(x => x.Y),
                            (a, b) => new double[] { a, b }).ToArray()
                    }).ToList();



double[][][] inputs = Sequences.Select(x=>x.Values).ToArray(); // X,Y values sequences from attached file.
int[] outputs = Sequences.Select(x=>x.ClassID).ToArray();      // Class 0,1,2


// Create the learning algorithm to teach the multiple class classifier
var teacher = new MulticlassSupportVectorLearning<DynamicTimeWarping, double[][]>()
{
    Learner = (param) => new SequentialMinimalOptimization<DynamicTimeWarping, double[][]>()
    {
        Kernel = new DynamicTimeWarping(length: 2), // (x, y) pairs
    }
};

// Learn the SVM using the SMO algorithm
var svm = teacher.Learn(inputs, outputs);

// Compute predicted values (at once, faster)
int[] predicted = svm.Decide(inputs);  

// Or, compute individual values and compare
for (int i = 0; i < inputs.Length; i++)
{
    int outClass = svm.Decide(inputs[i]); 
    Assert.AreEqual(predicted[i], outClass);
}

Regards,
Cesar

[vinicius121]

Hi cesarsouza,

Looking at the example that you post it, the Accord library has a function to read data from a CSV file.
But I don't understand how to use it.
What I don't understand is this line of the code: "CsvReader.FromText(Resources.Shapes, hasHeaders: false)". What is Resources?
What do you mean by this -> "The text containing the fields in the CSV format"? Is it the filepath?

Regards,
Vinícius

[cesarsouza]

Hi Vinícius,

Resources.Shapes is just the name of a variable that contained the .csv file as single string. Something like

string shapes = "col1, col2, col3\na,b,c\n,d,e,f";

If you would like to read from a file, just use the class constructor instead:

var reader = new CsvReader(path_to_your_csv_file_in_your_hd, hasHeaders: true);

Hope it helps!

Regards,
Cesar

[vinicius121]

Hi Cesar,

Thanks. My csv file is similar to the shapes.csv file with addition of one more column and I would like to organize it as you commented in the last post:
var instances = CsvReader.FromText(Resources.Shapes, hasHeaders: false).Select(x => new
{
InstanceId = int.Parse(x[0]),
ClassId = int.Parse(x[1]),
X = Double.Parse(x[2]),
Y = Double.Parse(x[3]),
Z = Double.Parse(x[4])
});
How can I do this after reading the file?

Regards,
Vinícius Silva

[vinicius121]

Hi Cesar,

I have just figured it out.
But I have one more question.
In this part of your code, since I have 3 columns of data (X,Y,Z), I tried to add what is in **** but I get the following error: "No overload for method 'Zip' takes 4 arguments"
var Sequences = (from a in instances group a by a.InstanceId into g select new { InstanceId = g.Key, ClassID = g.First().ClassId, Values = Enumerable.Zip(g.Select(x => x.X), g.Select(x => x.Y), **g.Select(x=> x.Z**) (a, b, **c**) => new double[] { a, b, **c** }).ToArray() }).ToList();
Is there an alternative to Zip?
Regards,
Vinícius Silva

[cesarsouza]

Hmmm... Linq can be daunting some times. If you are in doubt, just write it using plain for/foreach loops.

However, in your case, I think you can still use Zip if you Zip the columns two by two. Something like

Enumerable.Zip(Enumerable.Zip(g.Select(x => x.X), g.Select(x => x.Y)), g.Select(x=> x.Z))

But as I said, just write it using plain for loops. Linq is supposed to help with the readability (not with speed). So if it isn't helping with the readability for you, there is very little use in using it in the first place.

[vinicius121]

Hi Cesar,

Thanks for the replay.
I Zip the columns two by two as you said:
Values = Enumerable.Zip(Enumerable.Zip(g.Select(x => x.X), g.Select(x => x.Y), (a,b)=>new double[] { a,b}), g.Select(x=> x.Z), (b, d) => new double[] {b[0], b[1], d}).ToArray()

Finally, what namespace that I have to add for using this:
Assert.AreEqual(predicted[i], outClass);

Regards,
Vinícius Silva

[cesarsouza]

The Assert command comes from NUnit (the example was posted from inside a NUnit unit test). You do not need to use it in your application. Rather, just compare the contents of the predicted and output vectors and see if they are similar.

You can compute the error between those two by using the ConfusionMatrix, for example:

var cm = new ConfusionMatrix(expected: output, predicted: predicted);
double error = cm.Accuracy; // should be a value between 0 and 1 where 1 means everything got classified perfectly

Hope it helps,
Cesar

[vinicius121]

Hi Cesar,

It helped thanks.
I have a few questions.
My original data is from accelerometers (X, Y, Z). I have two gestures (2 classes).
We usually use a DTW kernel to classify sequences, but can we also use other kernels to do the same?
Like can I use the Gaussian kernel to classify sequences? Or I can combine both?
I ask this because I obtained better results by using only the Gaussian Kernel.
Here is the code:
double[][][] inputs = Sequences.Select(x => x.Values).ToArray(); int[] outputs = Sequences.Select(x => x.ClassID).ToArray();
double[][] inputsF = new double[inputs.Length][]; for (int i = 0; i < inputs.Length; i++) inputsF[i] = Accord.Math.Matrix.Concatenate(inputs[i]);
var learn = new SequentialMinimalOptimization<Gaussian>() { UseComplexityHeuristic = true, UseKernelEstimation = true };
SupportVectorMachine<Gaussian> svm = learn.Learn(inputsF, outputs);
bool[] prediction = svm.Decide(inputsF); double error = new ZeroOneLoss(outputs).Loss(prediction);

Regards,
Vinícius Silva

[cesarsouza]

Hi Vinicius,

Well, actually that is a bit surprising that the Gaussian kernel worked since its very probable that you have sequences with a variable number of observations. It should have thrown an exception in this case.

What is the average length of your sequences? Do different sequences have different lengths in the problem you are trying to solve?

Regards,
Cesar

[vinicius121]

Hi Cesar,

All my sequences has 30 samples each. They all have the same size. I did that on purpose in order to test with other kernels (to not get that exception).
Each sequence gesture has 1 sec of duration which translates to 30 samples for each sequence. My features are the euler angles: pitch, yaw, and roll (in that order, see the attachment file).
Other question I saw that Accord has Cross Validation methods. So, when I do a 10 fold cross validation, it means that 10 svm machines are trained and then my accuracy is the mean value? Can I train the same model 10 times, and obtain a more robust model, or I have to choose one of the trained models and use the one that I obtained the best performance? I ask this because in Matlab (in a previous work), I trained a multi-class SVM, and I used the cross validation method to obtain a more robust model, which can avoid overfitting.

Regards,
Vinícius Silva
Angles.zip

[vutle]

Thanks cesarouza, there is no more error.
Now I need to improve the data filter to get better recognition.

[cesarsouza]

Hi Vinicius,

If all your sequences have the same length it implies that you have already aligned them somehow. And if they are already aligned, then maybe there is no need to use Dynamic Time Warping and a standard kernel could also work as good or better than DTW.

Yes, Accord has methods for doing cross-validation, but the way it behaves depends on how you use them. If you do it in a way similar to this example and use K as 10, then it should create 10 SVMs on different partitions of your dataset and output the average classification error.

However, please note that k-fold cross validation should not be used to select the best model with the smallest error, at all. Instead, it should be used to help you determine which are the best parameters for your model, so that you can draw conclusions about the model performance without being bogged with fears that the performance you are seeing are actually due to a bias in the dataset rather than your choice of model / hyperparameters of your model.

I am not completely sure if that is what you did, but you should never "pick" one of the models trained on one of the folds in cross-validation. Rather, you should only consider the mean error to determine which hyperparameters you should use when training your final model in your entire dataset.

Regards,
Cesar

[vinicius121]

Hi Cesar,

Yes. I followed your example on cross validation and I only used it as a validation method for tuning parameters, as you recommended.
Thanks.

Regards,
Vinícius Silva

[cesarsouza]

Fixed in release 3.6.0.