Machine Learning SVM

Question

If one trains a model using a SVM from kernel data, the resultant trained model contains support vectors. Now consider the case of training a new model using the old data already present plus a small amount of new data as well. SO:

1. Should the new data just be combined with the support vectors from the previously formed model to form the new training set? (If yes, then how to combine the support vectors with new graph data? I am working on libsvm.)

   Or:

2. Should the new data and the complete old data be combined together and form the new training set and not just the support vectors of the old one?

Which approach is better for retraining, and/or is more doable and efficient in terms of accuracy and memory?

Answer 1

1 and 2 both describe techniques that would be valid in various circumstances.

In (1), you're basically using a partially stacked model (holding out the "new data", by which I assume you mean additional variables not considered by the initial SVM model).

In (2), you would be applying the SVM model to the entire dataset including the new variables.

It's very difficult to say with no other contextual knowledge which of these approaches would produce "better" results for your model- it would, as always, be best to try both approaches (with proper cross-validation) to determine the "better" approach in terms of your goals.

Assuming you already have the support vectors for the initial training of the model, (1) will be much cheaper (proportionally so to the number of independent variables in the initial model) in terms of memory than (2), as the number of dimensions (variables) passed to the algorithm is comparatively lower in (1).