Topic Categorization with PySS3

To open and run this notebook online, click here:

This is the notebook for the “Topic Categorization” tutorial. In this notebook, we will see how we can use the PySS3 Python package to deploy models for Sentiment Analysis on Movie Reviews.

Let us begin! First, we need to import the modules we will be using:

Before we begin, let’s import needed modules…

from pyss3 import SS3
from pyss3.util import Dataset, Evaluation, span
from pyss3.server import Live_Test

from sklearn.metrics import accuracy_score

… and unzip the “” dataset inside the datasets folder.

!unzip -u datasets/ -d datasets/

Ok, now we are ready to begin. Let’s create a new SS3 instance

clf = SS3()

What are the default hyperparameter values? let’s see

s, l, p, _ = clf.get_hyperparameters()

print("Smoothness(s):", s)
print("Significance(l):", l)
print("Sanction(p):", p)
Smoothness(s): 0.45
Significance(l): 0.5
Sanction(p): 1

Ok, now let’s load the training and the test set using the load_from_files function from pyss3.util. Since, in this dataset, there’s a single file for each category, we will use the argument folder_label=False to tell PySS3 to use each file as a different category and each line inside of it as a different document:

x_train, y_train = Dataset.load_from_files("datasets/topic/train", folder_label=False)
x_test, y_test = Dataset.load_from_files("datasets/topic/test", folder_label=False)
Loading 'sports' documents: 100%|██████████| 8/8 [00:00<00:00, 82.58it/s]
Loading 'sports' documents: 100%|██████████| 8/8 [00:00<00:00, 1220.96it/s]

Let’s train our model…

clf.train(x_train, y_train)  #, y_train)
Training on 'sports': 100%|██████████| 8/8 [00:29<00:00,  3.65s/it]

Note that we don’t have to create any document-term matrix! we are using just the plain x_train documents :D cool uh? SS3 learns a (spacial kind of) language model for each category and therefore it doesn’t need to create any document-term matrices :) in fact, the very concept of “document” becomes irrelevant…

Now that the model has been trained, let’s test it using the documents in x_test. First, we will do it “in the sklearn’s own way” with:

y_pred = clf.predict(x_test)

accuracy = accuracy_score(y_pred, y_test)

print("Accuracy was:", accuracy)
Classification: 100%|██████████| 800/800 [00:00<00:00, 842.02it/s]

Accuracy was: 0.70375

Alternatively, we could’ve done it “in the PySS3’s own way” by using the built-in test function provided by the Evaluation class that we have imported from pyss3.util at the beginning of this notebook, as follows:

Evaluation.test(clf, x_test, y_test)
                    precision    recall  f1-score   support

   art&photography       0.63      0.88      0.73       100
    beauty&fashion       0.77      0.68      0.72       100
  business&finance       0.76      0.71      0.73       100
              food       0.76      0.25      0.38       100
            health       0.53      0.80      0.64       100
             music       0.85      0.75      0.80       100
science&technology       0.60      0.75      0.67       100
            sports       0.99      0.81      0.89       100

          accuracy                           0.70       800
         macro avg       0.74      0.70      0.69       800
      weighted avg       0.74      0.70      0.69       800

The advantage of using this built-in function is that with just a single line of code we get:

  • The performance measured in terms of all the well-known metrics (‘accuracy’, ‘precision’, ‘recall’, and ‘f1-score’).

  • A plot showing the obtained confusion matrix, and…

  • Since all the evaluations performed using the Evaluation class are permanently cached, if we ever perform this test again, the evaluation will be skipped and values will be retrieved from the cache storage (saving us a lot of time! especially when performing long evaluations).

As we can see, the performance doesn’t look bad using the default hyperparameter values, however, let’s now manually analyze what our model has actually learned by using the interactive “live test”. Note that since we are not going to use the x_test for this live test(*) but instead the documents in “datasets/topic/live_test”, we must use the set_testset_from_files method to tell the Live Test to load documents from there instead.

(*) try it if you want but since ``x_test`` contains (preprocessed) tweets, they don’t look really good and clean.

#, x_test, y_test)  # <- this visualization doesn't look really clean and good so, instead,
                                      #    we will use the documents in "live_test" folder:

Classifying 'test docs' docs: 100%|██████████| 10/10 [00:00<00:00, 125.39it/s]

Makes sense to you? (remember you can use the options to select “words” as the Description Level if you want to know based on what words, and to what degree, is making classification decisions)

Live test doesn’t look bad, however, we will create a “more intelligent” version of this model, a version that can recognize variable-length word n-grams “on the fly”. Thus, when calling the train we will pass an extra argument n_grams=3 to indicate we want SS3 to learn to recognize important words, bigrams, and 3-grams (If you’re curious and want to know how this is actually done by SS3, read the paper “t-SS3: a text classifier with dynamic n-grams for early risk detection over text streams”, preprint available here ).

# ignore this (optional) ``name`` argument here, we're giving our
# model the name "topics-3grams" only to make things clearer
# for the (optional) last section
clf = SS3(name="topics_3grams")

clf.train(x_train, y_train, n_grams=3)  # <-- note the n_grams=3 argument here
Training on 'sports': 100%|██████████| 8/8 [00:36<00:00,  4.51s/it]

Now let’s see if the performance has improved…

y_pred = clf.predict(x_test)
print("Accuracy:", accuracy_score(y_pred, y_test))
Classification: 100%|██████████| 800/800 [00:01<00:00, 665.29it/s]

Accuracy: 0.71875

Yeah, the accuracy slightly improved but more importantly, we should now see that the model has learned “more intelligent patterns” involving sequences of words when using the interactive “live test” (like “machine learning”, “artificial intelligence”, “self-driving cars”, etc. for the “science&technology” category). Let’s see…

Fortunately, our model has learned to recognize these important sequences (such as “artificial intelligence” and “machine learning” in doc_2.txt, “self-driving cars” in doc_6.txt, etc.). However, some documents aren’t perfectly classified, for instance, doc_10.txt was classified as “science&technology” (as the second topic) which is clearly wrong…

So, one last thing we are going to do is to try to find better hyperparameter values to improve our model’s performance. For example, the following values will improve our classification performance:

clf.set_hyperparameters(s=0.32, l=1.62, p=2.55)

Let’s see if it’s true…

Evaluation.test(clf, x_test, y_test)
                    precision    recall  f1-score   support

   art&photography       0.70      0.85      0.77       100
    beauty&fashion       0.83      0.75      0.79       100
  business&finance       0.74      0.79      0.76       100
              food       0.87      0.62      0.73       100
            health       0.67      0.65      0.66       100
             music       0.87      0.86      0.86       100
science&technology       0.61      0.75      0.68       100
            sports       0.97      0.90      0.93       100

          accuracy                           0.77       800
         macro avg       0.78      0.77      0.77       800
      weighted avg       0.78      0.77      0.77       800

The accuracy has improved as expected and the confusion matrix look much better now :)

Finally, we could take a look at what our final model looks like using the Live Test tool one last time.

Want to know how we found out those hyperparameter values were going to improve our classifier accuracy? Just read the next section! ;)

Hyperparameter Optimization

In this section we will see how we can use PySS3’s Evaluation class to perform Hyperparameter optimization, which allows us to find better hyperparameter values for our models. To do this, we will perform grid searches using the Evaluation.grid_search() function.

Let’s create a new (standard) instance of the SS3 classifier. This will speed things up because the model we currently have in clf recognize variable-length word n-grams, the grid search won’t run as fast as with a (standard) model that recognize only words (and the same “best” hyperparameter values usually work for both of them).


Just ignore the (optional) name argument below, we’re giving our model the name “topics” only to make things clearer when we create the interactive 3D evaluation plot.

clf = SS3(name="topics")

clf.train(x_train, y_train)

The Evaluation.grid_search() takes, for each hyperparameter, the list of values to use in the search, for instance s=[0.25, 0.5, 0.75, 1] indicates you want the grid_search to try out evaluating the classifier using those 4 values for the sigma (s) hyperparameter. However, for simplicity, instead of using a manually crafted long list of values, we will use the span function we have imported from pyss3.util at the beginning of this notebook. This function will create a list of values for us, giving a lower and upper bound, and the number of elements to be generated. For instance, if we want a list of 6 numbers between 0 and 1, we could use:

span(0, 1, 6)
[0. , 0.2, 0.4, 0.6, 0.8, 1. ]

Thus, we will use the following values for each of the three hyperparameters:

s_vals=span(0.2, 0.8, 6)  # [0.2 , 0.32, 0.44, 0.56, 0.68, 0.8]
l_vals=span(0.1, 2, 6)    # [0.1 , 0.48, 0.86, 1.24, 1.62, 2]
p_vals=span(1.75, 2.75, 6)    # [1.75, 1.95, 2.15, 2.35, 2.55, 2.75]

To speed things up, unlike in the “Movie Reviews (Sentiment Analysis)” tutorial, we will perform the grid search using only the test set (we won’t use k-fold cross-validation). Once the search is over, Evaluation.grid_search will return the hyperparamter values that obtained the best accuracy for us.

# the search should take about 15 minutes
best_s, best_l, best_p, _ = Evaluation.grid_search(
    clf, x_test, y_test,
    s=s_vals, l=l_vals, p=p_vals
Grid search: 100%|██████████| 216/216 [15:04<00:00,  6.00s/it]
print("The hyperparameter values that obtained the best Accuracy are:")
print("Smoothness(s):", best_s)
print("Significance(l):", best_l)
print("Sanction(p):", best_p)
The hyperparameter values that obtained the best Accuracy are:
Smoothness(s): 0.32
Significance(l): 1.62
Sanction(p): 2.55

And that’s how we found out that these hyperparameter values (s=0.32, l=1.62, p=2.55) were going to improve our classifier accuracy.


What if we want to find hyperparameter values that performed the best but using a different metric other than accuracy? for example, what if we wanted to find the hyperparameter values that will improve the precision for the sports topic? we can use the Evaluation.get_best_hyperparameters() function as follows:

s, l, p, _ = Evaluation.get_best_hyperparameters(metric="precision", metric_target="sports")

print("s=%.2f, l=%.2f, and p=%.2f" % (s, l, p))
s=0.44, l=0.86, and p=1.75

Or the macro averaged f1 score?

s, l, p, _ = Evaluation.get_best_hyperparameters(metric="f1-score", metric_target="macro avg")

print("s=%.2f, l=%.2f, and p=%.2f" % (s, l, p))
s=0.32, l=1.62, and p=2.55

Alternatively, we could have also added these 2 arguments, metric and target, to the grid search in the first place :) (e.g. Evaluation.grid_search(..., metric="f1-score", metric_target="macro avg")).

Note that this get_best_hyperparameters function gave us the values right away! this is because instead of performing the grid search again, this function uses the evaluation cache to retrieve the best values from disk, which save us a lot of time!

Interactive 3D Evaluation Plot

The Evaluation class comes with a really useful function, Evaluation.plot(), that we can use to create an interactive 3D evaluation plot (We highly recommend reading this brief section, from the documentation, in which it is briefly described). Instead of using the single value returned from the Evaluation.grid_search() we could use this plot to have a broader view of the relationship between the different hyperparameter values and the performance of our model in the task being addressed. The Evaluation.plot() function creates a portable HTML file containing the interactive plot for us, and then opens it up in your browser. Let’s give it a shot:


You should see a plot like this one:

Rotate the camera and move the cursor over the pink point, that is the point that obtained the best performance (in terms of accuracy), that is, this is the point that the grid_search gave us. Among the information that is displayed when moving the cursor over the points, a compact version of the obtained confusion matrix is shown. Feel free to play a little bit with this interactive 3D evaluation plot, for instance try changing the metric and target from the options panel :D