PySS3 Package

Main module

This is the main module containing the implementation of the SS3 classifier.

(Please, visit https://github.com/sergioburdisso/pyss3 for more info)

exception pyss3.EmptyModelError(msg='')

Bases: Exception

Exception to be thrown when the model is empty.

exception pyss3.InvalidCategoryError(msg='')

Bases: Exception

Exception to be thrown when a category is not valid.

class pyss3.SS3(s=None, l=None, p=None, a=None, name='', cv_m='norm_gv_xai', sg_m='xai')

Bases: object

The SS3 classifier class.

The SS3 classifier was originally defined in Section 3 of https://dx.doi.org/10.1016/j.eswa.2019.05.023 (preprint avialable here: https://arxiv.org/abs/1905.08772)

Parameters
  • s (float) – the “smoothness”(sigma) hyperparameter value

  • l (float) – the “significance”(lambda) hyperparameter value

  • p (float) – the “sanction”(rho) hyperparameter value

  • a (float) – the alpha hyperparameter value (i.e. all terms with a confidence value (cv) less than alpha will be ignored during classification)

  • name (str) – the model’s name (to save and load the model from disk)

  • cv_m (str) – method used to compute the confidence value (cv) of each term (word or n-grams), options are: “norm_gv_xai”, “norm_gv” and “gv” (default: “norm_gv_xai”)

  • sg_m (str) – method used to compute the significance (sg) function, options are: “vanilla” and “xai” (default: “xai”)

classify(doc, prep=True, sort=True, json=False, prep_func=None)

Classify a given document.

Parameters
  • doc (str) – the content of the document

  • prep (bool) – enables the default input preprocessing (default: True)

  • sort (bool) – sort the classification result (from best to worst)

  • json (bool) – return a debugging version of the result in JSON format.

  • prep_func (function) – the custom preprocessing function to be applied to the given document before classifying it. If not given, the default preprocessing function will be used (as long as prep=True)

Returns

the document confidence vector if sort is False. If sort is True, a list of pairs (category index, confidence value) ordered by confidence value.

Return type

list

Raises

EmptyModelError

classify_label(doc, def_cat='most-probable', labels=True, prep=True)

Classify a given document returning the category label.

Parameters
  • doc (str) – the content of the document

  • def_cat (str) – default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “most-probable”)

  • labels (bool) – whether to return the category label or just the category index (default: True)

  • prep (bool) – enables the default input preprocessing process (default: True)

Returns

the category label or the category index.

Return type

str or int

Raises

InvalidCategoryError

classify_multilabel(doc, def_cat='unknown', labels=True, prep=True)

Classify a given document returning multiple category labels.

This method could be used to perform multi-label classification. Internally, it uses k-mean clustering on the confidence vector to select the proper group of labels.

Parameters
  • doc (str) – the content of the document

  • def_cat (str) – default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “unknown”)

  • labels (bool) – whether to return the category labels or just the category indexes (default: True)

  • prep (bool) – enables the default input preprocessing (default: True)

Returns

the list of category labels (or indexes).

Return type

list (of str or int)

Raises

InvalidCategoryError

cv(ngram, cat)

Return the “confidence value” of a given word n-gram for the given category.

This value is obtained applying a final transformation on the global value of the given word n-gram using the gv function [*].

These transformation are given when creating a new SS3 instance (see the SS3 class constructor’s cv_m argument for more information).

[*] the gv function is defined in Section 3.2.2 of the original paper:

https://arxiv.org/pdf/1905.08772.pdf

Examples:

>>> clf.cv("chicken", "food")
>>> clf.cv("roast chicken", "food")
>>> clf.cv("chicken", "sports")
Parameters
  • ngram (str) – the word or word n-gram

  • cat (str) – the category label

Returns

the confidence value

Return type

float

Raises

InvalidCategoryError

extract_insight(doc, cat='auto', level='word', window_size=3, min_cv=0.01, sort=True)

Get the list of text blocks involved in the classification decision.

Given a document, return the pieces of text that were involved in the classification decision, along with the confidence values associated with them. If a category is given, perform the process as if the given category were the one assigned by the classifier.

Parameters
  • doc (str) – the content of the document

  • cat (str) – the category in relation to which text blocks are obtained. If not present, it will automatically use the category assigned by SS3 after classification. Options are ‘auto’ or a given category name. (default: ‘auto’)

  • level (str) – the level at which text blocks are going to be extracted. Options are ‘word’, ‘sentence’ or ‘paragraph’. (default: ‘word’)

  • window_size (int) – the number of words, before and after each identified word, to be also included along with the identified word. For instance, window_size=0 means return only individual words, window_size=1 means also include the word that was before and the one that was after them. If multiple selected words are close enough for their word windows to be overlapping, then those word windows will be merged into a longer and single one. This argument is ignored when level is not equal to ‘word’. (default: 3)

  • min_cv (float) – the minimum confidence value each text block must have to be included in the output. (default 0.01)

  • sort (bool) – whether to return the text blocks ordered by their confidence value or not. If sort=False then blocks will be returned following the order they had in the input document. (default: True)

Returns

a list of pairs (text, confidence value) containing the text (blocks) involved, and to what degree (*), in the classification decision. (*) given by the confidence value

Return type

list

Raises

InvalidCategoryError, ValueError

fit(x_train, y_train, n_grams=1, prep=True, leave_pbar=True)

Train the model given a list of documents and category labels.

Parameters
  • x_train (list (of str)) – the list of documents

  • y_train (list of str for singlelabel classification; list of list of str for multilabel classification.) – the list of document labels

  • n_grams (int) – indicates the maximum n-grams to be learned (e.g. a value of 1 means only 1-grams (words), 2 means 1-grams and 2-grams, 3, 1-grams, 2-grams and 3-grams, and so on.

  • prep (bool) – enables the default input preprocessing (default: True)

  • leave_pbar (bool) – controls whether to leave the progress bar or remove it after finishing.

Raises

ValueError

get_a()

Get the alpha hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_alpha()

Get the alpha hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_categories(all=False)

Get the list of category names.

Returns

the list of category names

Return type

list (of str)

get_category_index(name)

Given its name, return the category index.

Parameters

name (str) – The category name

Returns

the category index (or IDX_UNKNOWN_CATEGORY if the category doesn’t exist).

Return type

int

get_category_name(index)

Given its index, return the category name.

Parameters

index (int) – The category index

Returns

the category name (or STR_UNKNOWN_CATEGORY if the category doesn’t exist).

Return type

str

get_hyperparameters()

Get hyperparameter values.

Returns

a tuple with hyperparameters current values (s, l, p, a)

Return type

tuple

get_l()

Get the “significance” (lambda) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_most_probable_category()

Get the name of the most probable category.

Returns

the name of the most probable category

Return type

str

get_name()

Return the model’s name.

Returns

the model’s name.

Return type

str

get_next_words(sent, cat, n=None)

Given a sentence, return the list of n (possible) following words.

Parameters
  • sent (str) – a sentence (e.g. “an artificial”)

  • cat (str) – the category name

  • n (int) – the maximum number of possible answers

Returns

a list of tuples (word, frequency, probability)

Return type

list (of tuple)

Raises

InvalidCategoryError

get_ngrams_length()

Return the length of longest learned n-gram.

Returns

the length of longest learned n-gram.

Return type

int

get_p()

Get the “sanction” (rho) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_s()

Get the “smoothness” (sigma) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_sanction()

Get the “sanction” (rho) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_significance()

Get the “significance” (lambda) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_smoothness()

Get the “smoothness” (sigma) hyperparameter value.

Returns

the hyperparameter value

Return type

float

get_stopwords(sg_threshold=0.01)

Get the list of (recognized) stopwords.

Parameters

sg_threshold (float) – significance (sg) value used as a threshold to consider words as stopwords (i.e. words with sg < sg_threshold for all categories will be considered as “stopwords”)

Returns

a list of stopwords

Return type

list (of str)

get_word(index)

Given the index, return the word.

Parameters

index (int) – the word index

Returns

the word (or STR_UNKNOWN_WORD if the word doesn’t exist).

Return type

int

Return type

str

get_word_index(word)

Given a word, return its index.

Parameters

name (str) – a word

Returns

the word index (or IDX_UNKNOWN_WORD if the word doesn’t exist).

Return type

int

gv(ngram, cat)

Return the “global value” of a given word n-gram for the given category.

(gv function is defined in Section 3.2.2 of the original paper:

https://arxiv.org/pdf/1905.08772.pdf)

Examples:

>>> clf.gv("chicken", "food")
>>> clf.gv("roast chicken", "food")
>>> clf.gv("chicken", "sports")
Parameters
  • ngram (str) – the word or word n-gram

  • cat (str) – the category label

Returns

the global value

Return type

float

Raises

InvalidCategoryError

learn(doc, cat, n_grams=1, prep=True, update=True)

Learn a new document for a given category.

Parameters
  • doc (str) – the content of the document

  • cat (str) – the category name

  • n_grams (int) – indicates the maximum n-grams to be learned (e.g. a value of 1 means only 1-grams (words), 2 means 1-grams and 2-grams, 3, 1-grams, 2-grams and 3-grams, and so on.

  • prep (bool) – enables the default input preprocessing (default: True)

  • update (bool) – enables model auto-update after learning (default: True)

load(path=None)

Load model from disk.

if a path is not present, the default will be used (“./”), However, if a path is given, it will not only used to load the model but also will overwrite the default path calling the SS3’s set_model_path(path) method (see set_model_path method documentation for more detail).

Parameters

path (str) – the path to load the model from

Raises

OSError

load_model(path=None)

Load model from disk.

if a path is not present, the default will be used (“./”), However, if a path is given, it will not only used to load the model but also will overwrite the default path calling the SS3’s set_model_path(path) method (see set_model_path method documentation for more detail).

Parameters

path (str) – the path to load the model from

Raises

OSError

lv(ngram, cat)

Return the “local value” of a given word n-gram for the given category.

(lv function is defined in Section 3.2.2 of the original paper:

https://arxiv.org/pdf/1905.08772.pdf)

Examples:

>>> clf.lv("chicken", "food")
>>> clf.lv("roast chicken", "food")
>>> clf.lv("chicken", "sports")
Parameters
  • ngram (str) – the word or word n-gram

  • cat (str) – the category label

Returns

the local value

Return type

float

Raises

InvalidCategoryError

plot_value_distribution(cat)

Plot the category’s global and local value distribution.

Parameters

cat (str) – the category name

Raises

InvalidCategoryError

predict(x_test, def_cat=None, labels=True, multilabel=False, prep=True, leave_pbar=True)

Classify a list of documents.

Parameters
  • x_test (list (of str)) – the list of documents to be classified

  • def_cat (str) –

    default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “most-probable”, or “unknown” for

    multi-label classification)

  • labels (bool) – whether to return the list of category names or just category indexes

  • multilabel (bool) – whether to perform multi-label classification or not. if enabled, for each document returns a list of labels instead of a single label (str). If the model was trained using multilabeled data, then this argument will be ignored and set to True.

  • prep (bool) – enables the default input preprocessing (default: True)

  • leave_pbar (bool) – controls whether to leave the progress bar or remove it after finishing.

Returns

if labels is True, the list of category names, otherwise, the list of category indexes.

Return type

list (of int or str)

Raises

EmptyModelError, InvalidCategoryError

predict_proba(x_test, prep=True, leave_pbar=True)

Classify a list of documents returning a list of confidence vectors.

Parameters
  • x_test (list (of str)) – the list of documents to be classified

  • prep (bool) – enables the default input preprocessing (default: True)

  • leave_pbar (bool) – controls whether to leave the progress bar after finishing or remove it.

Returns

the list of confidence vectors

Return type

list (of list of float)

Raises

EmptyModelError

print_categories_info()

Print information about learned categories.

print_hyperparameters_info()

Print information about hyperparameters.

print_model_info()

Print information regarding the model.

print_ngram_info(ngram)

Print debugging information about a given n-gram.

Namely, print the n-gram frequency (fr), local value (lv), global value (gv), confidence value (cv), sanction (sn) weight, significance (sg) weight.

Parameters

ngram (str) – the n-gram (e.g. “machine”, “machine learning”, etc.)

save(path=None)

Save the model to disk.

if a path is not present, the default will be used (“./”), However, if a path is given, it will not only used to save the model but also will overwrite the default path calling the SS3’s set_model_path(path) method (see set_model_path method documentation for more detail).

Parameters

path (str) – the path to save the model to

Raises

OSError

save_cat_vocab(cat, path='./', n_grams=- 1)

Save category vocabulary to disk.

Parameters
  • cat (str) – the category name

  • path (str) – the path in which to store the vocabulary

  • n_grams (int) – indicates the n-grams to be stored (e.g. only 1-grams, 2-grams, 3-grams, etc.). Default -1 stores all learned n-grams (1-grams, 2-grams, 3-grams, etc.)

Raises

InvalidCategoryError

save_model(path=None)

Save the model to disk.

if a path is not present, the default will be used (“./”), However, if a path is given, it will not only used to save the model but also will overwrite the default path calling the SS3’s set_model_path(path) method (see set_model_path method documentation for more detail).

Parameters

path (str) – the path to save the model to

Raises

OSError

save_vocab(path='./', n_grams=- 1)

Save learned vocabularies to disk.

Parameters
  • path (str) – the path in which to store the vocabularies

  • n_grams (int) – indicates the n-grams to be stored (e.g. only 1-grams, 2-grams, 3-grams, etc.). Default -1 stores all learned n-grams (1-grams, 2-grams, 3-grams, etc.)

set_a(value)

Set the alpha hyperparameter value.

All terms with a confidence value (cv) less than alpha will be ignored during classification.

Parameters

value (float) – the hyperparameter value

set_alpha(value)

Set the alpha hyperparameter value.

All terms with a confidence value (cv) less than alpha will be ignored during classification.

Parameters

value (float) – the hyperparameter value

set_block_delimiters(parag=None, sent=None, word=None)

Overwrite the default delimiters used to split input documents into blocks.

delimiters are any regular expression from simple ones (e.g. " ") to more complex ones (e.g. r"[^\s\w\d]"). Note: remember that there are certain reserved characters for regular expression,

for example, the dot (.), in which case use the backslash to indicate you’re referring the character itself and not its interpretation (e.g. \.)

e.g.

>>> ss3.set_block_delimiters(word="\s")
>>> ss3.set_block_delimiters(word="\s", parag="\n\n")
>>> ss3.set_block_delimiters(parag="\n---\n")
>>> ss3.set_block_delimiters(sent="\.")
>>> ss3.set_block_delimiters(word="\|")
>>> ss3.set_block_delimiters(word=" ")
Parameters
  • parag (str) – the paragraph new delimiter

  • sent (str) – the sentence new delimiter

  • word (str) – the word new delimiter

set_delimiter_paragraph(regex)

Set the delimiter used to split documents into paragraphs.

Remember that there are certain reserved characters for regular expression, for example, the dot (.), in which case use the backslash to indicate you’re referring the character itself and not its interpretation (e.g. \.)

Parameters

regex (str) – the regular expression of the new delimiter

set_delimiter_sentence(regex)

Set the delimiter used to split documents into sentences.

Remember that there are certain reserved characters for regular expression, for example, the dot (.), in which case use the backslash to indicate you’re referring the character itself and not its interpretation (e.g. \.)

Parameters

regex (str) – the regular expression of the new delimiter

set_delimiter_word(regex)

Set the delimiter used to split documents into words.

Remember that there are certain reserved characters for regular expression, for example, the dot (.), in which case use the backslash to indicate you’re referring the character itself and not its interpretation (e.g. \.)

Parameters

regex (str) – the regular expression of the new delimiter

set_hyperparameters(s=None, l=None, p=None, a=None)

Set hyperparameter values.

Parameters
  • s (float) – the “smoothness” (sigma) hyperparameter

  • l (float) – the “significance” (lambda) hyperparameter

  • p (float) – the “sanction” (rho) hyperparameter

  • a (float) – the alpha hyperparameter (i.e. all terms with a confidence value (cv) less than alpha will be ignored during classification)

set_l(value)

Set the “significance” (lambda) hyperparameter value.

Parameters

value (float) – the hyperparameter value

set_model_path(path)

Overwrite the default path from which the model will be loaded (or saved to).

Note: be aware that the PySS3 Command Line tool looks for a local folder called ss3_models to load models. Therefore, the ss3_models folder will be always automatically append to the given path (e.g. if path="my/path/", it will be converted into my/path/ss3_models).

Parameters

path (str) – the path

set_name(name)

Set the model’s name.

Parameters

name (str) – the model’s name.

set_p(value)

Set the “sanction” (rho) hyperparameter value.

Parameters

value (float) – the hyperparameter value

set_s(value)

Set the “smoothness” (sigma) hyperparameter value.

Parameters

value (float) – the hyperparameter value

set_sanction(value)

Set the “sanction” (rho) hyperparameter value.

Parameters

value (float) – the hyperparameter value

set_significance(value)

Set the “significance” (lambda) hyperparameter value.

Parameters

value (float) – the hyperparameter value

set_smoothness(value)

Set the “smoothness” (sigma) hyperparameter value.

Parameters

value (float) – the hyperparameter value

sg(ngram, cat)

Return the “significance factor” of a given word n-gram for the given category.

(sg function is defined in Section 3.2.2 of the original paper:

https://arxiv.org/pdf/1905.08772.pdf)

Examples:

>>> clf.sg("chicken", "food")
>>> clf.sg("roast chicken", "food")
>>> clf.sg("chicken", "sports")
Parameters
  • ngram (str) – the word or word n-gram

  • cat (str) – the category label

Returns

the significance factor

Return type

float

Raises

InvalidCategoryError

sn(ngram, cat)

Return the “sanction factor” of a given word n-gram for the given category.

(sn function is defined in Section 3.2.2 of the original paper:

https://arxiv.org/pdf/1905.08772.pdf)

Examples:

>>> clf.sn("chicken", "food")
>>> clf.sn("roast chicken", "food")
>>> clf.sn("chicken", "sports")
Parameters
  • ngram (str) – the word or word n-gram

  • cat (str) – the category label

Returns

the sanction factor

Return type

float

Raises

InvalidCategoryError

summary_op_ngrams(cvs)

Summary operator for n-gram confidence vectors.

By default it returns the addition of all confidence vectors. However, in case you want to use a custom summary operator, this function must be replaced as shown in the following example:

>>> def my_summary_op(cvs):
>>>     return cvs[0]
>>> ...
>>> clf = SS3()
>>> ...
>>> clf.summary_op_ngrams = my_summary_op

Note that any function receiving a list of vectors and returning a single vector could be used. In the above example the summary operator is replaced by the user-defined my_summary_op which ignores all confidence vectors returning only the confidence vector of the first n-gram (which besides being an illustrative example, makes no real sense).

Parameters

cvs (list (of list of float)) – a list n-grams confidence vectors

Returns

a sentence confidence vector

Return type

list (of float)

summary_op_paragraphs(cvs)

Summary operator for paragraph confidence vectors.

By default it returns the addition of all confidence vectors. However, in case you want to use a custom summary operator, this function must be replaced as shown in the following example:

>>> def dummy_summary_op(cvs):
>>>     return cvs[0]
>>> ...
>>> clf = SS3()
>>> ...
>>> clf.summary_op_paragraphs = dummy_summary_op

Note that any function receiving a list of vectors and returning a single vector could be used. In the above example the summary operator is replaced by the user-defined dummy_summary_op which ignores all confidence vectors returning only the confidence vector of the first paragraph (which besides being an illustrative example, makes no real sense).

Parameters

cvs (list (of list of float)) – a list paragraph confidence vectors

Returns

the document confidence vector

Return type

list (of float)

summary_op_sentences(cvs)

Summary operator for sentence confidence vectors.

By default it returns the addition of all confidence vectors. However, in case you want to use a custom summary operator, this function must be replaced as shown in the following example:

>>> def dummy_summary_op(cvs):
>>>     return cvs[0]
>>> ...
>>> clf = SS3()
>>> ...
>>> clf.summary_op_sentences = dummy_summary_op

Note that any function receiving a list of vectors and returning a single vector could be used. In the above example the summary operator is replaced by the user-defined dummy_summary_op which ignores all confidence vectors returning only the confidence vector of the first sentence (which besides being an illustrative example, makes no real sense).

Parameters

cvs (list (of list of float)) – a list sentence confidence vectors

Returns

a paragraph confidence vector

Return type

list (of float)

train(x_train, y_train, n_grams=1, prep=True, leave_pbar=True)

Train the model given a list of documents and category labels.

Parameters
  • x_train (list (of str)) – the list of documents

  • y_train (list of str for singlelabel classification; list of list of str for multilabel classification.) – the list of document labels

  • n_grams (int) – indicates the maximum n-grams to be learned (e.g. a value of 1 means only 1-grams (words), 2 means 1-grams and 2-grams, 3, 1-grams, 2-grams and 3-grams, and so on.

  • prep (bool) – enables the default input preprocessing (default: True)

  • leave_pbar (bool) – controls whether to leave the progress bar or remove it after finishing.

Raises

ValueError

update(force=False)

Update model values (cv, gv, lv, etc.).

Parameters

force (bool) – force update (even if hyperparameters haven’t changed)

update_values(force=False)

Update model values (cv, gv, lv, etc.).

Parameters

force (bool) – force update (even if hyperparameters haven’t changed)

pyss3.key_as_int(dct)

Cast the given dictionary (numerical) keys to int.

pyss3.kmean_multilabel_size(res)

Use k-means to tell where to split the ``SS3.classify’’’s output.

Given a SS3.classify’s output (res), tell where to partition it into 2 clusters so that one of the cluster holds the category labels that the classifier should output when performing multi-label classification. To achieve this, implement k-means (i.e. 2-means) clustering over the category confidence values in res.

Parameters

res (list (of sorted pairs (category, confidence value))) – the classification output of SS3.classify

Returns

a positive integer indicating where to split res

Return type

int

pyss3.list_hash(str_list)

Return a hash value for a given list of string.

Parameters

str_list (list (of str)) – a list of strings (e.g. x_test)

Returns

an MD5 hash value

Return type

str

pyss3.mad(values, n)

Median absolute deviation mean.

pyss3.re_split_keep(regex, string)

Force the inclusion of unmatched items by re.split.

This allows keeping the original content after splitting the input document for later use (e.g. for using it from the Live Test)

pyss3.set_verbosity(level)

Set the verbosity level.

  • 0 (quiet): do not output any message (only error messages)

  • 1 (normal): default behavior, display only warning messages and progress bars

  • 2 (verbose): display also the informative non-essential messages

The following built-in constants can also be used to refer to these 3 values: VERBOSITY.QUIET, VERBOSITY.NORMAL, and VERBOSITY.VERBOSE, respectively.

For example, if you want PySS3 to hide everything, even progress bars, you could simply do:

>>> import pyss3
...
>>> pyss3.set_verbosity(0)
...
>>> # here's the rest of your code :D

or, equivalently:

>>> import pyss3
>>> from pyss3 import VERBOSITY
...
>>> pyss3.set_verbosity(VERBOSITY.QUIET)
...
>>> # here's the rest of your code :D
Parameters

level (int) – the verbosity level

pyss3.sigmoid(v, l)

A sigmoid function.

pyss3.vdiv(v0, v1)

Vectorial version of division.

pyss3.vmax(v0, v1)

Vectorial version of max.

pyss3.vsum(v0, v1)

Vectorial version of sum.

Submodules

pyss3.server module

SS3 classification server with visual explanations for live tests.

(Please, visit https://github.com/sergioburdisso/pyss3 for more info)

pyss3.server.Live_Test

alias of pyss3.server.Server

class pyss3.server.Server

Bases: object

SS3’s Live Test HTTP server class.

static get_port()

Return the server port.

Returns

the server port

Return type

int

run(x_test=None, y_test=None, port=0, browser=True, quiet=True, prep=True, prep_func=None, def_cat=None)

Wait for classification requests and serve them.

Parameters
  • clf (pyss3.SS3) – the SS3 model to be attached to this server.

  • x_test (list (of str)) – the list of documents to classify and visualize

  • y_label (list (of str)) – the list of category labels

  • port (int) – the port to listen on (default: random free port)

  • browser (bool) – if True, it automatically opens up the live test on your browser

  • quiet (bool) – if True, use quiet mode. Otherwise use verbose mode (default: False)

  • prep (bool) – enables the default input preprocessing when classifying (default: True)

  • prep_func (function) – the custom preprocessing function to be applied to the given document before classifying it. If not given, the default preprocessing function will be used

  • def_cat (str) –

    default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “most-probable”, or “unknown” for

    multi-label classification)

Raises

ValueError

static serve(clf=None, x_test=None, y_test=None, port=0, browser=True, quiet=True, prep=True, prep_func=None, def_cat=None)

Wait for classification requests and serve them.

Parameters
  • clf (pyss3.SS3) – the SS3 model to be attached to this server.

  • x_test (list (of str)) – the list of documents to classify and visualize

  • y_label (list (of str)) – the list of category labels

  • port (int) – the port to listen on (default: random free port)

  • browser (bool) – if True, it automatically opens up the live test on your browser

  • quiet (bool) – if True, use quiet mode. Otherwise use verbose mode (default: False)

  • prep (bool) – enables the default input preprocessing when classifying (default: True)

  • prep_func (function) – the custom preprocessing function to be applied to the given document before classifying it. If not given, the default preprocessing function will be used

  • def_cat (str) –

    default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “most-probable”, or “unknown” for

    multi-label classification)

Raises

ValueError

static set_model(clf)

Attach a given SS3 model to this server.

Parameters

clf (pyss3.SS3) – an SS3 model

static set_testset(x_test, y_test=None, def_cat=None)

Assign the test set to visualize.

Parameters
  • x_test (list (of str)) – the list of documents to classify and visualize

  • y_label (list (of str)) – the list of category labels

  • def_cat (str) –

    default category to be assigned when SS3 is not able to classify a document. Options are “most-probable”, “unknown” or a given category name. (default: “most-probable”, or “unknown” for

    multi-label classification)

Raises

ValueError

static set_testset_from_files(test_path, folder_label=True, sep_doc='\n')

Load the test set files to visualize from test_path.

Parameters
  • test_path (str) – the test set path

  • folder_label (bool) – if True, read category labels from folders, otherwise, read category labels from file names. (default: True)

Returns

True if category documents were found, False otherwise

Return type

bool

static set_testset_from_files_multilabel(docs_path, labels_path, sep_label=None, sep_doc='\n')

Multilabel version of the Live_Test.set_testset_from_files() function.

Load test documents and category labels from disk to visualize in the Live Test tool.

Parameters
  • docs_path (str) – the file or the folder containing the test documents.

  • labels_path (str) – the file containing the labels for each document. * if docs_path is a file, then the labels_path file should contain a line with the corresponding list of category labels for each document in docs_path. * if docs_path is a folder containing the documents, then the labels_path file should contain a line for each document and category label. Each line should have the following format: document_name<sep_label>label.

  • sep_label (str) – the separator/delimiter used to separate either each label (if docs_path is a file) or the document name from its category (if docs_path is a folder). (default: ';' when docs_path is a file, the '\s+' regular expression otherwise).

  • sep_doc (str) – the separator/delimiter used to separate each document when loading training/test documents from single file. Valid only when folder_label=False. (default: \n')

static start_listening(port=0)

Start listening on a port and return its number.

(If a port number is not given, it uses a random free port).

Parameters

port (int) – the port to listen on

pyss3.server.content_type(ext)

Given a file extension, return the content type.

pyss3.server.get_http_body(http_request)

Given a HTTP request, return the body.

pyss3.server.get_http_contlength(http_request)

Given a HTTP request, return the Content-Length value.

pyss3.server.get_http_path(http_request)

Given a HTTP request, return the resource path.

pyss3.server.main()

The main function to be called when called from the command-line.

pyss3.server.parse_and_sanitize(rsc_path)

Very simple function to parse and sanitize the given path.

pyss3.cmd_line module

This module lets you interact with your SS3 models through a Command Line.

(Please, visit https://github.com/sergioburdisso/pyss3 for more info)

exception pyss3.cmd_line.ArgsParseError

Bases: Exception

Exception thrown when an error occur parsing commands arguments.

exception pyss3.cmd_line.LoadDataError

Bases: Exception

Exception thrown when an error occur while retrieving the test data.

class pyss3.cmd_line.SS3Prompt(completekey='tab', stdin=None, stdout=None)

Bases: cmd.Cmd

Prompt main class.

args_classify(args)

Parse classify arguments.

args_evaluations(args)

Parse evaluations arguments.

Parse grid_search arguments.

args_k_fold(args)

Parse k_fold arguments.

args_learn(args)

Parse learn arguments.

args_live_test(args)

Parse live_test arguments.

args_save(args)

Parse save arguments.

args_set(args)

Parse set arguments.

args_test(args)

Parse test arguments.

args_train(args)

Parse train arguments.

complete_evaluations(text, line, begidx, endidx)

Complete arguments for ‘grid_search’ command.

complete_get(text, line, begidx, endidx)

Complete arguments for ‘set’ command.

Complete arguments for ‘grid_search’ command.

complete_info(text, line, begidx, endidx)

Complete arguments for ‘info’ command.

complete_k_fold(text, line, begidx, endidx)

Complete arguments for ‘grid_search’ command.

complete_ld(text, line, begidx, endidx)

Complete arguments for ‘load’ command.

complete_learn(text, line, begidx, endidx)

Complete arguments for ‘learn’ command.

complete_live_test(text, line, begidx, endidx)

Complete arguments for ‘test’ command.

complete_load(text, line, begidx, endidx)

Complete arguments for ‘load’ command.

complete_plot(text, line, begidx, endidx)

Complete arguments for ‘plot’ command.

complete_save(text, line, begidx, endidx)

Complete arguments for ‘save’ command.

complete_set(text, line, begidx, endidx)

Complete arguments for ‘set’ command.

complete_sv(text, line, begidx, endidx)

Complete arguments for ‘save’ command.

complete_test(text, line, begidx, endidx)

Complete arguments for ‘test’ command.

complete_train(text, line, begidx, endidx)

Complete arguments for ‘train’ command.

default(line)

Default error message.

do_EOF(args='')

Quit the program.

do_classify(**kwargs)

Classify a document.

usage:

classify [DOCUMENT_PATH]

optional arguments:

DOCUMENT_PATH the path to the document file

do_clone(**kwargs)

Create a copy of the current model with a given name.

usage:

clone NEW_MODEL_NAME

required arguments:

NEW_MODEL_NAME the new model’s name

do_debug_term(**kwargs)

Show debugging information about a given n-gram.

Namely, print the n-gram frequency (fr), local value (lv), global value (gv), confidence value (cv), sanction (sn) weight and significance (sg) weight.

usage:

debug_term N_GRAM

required arguments:

N_GRAM the n-gram (word, bigram, trigram, etc.) to debug

examples:

debug_term the debug_term potato debug_term “machine learning” debug_term “self driving car”

do_evaluations(**kwargs)

Perform different actions linked to evaluations results.

usage:

evaluations OPTION [PATH] [METHOD] [DEF_CAT] [P VAL [P VAL …]

required arguments:
OPTION indicates the action to perform
values: {info,plot,save,remove} (default: info)
info - show information about evaluations (including

best values).

plot - show an interactive 3-D plot with evaluation

results in the web browser (it also save it to disk).

save - save the interactive 3-D plot to disk. remove - delete evaluations results from history

optional arguments:

PATH the dataset path used in the evaluate of interest

METHOD the method that was used in the evaluate of interest

values: {test,K-fold} where K is an integer > 1

DEF_CAT default category used in the evaluate of interest

values: {most-probable,unknown} or a category label

P VAL the hyperparameter value (only for option “remove”)

P values: {s,l,p,a} VAL values: float

examples:
  • show information about all evaluations:

    evaluations info

  • show information about evaluations in path “a/dataset/path”:

    evaluations info a/dataset/path

  • information about 3-fold evaluations in path “a/dataset/path”:

    evaluations info a/dataset/path 3-fold

  • information about test evaluations in path “a/dataset/path”:

    evaluations info a/dataset/path test

  • plot evaluations:

    evaluations plot

  • save evaluations:

    evaluations save

  • remove all evaluation result(s) in path “a/dataset/path”:

    evaluations remove a/dataset/path

  • remove 4-fold evaluation result(s) in path “a/dataset/path” with l = 1.1 and s = .45:

    evaluations remove a/dataset/path 4-fold l 1.1 s .45

do_exit(args='')

Quit the program.

do_get(**kwargs)

Get a given hyperparameter value.

usage:

get PARAM

required arguments:
PARAM the hyperparameter name

values: {s,l,p,a}

examples:

get s get l get p get a

Given a dataset, perform a grid search using the given hyperparameters values.

usage:

grid_search PATH [LABEL] [DEF_CAT] [METHOD] P EXP [P EXP …] [no-cache]

required arguments:

PATH the dataset path P EXP a list of values for a given hyperparameter.

where:

P is a hyperparameter name. values: {s,l,p,a} EXP is a python expression returning a float or

a list of floats. Note: if this expression contains whitespaces, use quotations marks (e.g. “[0.5, 1.5]”)

examples:

s [.3,.4,.5] s “[.3, .4, .5]” (Note the whitespaces and the “”) p r(.2,.8,6) (i.e. 6 points between .2 to .8)

optional arguments:
LABEL where to read category labels from.

values:{file,folder} (default: folder)

DEF_CAT default category to be assigned when the model is not

able to actually classify a document. values: {most-probable,unknown} or a category label (default: most-probable)

METHOD the method to be used

values: {test, K-fold} (default: test) where:

K-fold indicates the number of folds to be used.

K is an integer > 1 (e.g 4-fold, 10-fold, etc.)

no-cache if present, disable the cache and recompute all the values

examples:

grid_search a/testset/path s r(.2,.8,6) l r(.1,2,6) -p r(.5,2,6) a [0,.01] grid_search a/dataset/path 4-fold -s [.2,.3,.4,.5] -l [.5,1,1.5] -p r(.5,2,6)

do_info(**kwargs)

Show useful information.

usage:

info OPTION

required arguments:
OPTION indicates what information to show
values: {all, parameters, categories, evaluations}

(default: all)

examples:

info info evaluations

do_k_fold(**kwargs)

Perform a stratified k-fold validation using the given dataset set.

usage:

k_fold PATH [LABEL] [DEF_CAT] [N-grams] [N-fold] [P VAL …] [no-cache]

required arguments:

PATHthe dataset path

optional arguments:
LABEL where to read category labels from.

values:{file,folder} (default: folder)

DEF_CAT default category to be assigned when the model is not

able to actually classify a document. values: {most-probable,unknown} or a category label (default: most-probable)

N-grams indicates the maximum n-grams to be learned (e.g. a

value of “1-grams” means only words will be learned; “2-grams” only 1-grams and 2-grams; “3-grams”, only 1-grams, 2-grams and 3-grams; and so on). value: {N-grams} with N integer > 0 (default: 1-grams)

K-fold indicates the number of folds to be used.

value: {K-fold} with K integer > 1 (default: 4-fold)

P VAL sets a hyperparameter value (e.g. s 0.45)

P values: {s,l,p,a} VAL values: float

no-cache if present, disable the cache and recompute values

examples:

k_fold a/dataset/path 10-fold k_fold a/dataset/path 4-fold -s .45 -l 1.1 -p 1

do_learn(**kwargs)

Learn a new document.

usage:

learn CAT [N-grams] [DOCUMENT_PATH]

required arguments:

CAT the category label

optional arguments:
N-grams indicates the maximum n-grams to be learned (e.g. a

value of “1-grams” means only words will be learned; “2-grams” only 1-grams and 2-grams; “3-grams”, only 1-grams, 2-grams and 3-grams; and so on). value: {N-grams} with N integer > 0 (default: 1-grams)

DOCUMENT_PATH the path to the document file

do_license(args)

Print the license.

do_live_test(**kwargs)

Interactively and graphically test the model.

usage:

live_test [TEST_PATH [LABEL]] [verbose]

optional arguments:

TEST_PATH the test set path

LABEL where to read category labels from.

values: {file,folder} (default: folder)

verbose if present, run in verbose mode

examples:

live_test live_test a/testset/path live_test a/testset/path verbose

do_load(**kwargs)

Load a local model (given its name).

usage:

load MODEL_NAME

required arguments:

MODEL_NAME the model’s name

do_new(**kwargs)

Create a new empty SS3 model with a given name.

usage:

new MODEL_NAME

required arguments:

MODEL_NAME the model’s name

do_next_word(**kwargs)

Show up to 3 possible words to follow after the given sentence.

usage:

next_word SENT

required arguments:

SENT a sentence

examples:

next_word “the self driving” next_word “a machine learning”

do_plot(**kwargs)

Plot word value distribution curve or the evaluation results.

usage:

plot OPTION

required arguments:
OPTION indicates what to plot
values:

evaluations; distribution CAT;

where:

CAT the category label

examples:

plot distribution a_category plot evaluations

do_rename(**kwargs)

Rename the current model with a given name.

usage:

rename NEW_MODEL_NAME

required arguments:

NEW_MODEL_NAME the model’s new name

do_save(**kwargs)

Save to disk the model, learned vocabulary, evaluations results, etc.

usage:

save OPTION

required arguments:
OPTION indicates what to save to disk
values:

model; (default) evaluations; vocabulary [CAT]; stopwords [SG_THRESHOLD];

where:

CAT the category label

SG_THRESHOLD significance (sg) value used as a

threshold to consider words as stopwords (i.e. words with sg < sg_threshold for all categories will be considered as “stopwords”) (default: .01)

examples:

save save model save vocabulary save vocabulary a_category save stopwords save stopwords .1

do_set(**kwargs)

Set a given hyperparameter value.

usage:

set P VAL [P VAL …]

required arguments:
P VAL sets a hyperparameter value

examples: s .45; s .5; P values: {s,l,p,a} VAL values: float

examples:

set s .5 set l 0.5 set p 2 set s .5 l 0.5 p 2

do_test(**kwargs)

Test the model using the given test set.

usage:

test TEST_PATH [LABEL] [DEF_CAT] [P VAL …] [no-cache]

required arguments:

TEST_PATH the test set path

optional arguments:
LABEL where to read category labels from.

values:{file,folder} (default: folder)

DEF_CAT default category to be assigned when the model is not

able to actually classify a document. values: {most-probable,unknown} or a category label (default: most-probable)

P VAL sets a hyperparameter value

examples: s .45; s .5; P values: {s,l,p,a} VAL values: float

no-cache if present, disable the cache and recompute values

examples:

test a/testset/path test a/testset/path -s .45 -l 1.1 -p 1 test a/testset/path unknown -s .45 -l 1.1 -p 1 no-cache

do_train(**kwargs)

Train the model using a training set and then save it.

usage:

train TRAIN_PATH [LABEL] [N-gram]

required arguments:

TRAIN_PATH the training set path

optional arguments:
LABEL where to read category labels from.

values:{file,folder} (default: folder)

N-grams indicates the maximum n-grams to be learned (e.g. a

value of “1-grams” means only words will be learned; “2-grams” only 1-grams and 2-grams; “3-grams”, only 1-grams, 2-grams and 3-grams; and so on). value: {N-grams} with N integer > 0 (default: 1-grams)

examples:

train a/training/set/path 3-grams

do_update(**kwargs)

Update model values (cv, gv, lv, etc.).

precmd(line)

Hook method executed just before the command.

preloop()

Hook method executed once when cmdloop() is called.

pyss3.cmd_line.evaluation_plot(open_browser=True)

Plot the interactive 3D evaluation plot.

pyss3.cmd_line.evaluation_remove(data_path, method, def_cat, hparams)

Remove the evaluation results selected by the user.

pyss3.cmd_line.intersect(l0, l1)

Given two lists return the intersection.

pyss3.cmd_line.load_data(data_path, folder_label, cmd_name='test')

Load documents from disk, return the x_data, y_data and categories.

pyss3.cmd_line.main()

Main function.

pyss3.cmd_line.overwrite_model(model_path, model_name)

Remove both, the model and cache file.

pyss3.cmd_line.parse_hparams_args(op_args, defaults=True)

Parse hyperparameters arguments list.

pyss3.cmd_line.re_in(regex, l)

Given a list of strings, return the first match in the list.

pyss3.cmd_line.requires_args(func)

A @decorator.

pyss3.cmd_line.requires_model(func)

A @decorator.

pyss3.cmd_line.split_args(args)

Parse and split arguments.

pyss3.cmd_line.subtract(l0, l1)

Subtract list l1 from l0.

pyss3.cmd_line.train(x_train, y_train, n_grams, train_path='', folder_label=None, save=True, leave_pbar=True)

Train a new model with the given training set.

pyss3.util module

This is a helper module with utility classes and functions.

class pyss3.util.Dataset

Bases: object

A helper class with methods to read datasets from disk.

static load_from_files(data_path, folder_label=True, as_single_doc=False, sep_doc='\n')

Load training/test documents and category labels from disk.

Parameters
  • data_path (str) – the training or the test set path

  • folder_label (bool) – if True, read category labels from folders, otherwise, read category labels from file names. (default: True)

  • as_single_doc (bool) – read the documents as a single (and big) document (default: False)

  • sep_doc (str) – the separator/delimiter used to separate each document when loading training/test documents from single file. Valid only when folder_label=False. (default: '\n')

Returns

the (x_train, y_train) or (x_test, y_test) pairs.

Return type

tuple

static load_from_files_multilabel(docs_path, labels_path, sep_label=None, sep_doc='\n')

Multilabel version of the Dataset.load_from_files() function.

Load training/test documents and category labels from disk.

Parameters
  • docs_path (str) – the file or the folder containing the training/test documents.

  • labels_path (str) –

    the file containing the labels for each document.

    • if docs_path is a file, then the labels_path file

    should contain a line with the corresponding list of category labels for each document in docs_path. For instance, if sep_doc='\n' and the the content of docs_path is:

    this is document 1
    this is document 2
    this is document 3
    

    then, if sep_label=';', the labels_path file should contain the labels for each document (in order) separated by ;, as follows:

    labelA;labelB
    labelA
    labelB;labelC
    
    • if docs_path is a folder containing the documents, then

    the labels_path file should contain a line for each document and category label. Each line should have the following format: document_name<the sep_label>label. For instance, if the docs_path folder contains the following 3 documents:

    doc1.txt
    doc2.txt
    doc3.txt
    

    Then, following the above example, the labels_path file should be:

    doc1    labelA
    doc1    labelB
    doc2    labelA
    doc3    labelB
    doc3    labelC
    

  • sep_label (str) – the separator/delimiter used to separate either each label (if docs_path is a file) or the document name from its category (if docs_path is a folder). (default: ';' when docs_path is a file, the '\s+' regular expression otherwise).

  • sep_doc (str) – the separator/delimiter used to separate each document when loading training/test documents from single file. Valid only when folder_label=False. (default: \n')

Returns

the (x_train, y_train) or (x_test, y_test) pairs.

Return type

tuple

Raises

ValueError

class pyss3.util.Evaluation

Bases: object

Evaluation class.

This class provides the user easy-to-use methods for model evaluation and hyperparameter optimization, like, for example, the Evaluation.test, Evaluation.kfold_cross_validation, Evaluation.grid_search, Evaluation.plot methods for performing tests, stratified k-fold cross validations, grid searches for hyperparameter optimization, and visualizing evaluation results using an interactive 3D plot, respectively.

All the evaluation methods provided by this class internally use a cache mechanism in which all the previously computed evaluation will be permanently stored for later use. This will prevent the user to waste time performing the same evaluation more than once, or, in case of computer crashes or power failure during a long evaluation, once relunched, it will skip previously computed values and continue the evaluation from the point were the crashed happened on.

Usage:

>>> from pyss3.util import Evaluation

For examples usages for the previously mentioned method, read their documentation, or do one of the tutorials.

static clear_cache(clf=None)

Wipe out the evaluation cache (for the given classifier).

Parameters

clf (SS3) – the classifier (optional)

static get_best_hyperparameters(metric=None, metric_target='macro avg', tag=None, method=None, def_cat=None)

Return the best hyperparameter values for the given metric.

From all the evaluations performed using the given method, default category(def_cat) and cache tag, this method returns the hyperparameter values that performed the best, according to the given metric, if not supplied, these values will automatically use the ones matching the last performed evaluation.

Available metrics are: ‘accuracy’, ‘f1-score’, ‘precision’, and ‘recall’. In addition, In multi-label classification also ‘hamming-loss’ and ‘exact-match’

Except for accuracy, a metric_target option must also be supplied along with the metric indicating the target we aim at measuring, that is, whether we want to measure some averaging performance or the performance on a particular category.

Parameters
  • metric (str) – the evaluation metric to return, options are: ‘accuracy’, ‘f1-score’, ‘precision’, or ‘recall’ When working with multi-label classification problems, two more options are allowed: ‘hamming-loss’ and ‘exact-match’. Note: exact match will produce the same result than ‘accuracy’. (default: ‘accuracy’, or ‘hamming-loss’ for multi-label case).

  • metric_target (str) – the target we aim at measuring with the given metric. Options are: ‘macro avg’, ‘micro avg’, ‘weighted avg’ or a category label (default ‘macro avg’).

  • tag (str) – the cache tag from where to look up the results (by default it will automatically use the tag of the last evaluation performed)

  • method (str) – the evaluation method used, options are: ‘test’, ‘K-fold’, where K is a positive integer (by default it will match the method of the last evaluation performed).

  • def_cat (str) – the default category used the evaluations, options are: ‘most-probable’, ‘unknown’ or a category label (by default it will use the same as the last evaluation performed).

Returns

a tuple of the hyperparameter values: (s, l, p, a).

Return type

tuple

Raises

ValueError, LookupError, KeyError

Perform a grid search using the provided hyperparameter values.

Given a test or a training set, this method performs a grid search using the given lists of hyperparameters values. Once finished, it returns the best hyperparameter values found for the given metric.

If the argument k_fold is provided, the grid search will perform a stratified k-fold cross validation for each hyperparameter value combination. If k_fold is not given, will use the x_data as if it were a test set (x_test) and will use this test set to evaluate the classifier performance for each hyperparameter value combination.

Examples:

>>> from pyss3.util import Evaluation
>>> ...
>>> best_s, _, _, _ = Evaluation.grid_search(clf, x_test, y_test, s=[.3, .4, .5])
>>> print("For this test set, the value of s that obtained the "
>>>       "best accuracy, among .3, .4, and .5, was:", best_s)
>>> ...
>>> s, l, p, _ = Evaluation.grid_search(clf,
>>>                                     x_test, y_test,
>>>                                     s = [.3, .4, .5],
>>>                                     l = [.5, 1, 1.5],
>>>                                     p = [.5, 1, 2])
>>> print("For this test set and these hyperparameter values, "
>>>       "the value of s, l and p that obtained the best accuracy were, "
>>>       "respectively:", s, l, p)
>>> ...
>>> # since this grid search performs the same computation than the above
>>> # cached values will be used, instead of computing all over again.
>>> s, l, p, _ = Evaluation.grid_search(clf,
>>>                                     x_test, y_test,
>>>                                     s = [.3, .4, .5],
>>>                                     l = [.5, 1, 1.5],
>>>                                     p = [.5, 1, 2],
>>>                                     metric="f1-score")
>>> print("For this test set and these hyperparameter values, "
>>>       "the value of s, l and p that obtained the best F1 score were, "
>>>       "respectively:", s, l, p)
>>> ...
>>> s, l, p, _ = Evaluation.grid_search(clf,
>>>                                     x_train, y_train,
>>>                                     s = [.3, .4, .5],
>>>                                     l = [.5, 1, 1.5],
>>>                                     p = [.5, 1, 2],
>>>                                     k_fold=4)
>>> print("For this training set and these hyperparameter values, "
>>>       "and using stratified 4-fold cross validation, "
>>>       "the value of s, l and p that obtained the best accuracy were, "
>>>       "respectively:", s, l, p)
Parameters
  • clf (SS3) – the classifier to be evaluated.

  • x_data (list (of str)) – a list of documents

  • y_data (list (of str)) – a list of document category labels

  • s – the list of values for the s hyperparameter (optional). If not given, will take the classifier (clf) current value.

  • l – the list of values for the l hyperparameter (optional). If not given, will take the classifier (clf) current value.

  • p – the list of values for the p hyperparameter (optional). If not given, will take the classifier (clf) current value.

  • a – the list of values for the a hyperparameter (optional). If not given, will take the classifier (clf) current value.

  • k_fold (int) – indicates the number of folds to be used (optional). If not given, it will perform the grid search using the x_data as the test test.

  • n_grams (int) – indicates the maximum n-grams to be learned (e.g. a value of 1 means only 1-grams (words), 2 means 1-grams and 2-grams, 3, 1-grams, 2-grams and 3-grams, and so on.

  • def_cat (str) – default category to be assigned when SS3 is not able to classify a document. Options are ‘most-probable’, ‘unknown’ or a given category name. (default: ‘most-probable’)

  • prep (bool) – enables the default input preprocessing when classifying (default: True)

  • tag (str) – the cache tag to be used, i.e. a string to identify this evaluation inside the cache storage (optional)

  • metric (str) – the evaluation metric to return, options are: ‘accuracy’, ‘f1-score’, ‘precision’, or ‘recall’ When working with multi-label classification problems, two more options are allowed: ‘hamming-loss’ and ‘exact-match’. Note: exact match will produce the same result than ‘accuracy’. (default: ‘accuracy’, or ‘hamming-loss’ for multi-label case).

  • metric_target (str) – the target we aim at measuring with the given metric. Options are: ‘macro avg’, ‘micro avg’, ‘weighted avg’ or a category label (default ‘macro avg’).

  • cache (bool) – whether to use cached values or not. Setting cache=False forces to completely perform the evaluation ignoring cached values (default: True).

  • extended_pbar (bool) – whether to show an extra status bar along with the progress bar (default: False).

Returns

a tuple of hyperparameter values (s, l, p, a) with the best values for the given metric

Return type

tuple

Raises

InvalidCategoryError, EmptyModelError, ValueError, TypeError

static kfold_cross_validation(clf, x_train, y_train, k=4, n_grams=None, def_cat='most-probable', prep=True, tag=None, plot=True, metric=None, metric_target='macro avg', cache=True)

Perform a Stratified k-fold cross validation on the given training set.

Examples:

>>> from pyss3.util import Evaluation
>>> ...
>>> acc = Evaluation.kfold_cross_validation(clf, x_train, y_train)
>>> print("Accuracy obtained using (default) 4-fold cross validation:", acc)
>>> ...
>>> # this line won't perform the cross validation again, it will retrieve, from the
>>> #  cache storage, the f1-score value computed in previous evaluation
>>> f1 = Evaluation.kfold_cross_validation(clf, x_train, y_train, metric="f1-score")
>>> print("F1 score obtained using (default) 4-fold cross validation:", f1)
>>> ...
>>> f1 = Evaluation.kfold_cross_validation(clf, x_train, y_train, k=10, metric="f1-score")
>>> print("F1 score obtained using 10-fold cross validation:", f1)
Parameters
  • clf (SS3) – the classifier to be evaluated.

  • x_train (list (of str)) – the list of documents

  • y_train (list (of str)) – the list of document category labels

  • k (int) – indicates the number of folds to be used (default: 4).

  • n_grams (int) – indicates the maximum n-grams to be learned (e.g. a value of 1 means only 1-grams (words), 2 means 1-grams and 2-grams, 3, 1-grams, 2-grams and 3-grams, and so on.

  • def_cat (str) – default category to be assigned when SS3 is not able to classify a document. Options are ‘most-probable’, ‘unknown’ or a given category name. (default: ‘most-probable’)

  • prep (bool) – enables the default input preprocessing when classifying (default: True)

  • tag (str) – the cache tag to be used, i.e. a string to identify this evaluation inside the cache storage (optional)

  • plot (bool) – whether to plot the confusion matrix after finishing the test or not (default: True)

  • metric (str) – the evaluation metric to return, options are: ‘accuracy’, ‘f1-score’, ‘precision’, or ‘recall’ When working with multi-label classification problems, two more options are allowed: ‘hamming-loss’ and ‘exact-match’. Note: exact match will produce the same result than ‘accuracy’. (default: ‘accuracy’, or ‘hamming-loss’ for multi-label case).

  • metric_target (str) – the target we aim at measuring with the given metric. Options are: ‘macro avg’, ‘micro avg’, ‘weighted avg’ or a category label (default ‘macro avg’).

  • cache (bool) – whether to use cached values or not. Setting cache=False forces to completely perform the evaluation ignoring cached values (default: True).

Returns

the given metric value, by default, the obtained accuracy.

Return type

float

Raises

InvalidCategoryError, EmptyModelError, ValueError, KeyError

static plot(html_path='./', open_browser=True)

Open up an interactive 3D plot with the obtained results.

This 3D plot is opened up in the web browser and shows the results obtained from all the performed evaluations up to date. In addition, before showing the plot in the browser, this method also creates a portable HTML file containing the 3D plot.

Parameters
  • html_path (str) – the path in which to store the portable HTML file (default: ‘./’)

  • open_browser (bool) – whether to open the HTML in the browser or not (default: True)

Raises

ValueError

static remove(s=None, l=None, p=None, a=None, method=None, def_cat=None, tag=None, simulate=False)

Remove evaluation results from the cache storage.

If not arguments are given, this method will remove everything, and thus it will perform exactly like the clear_chache method. However, when arguments are given, only values matching that argument value will be removed. For example:

>>> # remove evaluation results using
>>> # s=.5, l=1, and p=1 hyperparameter values:
>>> Evaluation.remove(s=.5, l=1, p=1)
>>>
>>> # remove all 4-fold cross validation evaluations:
>>> Evaluation.remove(method="4-fold")

Besides, this method returns the number of items that were removed. If the argument simulate is set to True, items won’t be removed and only the number of items to be removed will be returned. For example:

>>> c, _ = Evaluation.remove(s=.45, method="test", simulate=True)
>>> if input("%d items will be removed, proceed? (y/n)") == 'y':
>>>     Evaluation.remove(s=.45, method="test")

Here is the full list of arguments that can be used to select what to be permanently removed from the cache storage:

Parameters
  • s (float) – a value for the s hyperparameter (optional).

  • l (float) – a value for the l hyperparameter (optional).

  • p (float) – a value for the p hyperparameter (optional).

  • a (float) – a value for the a hyperparameter (optional).

  • method (str) – an evaluation method, options are: ‘test’, ‘K-fold’, where K is a positive integer (optional).

  • def_cat (str) – a default category used in the evaluations, options are: ‘most-probable’, ‘unknown’ or a category label (optional).

  • metric – an evaluation metric, options are: ‘accuracy’, ‘f1-score’, ‘precision’, and ‘recall’ (optional).

  • tag (str) – a cache tag (optional).

  • simulate (bool) – whether to simulate the removal or not (default: False)

Returns

(number of items removed, details)

Return type

tuple

Raises

ValueError, TypeError

static set_classifier(clf)

Set the classifier to be evaluated.

Parameters

clf (SS3) – the classifier

static show_best(tag=None, method=None, def_cat=None, metric=None, avg=None)

Print information regarding the best obtained values according to all the metrics.

The information showed can be filtered out using any of the following arguments:

Parameters
  • tag (str) – a cache tag (optional).

  • method (str) – an evaluation method, options are: ‘test’, ‘K-fold’, where K is a positive integer (optional).

  • def_cat (str) – a default category used in the evaluations, options are: ‘most-probable’, ‘unknown’ or a category label (optional).

  • metric (str) –

    an evaluation metric, options are: ‘accuracy’, ‘f1-score’, ‘precision’, and ‘recall’. In addition, In multi-label

    classification also ‘hamming-loss’ and ‘exact-match’ (optional).

  • avg (str) – an averaging method, options are: ‘macro avg’, ‘micro avg’, and ‘weighted avg’ (optional).

Raises

ValueError

static test(clf, x_test, y_test, def_cat='most-probable', prep=True, tag=None, plot=True, metric=None, metric_target='macro avg', cache=True)

Test the model using the given test set.

Examples:

>>> from pyss3.util import Evaluation
>>> ...
>>> acc = Evaluation.test(clf, x_test, y_test)
>>> print("Accuracy:", acc)
>>> ...
>>> # this line won't perform the test again, it will retrieve, from the cache storage,
>>> # the f1-score value computed in previous test.
>>> f1 = Evaluation.test(clf, x_test, y_test, metric="f1-score")
>>> print("F1 score:", f1)
Parameters
  • clf (SS3) – the classifier to be evaluated.

  • x_test (list (of str)) – the test set documents, i.e, the list of documents to be classified

  • y_test (list (of str) or list (of list of str)) – the test set with category labels, i.e, the list of document labels

  • def_cat (str) – default category to be assigned when SS3 is not able to classify a document. Options are ‘most-probable’, ‘unknown’ or a given category name. (default: ‘most-probable’)

  • prep (bool) – enables the default input preprocessing when classifying (default: True)

  • tag (str) – the cache tag to be used, i.e. a string to identify this evaluation inside the cache storage (optional)

  • plot (bool) – whether to plot the confusion matrix after finishing the test or not (default: True)

  • metric (str) – the evaluation metric to return, options are: ‘accuracy’, ‘f1-score’, ‘precision’, or ‘recall’ When working with multi-label classification problems, two more options are allowed: ‘hamming-loss’ and ‘exact-match’. Note: exact match will produce the same result than ‘accuracy’. (default: ‘accuracy’, or ‘hamming-loss’ for multi-label case).

  • metric_target (str) – the target we aim at measuring with the given metric. Options are: ‘macro avg’, ‘micro avg’, ‘weighted avg’ or a category label (default ‘macro avg’).

  • cache (bool) – whether to use cached values or not. Setting cache=False forces to completely perform the evaluation ignoring cached values (default: True).

Returns

the given metric value, by default, the obtained accuracy.

Return type

float

Raises

EmptyModelError, KeyError, ValueError

class pyss3.util.Preproc

Bases: object

A helper class with methods for pre-processing documents.

static clean_and_ready(text, dots=True, normalize=True, min_len=1)

Clean and prepare the text.

class pyss3.util.Print

Bases: object

Helper class to handle print functionalities.

static error(msg='', raises=None, offset=0, decorator=True)

Print an error.

Parameters
  • msg (str) – the message to show

  • raises (Exception) – the exception to be raised after showing the message

  • offset (int) – shift the message to the right (offset characters)

  • decorator (bool) – if True, use error message decoretor

static get_verbosity()

Return the verbosity level.

  • 0 (quiet): do not output any message (only error messages)

  • 1 (normal): default behavior, display only warning messages and progress bars

  • 2 (verbose): display also the informative non-essential messages

Returns

the verbosity level

Return type

int

static info(msg='', newln=True, offset=0, decorator=True, force_show=False)

Print an info message.

Parameters
  • msg (str) – the message to show

  • newln (bool) – use new line after the message (default: True)

  • offset (int) – shift the message to the right (offset characters)

  • decorator (bool) – if True, use info message decoretor

  • force_show (bool) – if True, show message even when not in verbose mode

static is_quiet()

Check if the current verbosity level is quiet.

static is_verbose()

Check if the current verbosity level is verbose.

static set_decorator_error(start, end=None)

Set error messages decorator.

Parameters
  • start (str) – messages preffix

  • end (str) – messages suffix

static set_decorator_info(start, end=None)

Set info messages decorator.

Parameters
  • start (str) – messages preffix

  • end (str) – messages suffix

static set_decorator_warn(start, end=None)

Set warning messages decorator.

Parameters
  • start (str) – messages preffix

  • end (str) – messages suffix

static set_verbosity(level)

Set the verbosity level.

  • 0 (quiet): do not output any message (only error messages)

  • 1 (normal): default behavior, display only warning messages and progress bars

  • 2 (verbose): display also the informative non-essential messages

The following built-in constants can also be used to refer to these 3 values: VERBOSITY.QUIET, VERBOSITY.NORMAL, and VERBOSITY.VERBOSE, respectively.

For example, if you want PySS3 to hide everything, even progress bars, you could do:

>>> from pyss3.util import Print, VERBOSITY
...
>>> Print.set_verbosity(VERBOSITY.QUIET)  # or, equivalently, Print.set_verbosity(0)
...
>>> # here's the rest of your code :D
Parameters

level (int) – the verbosity level

static show(msg='', newln=True, offset=0, force_show=False)

Print a message.

Parameters
  • msg (str) – the message to show

  • newln (bool) – use new line after the message (default: True)

  • offset (int) – shift the message to the right (offset characters)

style

alias of Style

static verbosity_region_begin(level, force=False)

Indicate that a region with different verbosity begins.

When the region ends by calling verbosity_region_end, the previous verbosity will be restored.

Example:

>>> from pyss3.util import Print,VERBOSITY
...
>>> Print.verbosity_region_begin(VERBOSITY.QUIET)
>>> # inside this region (from now on), verbosity will be 'quiet'
...
>>> Print.verbosity_region_end()
>>> # the verbosity level is restored to what it was before entering the region
Parameters

level (int) – the verbosity level for this region (see set_verbosity documentation for valid values)

static verbosity_region_end()

Indicate that a region with different verbosity ends.

The verbosity will be restored to the value it had before beginning this region with verbosity_region_begin.

Example:

>>> from pyss3.util import Print,VERBOSITY
...
>>> Print.verbosity_region_begin(VERBOSITY.VERBOSE)
>>> # inside this region (from now on), verbosity will be 'verbose'
...
>>> Print.verbosity_region_end()
>>> # the verbosity level is restored to what it was before entering the region
static warn(msg='', newln=True, raises=None, offset=0, decorator=True)

Print a warning message.

Parameters
  • msg (str) – the message to show

  • newln (bool) – use new line after the message (default: True)

  • raises (Exception) – the exception to be raised after showing the message

  • offset (int) – shift the message to the right (offset characters)

  • decorator (bool) – if True, use warning message decoretor

class pyss3.util.RecursiveDefaultDict

Bases: dict

A dict whose default value is a dict.

class pyss3.util.Style

Bases: object

Helper class to handle print styles.

static blue(text)

Apply ‘blue’ style to text.

static bold(text)

Apply bold style to text.

static fail(text)

Apply the ‘fail’ style to text.

static green(text)

Apply ‘green’ style to text.

static header(text)

Apply ‘header’ style to text.

static ubold(text)

Apply underline and bold style to text.

static underline(text)

Underline text.

static warning(text)

Apply the ‘warning’ style to text.

class pyss3.util.VERBOSITY

Bases: object

verbosity “enum” constants.

NORMAL = 1
QUIET = 0
VERBOSE = 2
pyss3.util.is_a_collection(o)

Return True when the object o is a collection.

pyss3.util.list_by_force(v)

Convert any non-iterable object into a list.

pyss3.util.membership_matrix(clf, y_data, labels=True, show_pbar=True)

Transform a list of (multiple) labels into a “membership matrix”.

The membership matrix consists converting each list of category labels (i.e., each y in y_data) into a vector in which there’s a fixed position associated to each learned category, having the value 1 for each label in y, and 0 otherwise.

When working with multi-label classification problems, this representation enables measuring the performance using common evaluation metrics such as Hamming loss, exact match ratio, accuracy, precision, recall, F1, etc.

For instance, suppose y_data = [[], ['labelA'], ['labelB'], ['labelA', 'labelC']] and that the classifier clf has been trained on 3 categories whose labels are ‘labelA’, ‘labelB’, and ‘labelC’, then, we would have that:

>>> membership_matrix(clf, [[], ['labelA'], ['labelB'], ['labelA', 'labelC']])

returns the following membership matrix:

>>> [[0, 0, 0],  # []
>>>  [1, 0, 0],  # ['labelA']
>>>  [0, 1, 0],  # ['labelB']
>>>  [1, 0, 1]]  # ['labelA', 'labelC']
Parameters
  • clf (SS3) – the trained classifier

  • y_data (list of list of str) – the list of document labels

  • labels (bool) – whether the y_data list contains category labels or category indexes (default: True)

  • show_pbar (bool) – whether to show the progress bar or not (default: True)

Returns

a (sparse) matrix in which each row is the membership vector of each element (labels) in y_data.

Return type

scipy.sparse.lil.lil_matrix

Raises

ValueError

pyss3.util.round_fix(v, precision=4)

Round the number v (used to keep the results history file small).