The Serbian Semantic Textual Similarity News Corpus

View the Project on GitHub vukbatanovic/

Instructions used in annotation

This document contains the English version of the instructions used in the annotation of the Serbian STS News Corpus ( The instructions are also available in Serbian.

The general guidelines are identical to the ones used in the annotations for the SemEval STS shared task datasets (2012-2017). However, this document contains different examples for each score value. Furthermore, instead of one example per score value, there are three examples per score - this was found to improve task comprehension and annotation quality. The examples were taken from the MSRPar (2012) and the Headlines (2013-2016) portions of the annotated SemEval STS corpora in English, and were then professionally translated into Serbian.

Compare the Meaning of Two Statements


Two statements can mean the same thing even if they use very different words and phrases. Conversely, two statements that are superficially very similar in their word choice, phrasing and overall composition can have very different meanings.

Your job is to compare two statements and decide the type of relationship that holds between their underlying meanings or messages (i.e. what they say or refer to in the world).

To do this task successfully, picture what is being described and contrast exactly what is conveyed by one statement versus what is being conveyed by the other.

Do the statements refer to the exact same person, action, event, idea, or thing? Or, are they similar but differ according to either large or small details?


Scores and examples