Instructions used in STS.news.sr annotation

This document contains the English version of the instructions used in the annotation of the Serbian STS News Corpus (STS.news.sr). The instructions are also available in Serbian.

The general guidelines are identical to the ones used in the annotations for the SemEval STS shared task datasets (2012-2017). However, this document contains different examples for each score value. Furthermore, instead of one example per score value, there are three examples per score - this was found to improve task comprehension and annotation quality. The examples were taken from the MSRPar (2012) and the Headlines (2013-2016) portions of the annotated SemEval STS corpora in English, and were then professionally translated into Serbian.

Compare the Meaning of Two Statements

Instructions

Two statements can mean the same thing even if they use very different words and phrases. Conversely, two statements that are superficially very similar in their word choice, phrasing and overall composition can have very different meanings.

Your job is to compare two statements and decide the type of relationship that holds between their underlying meanings or messages (i.e. what they say or refer to in the world).

To do this task successfully, picture what is being described and contrast exactly what is conveyed by one statement versus what is being conveyed by the other.

Do the statements refer to the exact same person, action, event, idea, or thing? Or, are they similar but differ according to either large or small details?

Tips:

• Be precise in your assignments and try to avoid overusing any one of the category labels (e.g. don’t just label most of the pairs as “mostly equivalent” or “roughly equivalent”).
• Be careful of subtle differences between the pairs that have an important impact on what is being said or described.
• Ignore grammatical errors and awkward wordings within the statements as long as they do not obscure what a statement is supposed to convey.

Scores and examples

• (5) The two sentences are completely equivalent, as they mean the same thing.

  • Bush plans to meet with Israeli Prime Minister Ariel Sharon and the new Palestinian prime minister, Mahmoud Abbas, in the Jordanian port of Aqaba on Wednesday.

    On Wednesday next week, Mr Bush will meet Israeli Prime Minister Ariel Sharon and new Palestinian leader Mahmoud Abbas in Aqaba, Jordan.

  • The upcoming second-quarter earnings season will be particularly important in offering investors guidance, they say.

    They say second-quarter earnings reports will be key in giving investors that guidance.

  • He later learned that the incident was caused by the Concorde’s sonic boom.

    He later found out the alarming incident had been caused by Concorde’s powerful sonic boom.

• (4) The two sentences are mostly equivalent, but some unimportant details differ.

  • Ricky Clemons’ brief, troubled Missouri basketball career is over.

    Missouri kicked Ricky Clemons off its team, ending his troubled career there.

  • Several states and the federal government later passed similar or more strict bans.

    Following California’s lead, several states and the federal government passed similar or tougher bans.

  • “It appears from our initial report that this was a textbook landing considering the circumstances,” Burke said.

    Said Mr. Burke: “It was a textbook landing considering the circumstances.”

• (3) The two sentences are roughly equivalent, but some important information differs/missing.

  • It was the third time in four years that wildfires forced the park to close.

    For the third time in four years wildfires closed Mesa Verde National Park, the country’s only park dedicated to ancient ruins.

  • Coupling, an American version of a hit British comedy, will get the valuable Thursday 9:30 p.m. time slot.

    Coupling, an American version of a hit British comedy, will couple with Friends on Thursdays.

  • Selenski descended down the wall and used the mattress to climb over razor wire.

    Selenski used the mattress to scale a 10-foot razor wire, Fischi said.

• (2) The two sentences are not equivalent, but share some details.

  • The company didn’t detail the costs of the replacement and repairs.

    But company officials expect the costs of the replacement work to run into the millions of dollars.

  • Preliminary reports were that the men were not seen together at the airport, sources said.

    The men had Pakistani passports and reportedly were seen together at the airport earlier in the evening, law enforcement sources said.

  • Chinese icebreaker changes course towards suspicious objects.

    Chinese search plane finds ‘suspicious objects’.

• (1) The two sentences are not equivalent, but are on the same topic.

  • Syria urges refugees to return.

    Syria’s tenuous truce takes hold.

  • Islamist Morsi Wins Egypt’s Presidential Election.

    Al-Shater to run in presidential election in Egypt.

  • Japan PM calls for quicker tsunami waste processing.

    Japan marks one year since quake and tsunami disaster.

• (0) The two sentences are on different topics.

  • Moderate tremor in the Central Mediterranean.

    Iran: Moderate candidate wins presidential vote.

  • 1 person killed in sectarian clashes in Lebanon.

    180 killed in Iran earthquake.

  • Malaysia Airlines plane crashes on Ukraine-Russia border.

    US: Russia Massing Troops on Ukraine Border.