Tasks

Summarization Task

This MultiLing task aims to evaluate the application of (partially or fully) language-independent summarization algorithms on a variety of languages. Each system participating in the task will be called to provide summaries for a range of different languages, based on corresponding corpora.  In the MultiLing Pilot of 2011 the languages used were 7, while we aim for at least 8 languages in the MultiLing 2013 workshop. Participating systems will be required to apply their methods on a minimum of two languages. Evaluation will favour systems that apply their methods in more languages.

The MultiLing task requires to generate a single, fluent, representative summary from a set of documents describing an event sequence. The language of the document set will be within a given range of languages and all documents in a set share the same language. The output summary should be of the same language as its source documents. The output summary should be 250 words at most.

Evaluation Task

This task aims to examine how well automated systems can evaluate summaries from different languages. This task takes as input the summaries generated from automatic systems and humans in the Sumamrization Task. The output should be a grading of the summaries. Ideally, we would want the automatic evaluation to maximally correlate to human judgement.

Pilot: Multilingual single document summarization

This pilot aims to measure the ability of automated systems to apply single document summarization, in the context of Wikipedia texts. Given a single encyclopedic entry, possibly with several sections/subsection, describing a specific subject, the systems will be requested to provide a summary covering the main points of the entry (similarly to the lead section of a Wikipedia page). The corpus will consist of (non-parallel) documents in over 40 languages. Participating systems will be required to apply their methods to a minimum of two languages. Evaluation will favor systems that apply their methods to more languages.

The pilot corpus will be based on selected texts from Wikipedia. Details will follow in the MultiLing 2013 website (http://multiling.iit.demokritos.gr/)