Call for participation in the community tasks

The following tasks will run in the MultiLing community along the 2019 workshop. Participation is not restricted by the key dates of the workshop (i.e. participants can submit evaluations before, but also beyond MultiLing 2019) -- a summary of notable submissions however may be selected and outlined in the workshop.

Participants should fill this short form to register to the task of interest. Registration will result in subscription to the task's mailing list. To register for multiple tasks, repeat the above registration process for each task of interest. 

Task Descriptions:

* Headline generation
The objective of the Headline Generation (HG) task is to explore some of the challenges highlighted by current state of the art approaches on creating informative headlines to news articles: non-descriptive headlines, out-of-domain training data, and generating headlines from long documents which are not well represented by the head heuristic. We propose to make available a large set of training data for headline generation, and create evaluation conditions which emphasize those challenges. We will also rerun the task in DUC 2004 conditions in order to create comparable results.

* Summary evaluation
This task aims to examine how well automated systems can evaluate summaries from different languages. This task takes as input the summaries generated from automatic systems and humans in the Summarization Tasks of MultiLing 2015, but also in the Single document summarization tasks of 2015 and 2017 (when the latter is completed). The output should be a grading of the summaries. Ideally, we would want the automatic evaluation to maximally correlate to human judgement, thus the evaluation will be based on correlation measurement between estimated grades and human grades.

* Financial narrative summarization
This task aims to demonstrate the value and challenges of applying summarization to financial text written in English, usually referred to as financial narrative disclosures. The participants will be asked to provide structured summaries, based on real-world, publicly available financial annual reports of UK firms by extracting information from different key sections. Generated summaries should reflect the analysis and assessment of the financial trend of the business over the past year, as provided by annual reports.
The manual evaluation will be conducted by experts. Automatic evaluation will be carried through established summary evaluation measures, using as gold-standard models executive summaries provided by humans.

Important dates:
Participant registration deadline: 20 May 2019
Release of test data: 4 July 2019
Submissions deadline:  11 July 2019 21 July 2019
Evaluations announced: 28 July 2019

Submission process:
Community task report papers should be submitted via the START submission system,  where the 'Submission Type' field should be set to 'Community Task Report'. Test result scores should be sent via email to the respective task organizers -- refer to the webpage of each task for details.

Additionally, please consider submitting your work as a full paper on the regular Multiling 2019 workshop.