This file contains all the dataset files related to the MultiLing 2011 Pilot.This includes:

  • source texts
  • human summaries
  • system summaries
  • evaluation data (human and automatic)

This is a packaging of the existing individual packages, found as previously uploaded files.

We would appreciate it greatly, if you filled in the "Using the MultiLing Pilot 2011 Dataset" form, before downloading. You can find more information in the README files in the archive, as well as in the following paper:

George Giannakopoulos, Mahmoud El-Haj, Benoit Favre, Marina Litvak, Josef Steinberger, & Vasudeva Varma. (2011). TAC2011 MultiLing Pilot Overview. TAC 2011 Workshop. Presented at the TAC 2011, Gaithersburg, MD, U.S.A.
 
Creative Commons License
MultiLing Pilot 2011 Dataset by MultiLing Community is licensed under a Creative Commons Attribution 3.0 Unported License.
Based on a work at multiling.iit.demokritos.gr.
Permissions beyond the scope of this license may be available at http://multiling.iit.demokritos.gr.

Comments

  • John M. Conroy 1949 days ago

    Hi George,

     Can you post the options you used for ROUGE?  For example, was it

    using ROUGE1.5.5 with the options

    -x -u -d -m -n 2 -2 4 -c 95 -r 1000 -f A  -t 0 -a -l 250

    Also did you have to modify ROUGE to work with the other lanuages?

    Best regards,

    John

  • George Giannakopoulos (Admin) 1947 days ago

    Hi John.

    The command line we used was:

    ROUGE-1.5.5.pl -a -x -2 4 -u -c 95 -e <ROUGEDIR> -r 1000 -n 2 -f A -p 0.5 -t 0 -d <SETTINGS.XML File>

    We did not modify ROUGE to work with the other languages. This may have caused a problem, as we later saw (because things may change with the tokenization in some languages). However, whether the effect of this different tokenization was significant is unclear. Marina Litvak later tried to apply some changes (to apply to the Hebrew language) and may have more information.

  • Marina Litvak 1882 days ago

    Hi John, George,

     

    Yes, we also discussed it with Michael Elhadad - there may be a significant effect to ROUGE results depending on whether it is adapted to a particular language or not. I think Michael can explain ut better - I used adapted (by his student) to Hebrew version of Rouge. Then, I also adapted it to Arabic in the same manner.