Evaluation on Semantic Search

We need your help with a research project on semantic search algorithms!

We have developed new approaches, which of course have to be evaluated - and this is where we need you! Please assess the quality of our algorithms by telling us which documents are relevant to a search query, and by comparing different rankings. The evaluation can be done anywhere, anytime in the next two weeks, at http://s16a.org/percy. You can also stop or pause it whenever you want.

You will not only help us, but any future research in this area, because we intend to publish a gold standard based on your judgments. Helping fellow students and researchers should be a good enough motivation, but if it is not ... maybe the chance to win an Amazon gift card is?

Thank you very much - we really appreciate your help!

UPDATE 09.07.2015:

Thank You all for participating!!

From the data provided by You we were able to create a new evaluation dataset (download below) to measure the effectivity of some of our retrieval models for semantically annotated documents against a baseline. The results can be found in the following table:

Method MAP NDCG MAP@10 NDCG@10 Recipr. Rank Prec@1
Text (baseline) 0.696 0.848 0.555 0.743 0.960 0.943
Concept + Text 0.736 0.872 0.573 0.761 0.979 0.971
Connectedness (only) 0.711 0.862 0.567 0.752 0.981 0.971
Connectedness (with tf) 0.749 0.874 0.583 0.766 0.979 0.943
Taxonomic (no similarity) 0.766 0.875 0.603 0.758 0.961 0.943
Taxonomic (Resnik-Zhou) 0.774 0.880 0.624 0.773 0.974 0.974

The dataset with in NIF/RDF format can be downloaded here:

Mai 23, 2016: version 1.2 (added article publish dates) wes2015-dataset-nif-1.2.rdf (ca. 15MB)
Feb 4, 2016: version 1.1 wes2015-dataset-nif-1.1.rdf (ca. 15MB)

The dataset contains:

  • 331 semantically annotated documents from the yovisto blog on history in science, technology, and art
  • 35 semantically annotated queries inspired by the blog's query logs.
  • relevance assessments between queries and documents, generated from the user study above.

Download the Paper form ISWC2015 Workshop NLP & DBpedia: Joerg Waitelonis, Claudia Exeler and Harald Sack. Linked Data Enabled Generalized Vector Space Model To Improve Document Retrieval

How to cite:

Jörg Waitelonis, Claudia Exeler, and Harald Sack. Linked Data enabled Generalized Vector Space Model to improve document retrieval. In Proceedings of NLP & DBpedia 2015 workshop in conjunction with 14th International Semantic Web Conference (ISWC2015), CEUR Workshop Proceedings, Vol1581, pp 33-44, 2015.


Author = {J{\"o}rg Waitelonis and Claudia Exeler and Harald Sack},
Booktitle = {NLP \& DBpedia 2015 workshop at 14th Int. Semantic Web Conf.},
Title = {{Linked Data Enabled Generalized Vector Space Model to Improve Document Retrieval}},
Year = {2015},
Pages = {33-44},
Volume = {1581},
Publisher = {CEUR-WS}

Creative Commons Lizenzvertrag
Dieses Werk ist lizenziert unter einer Creative Commons Namensnennung - Nicht-kommerziell - Weitergabe unter gleichen Bedingungen 4.0 International Lizenz.