University of Mannheim
member: Heiko Paulheim
type: Organization

CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
creator: Anca Dumitrache
maker: Anca Dumitrache
maker: Lora Aroyo
type: InProceedings
creator: Lora Aroyo
maker: Chris Welty
abstract: A widespread use of linked data for information extraction is distant supervision, in which relation tuples from a data source are found in sentences in a text corpus, and those sentences are treated as training data for relation extraction systems. Distant supervision is a cheap way to acquire training data, but that data can be quite noisy, which limits the performance of a system trained with it. Human annotators can be used to clean the data, but in some domains, such as medical NLP, it is widely believed that only medical experts can do this reliably. We have been investigating the use of crowdsourcing as an affordable alternative to using experts to clean noisy data, and have found that with the proper analysis, crowds can rival and even out-perform the precision and recall of experts, at a much lower cost. We have further found that the crowd, by virtue of its diversity, can help us find evidence of ambiguous sentences that are difficult to classify, and we have hypothesized that such sentences are likely just as difficult for machines to classify. In this paper we outline CrowdTruth, a previously presented method for scoring ambiguous sentences that suggests that existing modes of truth are inadequate, and we present for the first time a set of weighted metrics for evaluating the performance of experts, the crowd, and a trained classifier in light of ambiguity. We show that our theory of truth and our metrics are a more powerful way to evaluate NLP performance over traditional unweighted metrics like precision and recall, because they allow us to account for the rather obvious fact that some sentences express the target relations more clearly than others.
author list: authorList
is presented: 817b4icma6mkao4j5jjf2hhu2o@google.com 1
month: 2015
creator: Chris Welty
year: 2015
is part of: proceedings

ecv20nbc0jsq0uee9bnjb197mk@google.com 3
dtstart: 2015-10-12T14:00:00Z
dtend: 2015-10-12T14:30:00Z
created: 2015-10-02T10:01:49Z
type: Vevent
location: Room RBC 271
transp: OPAQUE
presents: A Topic-Sensitive Model for Salient Entity Linking
status: CONFIRMED
description: Lei Zhang, Cong Liu and Achim Rettinger.
sequence: 0
last modified: 2015-10-02T12:00:25Z
summary: Research | A Topic-Sensitive Model for Salient Entity Linking
dtstamp: 2015-10-02T12:03:03Z
is sub event of: ri9oe6lcndmbii7j2mehclqu2o@google.com 2

main conference
prodid: -//Google Inc//Google Calendar 70.9054//EN
component: 817b4icma6mkao4j5jjf2hhu2o@google.com 1
component: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
component: 6qn80t9g85uvdoooftrhjjbbv4@google.com 5
calscale: GREGORIAN
version: 2.0
component: 2l7a5duv5ina9b5a8fofftgjdk@google.com 7
component: rlofdvrkt3tmr48533jbsgqt48@google.com 4
type: Vcalendar
component: dq0d4i00ndh35hp6ssijocvqrc@google.com 6
component: 96a4bh66arosfhfed2uu76kt08@google.com 8
method: PUBLISH
component: ecv20nbc0jsq0uee9bnjb197mk@google.com 3

Anna Lisa Gentile
affiliation: University of Sheffield
last name: Gentile
first name: Anna Lisa
holds role: Chair
type: Person

Goodbye to True: Advancing Semantics beyond the Black and White
type: InProceedings
year: 2015
is part of: proceedings
author list: authorList
is presented: 2l7a5duv5ina9b5a8fofftgjdk@google.com 7
month: 2015
maker: Chris Welty
creator: Chris Welty
abstract: The set-theoretic notion of truth proposed by Tarski is the basis of most work in machine semantics and probably has its roots in the work and influence of Aristotle. We take it for granted that the world can be described, not in shades of grey, but in terms of statements and propositions that are either true or false - and it seems most of western science stands on the same principle. This assumption at the core of our training as scientists should be questioned, because it stands in direct opposition to our human experience. Is there any statement that can be made that can actually be reduced to true or false? Only, it seems, in the artificial human-created realms of mathematics, games, and logic. We have been investigating a different mode of truth, inspired by results in Crowdsourcing, which allows for a highly dimension notion of semantic interpretation that makes true and false look like a childish simplifying assumption.

Anca Dumitrache
holds role: Paper author
last name: Dumitrache
first name: Anca
made: CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
type: Person

6qn80t9g85uvdoooftrhjjbbv4@google.com 5
presents: Semantic Relation Composition in Large Scale Knowledge Bases
summary: Research | Semantic Relation Composition in Large Scale Knowledge Bases
location: Room RBC 271
sequence: 0
status: CONFIRMED
type: Vevent
last modified: 2015-10-02T12:00:41Z
created: 2015-10-02T10:03:18Z
dtend: 2015-10-12T16:00:00Z
dtstart: 2015-10-12T15:30:00Z
is sub event of: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
transp: OPAQUE
dtstamp: 2015-10-02T12:03:03Z
description: Kristian Kolthoff and Arnab Dutta.

Katja Hose
holds role: Program committee member
last name: Hose
first name: Katja
type: Person

Varish Mulwad
holds role: Program committee member
last name: Mulwad
first name: Varish
type: Person

Arnab Dutta
holds role: Paper author
last name: Dutta
first name: Arnab
made: Semantic Relation Composition in Large Scale Knowledge Bases
type: Person

Keynote speaker
held by: Chris Welty
is role at: The 3rd International Workshop on Linked Data for Information Extraction
type: Presenter

Pablo Mendes
holds role: Program committee member
last name: Mendes
first name: Pablo
type: Person

Tim Finin
holds role: Paper author
last name: Finin
first name: Tim
made: Topic Modeling for RDF Graphs
type: Person

Semantic Annotation of Quantitative Textual Content
creator: John Cuzzola
is presented: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
maker: John Cuzzola
abstract: Semantic annotation techniques provide the basis for linking textual content with concepts in well grounded knowledge bases. In spite of their many application areas, current semantic annotation systems have some limitations. One of the prominent limitations of such systems is that none of the existing semantic annotator systems are able to identify and disambiguate quantitative (numerical) content. In textual documents such as Web pages, specially technical contents, there are many quantitative information such as product specifications that need to be semantically qualified. In this paper, we propose an approach for annotating quantitative values in short textual content. In our approach, we identify numeric values in the text and link them to an existing property in a knowledge base. Based on this mapping, we are then able to find the concept that the property is associated with; whereby, identifying both the concept and the specific property of that concept that the numeric value belongs to. Our experiments show that our proposed approach is able to reach an accuracy of over 70% for semantically annotating quantitative content.
maker: Ebrahim Bagheri
month: 2015
is part of: proceedings
type: InProceedings
creator: Mehrnaz Ghashghaei
maker: Mehrnaz Ghashghaei
maker: Ali Ghorbani
creator: Ali Ghorbani
author list: authorList
year: 2015
creator: Ebrahim Bagheri

University of Bari
member: Claudia D'Amato
type: Organization

Andrea Giovanni Nuzzolese
holds role: Program committee member
last name: Nuzzolese
first name: Andrea Giovanni
type: Person

Robert Meusel
holds role: Paper author
last name: Meusel
first name: Robert
made: Creating Large-scale Training and Test Corpora for Extracting Structured Data from the Web
type: Person

Heiko Paulheim
holds role: Chair
type: Person
holds role: Program committee member
holds role: Paper author
affiliation: University of Mannheim
made: Creating Large-scale Training and Test Corpora for Extracting Structured Data from the Web
last name: Paulheim
first name: Heiko

Victoria Uren
holds role: Program committee member
last name: Uren
first name: Victoria
type: Person

Chair
held by: Ziqi Zhang
held by: Heiko Paulheim
held by: Claudia D'Amato
held by: Anna Lisa Gentile
is role at: The 3rd International Workshop on Linked Data for Information Extraction

Christopher Brewster
holds role: Program committee member
last name: Brewster
first name: Christopher
type: Person

John Cuzzola
holds role: Paper author
last name: Cuzzola
first name: John
made: Semantic Annotation of Quantitative Textual Content
type: Person

Brian Davis
holds role: Program committee member
last name: Davis
first name: Brian
type: Person

Invited Talk: Chris Welty
dtend: 2015-06-03T10:30:00
dtstart: 2015-06-03T09:30:00
description: Goodbye to True: Advancing Semantics beyond the Black and White
summary: Invited Talk: Chris Welty
is sub event of: http://data.semanticweb.org/conference/ld4ie/2015
type: TalkEvent

Disheng Qiu
holds role: Program committee member
last name: Qiu
first name: Disheng
type: Person

Will Radford
holds role: Program committee member
last name: Radford
first name: Will
type: Person

dq0d4i00ndh35hp6ssijocvqrc@google.com 6
description: Heiko Paulheim.
sequence: 0
summary: Plenary | Creating Large-scale Training and Test Corpora for Extracting Structured Data from the Web
type: Vevent
dtstamp: 2015-10-02T12:03:03Z
transp: OPAQUE
created: 2015-10-02T10:04:01Z
location: Room RBC 271
dtstart: 2015-10-12T16:00:00Z
presents: Creating Large-scale Training and Test Corpora for Extracting Structured Data from the Web
last modified: 2015-10-02T12:00:56Z
dtend: 2015-10-12T16:30:00Z
is sub event of: 96a4bh66arosfhfed2uu76kt08@google.com 8
status: CONFIRMED

Petr Knoth
holds role: Program committee member
last name: Knoth
first name: Petr
type: Person

Paper author
held by: Lora Aroyo
held by: Mehrnaz Ghashghaei
held by: Achim Rettinger
held by: Jennifer Sleeman
is role at: The 3rd International Workshop on Linked Data for Information Extraction
held by: Lei Zhang
held by: Tim Finin
held by: Anupam Joshi
held by: Kristian Kolthoff
held by: Anca Dumitrache
held by: Chris Welty
held by: Arnab Dutta
held by: Cong Liu
type: Author
held by: Robert Meusel
held by: Heiko Paulheim
held by: Ali Ghorbani
held by: John Cuzzola
held by: Ebrahim Bagheri

Giuseppe Rizzo
holds role: Program committee member
last name: Rizzo
first name: Giuseppe
type: Person

Lei Zhang
holds role: Paper author
last name: Zhang
first name: Lei
made: A Topic-Sensitive Model for Salient Entity Linking
type: Person

Creating Large-scale Training and Test Corpora for Extracting Structured Data from the Web
maker: Robert Meusel
abstract: For making the web of linked data grow, information extraction methods are a good alternative for manual dataset curation, since there is an abundance of semi-structured and unstructured information which can be harvested that way. At the same time, existing structured data sets can be used for training and evaluating such information extraction systems. In this paper, we introduce a method for creating training and test corpora from websites annotated with structured data. Using different classes in schema.org and websites annotated with Microdata, we show how training and test data can be curated at large scale and across various domains. Furthermore, we discuss how negative examples can be generated as well as open challenges and future directs for this kind of training data curation.
month: 2015
is part of: proceedings
creator: Robert Meusel
author list: authorList
maker: Heiko Paulheim
is presented: dq0d4i00ndh35hp6ssijocvqrc@google.com 6
creator: Heiko Paulheim
type: InProceedings
year: 2015

Semantic Relation Composition in Large Scale Knowledge Bases
type: InProceedings
is presented: 6qn80t9g85uvdoooftrhjjbbv4@google.com 5
abstract: Semantic relation composition is a generalized approach for finding conjunctive relation paths in a knowledge base (KB). In semantic web, direct and inverse relationships between entities provide us with ample of explicit knowledge. But there is a plethora of implicit knowledge beyond these direct paths. Consider a knowledge graph, we can achieve deeper insights about a particular entity if we consider the information shared by its neighboring entities via its adjacent relation paths of arbitrary lengths. In this paper, we devise a technique to automatically discover semantically enriched conjunctive relations in a KB. Our approach is generalized for any KB and requires no additional parameter tuning. Particularly, we employ classical rule mining techniques to perform relation composition on knowledge graphs to learn first order rules. We evaluate our proposed methodology on two state of the art information extraction systems, DBpedia and Yago with promising results in terms of generating high precision rules. Furthermore, we make the rules publicly available for community usage.
creator: Arnab Dutta
month: 2015
year: 2015
author list: authorList
creator: Kristian Kolthoff
maker: Arnab Dutta
is part of: proceedings
maker: Kristian Kolthoff

Rabeeh Ayaz Abbasi
holds role: Program committee member
last name: Abbasi
first name: Rabeeh Ayaz
type: Person

A Topic-Sensitive Model for Salient Entity Linking
maker: Cong Liu
maker: Achim Rettinger
year: 2015
maker: Lei Zhang
creator: Lei Zhang
month: 2015
abstract: In recent years, the amount of entities in large knowledge bases available on the Web has been increasing rapidly. Such entities can be used to bridge textual data with knowledge bases and thus help with many tasks, such as text understanding, word sense disambiguation and information retrieval. The key issue is to link the entity mentions in documents with the corresponding entities in knowledge bases, referred to as entity linking. In addition, for many entity-centric applications, entity salience for a document has become a very important factor. This raises an impending need to identify a set of salient entities that are central to the input document. In this paper, we introduce a new task of salient entity linking and propose a graph-based disambiguation solution, which integrates several features, especially a topic-sensitive model based on Wikipedia categories. Experimental results show that our method significantly outperforms the state-of-the-art entity linking methods in terms of precision, recall and F-measure.
type: InProceedings
is part of: proceedings
creator: Achim Rettinger
is presented: ecv20nbc0jsq0uee9bnjb197mk@google.com 3
creator: Cong Liu
author list: authorList

Ali Ghorbani
holds role: Paper author
last name: Ghorbani
first name: Ali
made: Semantic Annotation of Quantitative Textual Content
type: Person

2l7a5duv5ina9b5a8fofftgjdk@google.com 7
dtstamp: 2015-10-02T12:03:03Z
dtend: 2015-10-12T19:15:00Z
description: Goodbye to True: Advancing semantics beyond the black and white. Chris Welty, Google Research
is sub event of: 96a4bh66arosfhfed2uu76kt08@google.com 8
location: Room RBC 271
presents: Goodbye to True: Advancing Semantics beyond the Black and White
transp: OPAQUE
summary: Plenary | Goodbye to True: Advancing Semantics beyond the Black and White
type: Vevent
last modified: 2015-10-02T12:01:17Z
sequence: 0
created: 2015-10-02T10:07:36Z
dtstart: 2015-10-12T18:00:00Z
status: CONFIRMED

Christina Feilmayr
holds role: Program committee member
last name: Feilmayr
first name: Christina
type: Person

817b4icma6mkao4j5jjf2hhu2o@google.com 1
status: CONFIRMED
presents: CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
is sub event of: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
description: Anca Dumitrache, Lora Aroyo and Chris Welty.
dtstart: 2015-10-12T13:00:00Z
transp: OPAQUE
created: 2015-10-02T09:46:55Z
dtstamp: 2015-10-02T12:03:03Z
summary: Research | CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
dtend: 2015-10-12T13:30:00Z
sequence: 1
type: Vevent
last modified: 2015-10-02T11:59:57Z
location: Room RBC 271

Ernesto William De Luca
holds role: Program committee member
last name: De Luca
first name: Ernesto William
type: Person

Lora Aroyo
holds role: Paper author
last name: Aroyo
first name: Lora
made: CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
type: Person

Anupam Joshi
holds role: Paper author
last name: Joshi
first name: Anupam
made: Topic Modeling for RDF Graphs
type: Person

Jennifer Sleeman
holds role: Paper author
last name: Sleeman
first name: Jennifer
made: Topic Modeling for RDF Graphs
type: Person

96a4bh66arosfhfed2uu76kt08@google.com 8
dtstart: 2015-10-12T19:15:00Z
status: CONFIRMED
description: Closing remarks and best paper Award announcment
location: Room RBC 271
created: 2015-10-02T10:08:17Z
transp: OPAQUE
is sub event of: The 3rd International Workshop on Linked Data for Information Extraction
last modified: 2015-10-02T12:01:23Z
is super event of: 2l7a5duv5ina9b5a8fofftgjdk@google.com 7
dtstamp: 2015-10-02T12:03:03Z
summary: Plenary | Closing and Awards
sequence: 1
is super event of: dq0d4i00ndh35hp6ssijocvqrc@google.com 6
dtend: 2015-10-12T19:30:00Z
type: Vevent

Ebrahim Bagheri
holds role: Paper author
last name: Bagheri
first name: Ebrahim
made: Semantic Annotation of Quantitative Textual Content
type: Person

Ziqi Zhang
affiliation: University of Sheffield
holds role: Chair
holds role: Program committee member
last name: Zhang
first name: Ziqi
type: Person

Topic Modeling for RDF Graphs
is presented: rlofdvrkt3tmr48533jbsgqt48@google.com 4
year: 2015
maker: Anupam Joshi
creator: Tim Finin
author list: authorList
month: 2015
maker: Jennifer Sleeman
abstract: Topic models are widely used to thematically describe a collection of text documents and have become an important technique for systems that measure document similarity for classification, clustering, segmentation, entity linking and more. While they have been applied to some non-text domains, their use for semi-structured graph data, such as RDF, has been less explored. We present a framework for applying topic modeling to RDF graph data and describe how it can be used in a number of linked data tasks. Since topic modeling builds abstract topics using the co-occurrence of document terms, sparse documents can be problematic, presenting challenges for RDF data. We outline techniques to overcome this problem and the results of experiments in using them. Finally, we show preliminary results of using Latent Dirichlet Allocation generative topic modeling for several linked data use cases.
creator: Jennifer Sleeman
is part of: proceedings
creator: Anupam Joshi
maker: Tim Finin
type: InProceedings

Volker Tresp
holds role: Program committee member
last name: Tresp
first name: Volker
type: Person

Pierpaolo Basile
holds role: Program committee member
last name: Basile
first name: Pierpaolo
type: Person

Program committee member
held by: Pablo Mendes
held by: Christina Feilmayr
held by: Disheng Qiu
held by: Andrea Giovanni Nuzzolese
held by: Martin Riedl
held by: Volker Tresp
held by: Varish Mulwad
held by: Victoria Uren
held by: Ziqi Zhang
held by: Payam Barnaghi
held by: Miriam Fernandez
held by: Craig Knoblock
held by: Will Radford
held by: Ernesto William De Luca
held by: Heiko Paulheim
is role at: The 3rd International Workshop on Linked Data for Information Extraction
type: ProgrammeCommitteeMember
held by: Christopher Brewster
held by: Simone Paolo Ponzetto
held by: Paul Buitelaar
held by: Andrea Moro
held by: Brian Davis
held by: Giuseppe Rizzo
held by: Pierpaolo Basile
held by: Monika Solanki
held by: Claudia D'Amato
held by: Chris Welty
held by: Katja Hose
held by: Petr Knoth
held by: Rabeeh Ayaz Abbasi
held by: Jay Pujara
held by: Matthias Nickles

Matthias Nickles
holds role: Program committee member
last name: Nickles
first name: Matthias
type: Person

Mehrnaz Ghashghaei
holds role: Paper author
last name: Ghashghaei
first name: Mehrnaz
made: Semantic Annotation of Quantitative Textual Content
type: Person

Martin Riedl
holds role: Program committee member
last name: Riedl
first name: Martin
type: Person

University of Sheffield
member: Ziqi Zhang
member: Anna Lisa Gentile
type: Organization

Achim Rettinger
holds role: Paper author
last name: Rettinger
first name: Achim
made: A Topic-Sensitive Model for Salient Entity Linking
type: Person

Chris Welty
made: CrowdTruth Measures for Language Ambiguity: The Case of Medical Relation Extraction
holds role: Program committee member
made: Goodbye to True: Advancing Semantics beyond the Black and White
first name: Chris
last name: Welty
holds role: Paper author
type: Person
holds role: Keynote speaker
made: http://data.semanticweb.org/keynote/1

Simone Paolo Ponzetto
holds role: Program committee member
last name: Ponzetto
first name: Simone Paolo
type: Person

Monika Solanki
holds role: Program committee member
last name: Solanki
first name: Monika
type: Person

The 3rd International Workshop on Linked Data for Information Extraction
is super event of: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
type: ConferenceEvent
has acronym: LD4IE2015
is super event of: 96a4bh66arosfhfed2uu76kt08@google.com 8
summary: The 3rd International Workshop on Linked Data for Information Extraction
description: The 3rd International Workshop on Linked Data for Information Extraction
dtend: 2015-10-12T15:30:00
has role: Chair
address: Bethlehem, Pennsylvania - USA
has location: Bethlehem, Pennsylvania - USA
dtstart: 2015-10-12T09:00:00

ri9oe6lcndmbii7j2mehclqu2o@google.com 2
last modified: 2015-10-02T12:00:16Z
transp: OPAQUE
is super event of: 817b4icma6mkao4j5jjf2hhu2o@google.com 1
presents: Semantic Annotation of Quantitative Textual Content
summary: Research | Semantic Annotation of Quantitative Textual Content
description: Mehrnaz Ghashghaei, John Cuzzola, Ebrahim Bagheri and Ali Ghorbani.
type: Vevent
location: Room RBC 271
dtend: 2015-10-12T14:00:00Z
is super event of: rlofdvrkt3tmr48533jbsgqt48@google.com 4
dtstart: 2015-10-12T13:30:00Z
is super event of: ecv20nbc0jsq0uee9bnjb197mk@google.com 3
sequence: 0
is sub event of: The 3rd International Workshop on Linked Data for Information Extraction
is super event of: 6qn80t9g85uvdoooftrhjjbbv4@google.com 5
created: 2015-10-02T10:01:21Z
status: CONFIRMED
dtstamp: 2015-10-02T12:03:03Z

Jay Pujara
holds role: Program committee member
last name: Pujara
first name: Jay
type: Person

Kristian Kolthoff
holds role: Paper author
last name: Kolthoff
first name: Kristian
made: Semantic Relation Composition in Large Scale Knowledge Bases
type: Person

Claudia D'Amato
affiliation: University of Bari
holds role: Chair
holds role: Program committee member
last name: D'Amato
first name: Claudia
type: Person

Andrea Moro
holds role: Program committee member
last name: Moro
first name: Andrea
type: Person

Miriam Fernandez
holds role: Program committee member
last name: Fernandez
first name: Miriam
type: Person

rlofdvrkt3tmr48533jbsgqt48@google.com 4
status: CONFIRMED
created: 2015-10-02T10:02:43Z
last modified: 2015-10-02T12:00:31Z
dtend: 2015-10-12T15:30:00Z
type: Vevent
dtstart: 2015-10-12T15:00:00Z
is sub event of: ri9oe6lcndmbii7j2mehclqu2o@google.com 2
dtstamp: 2015-10-02T12:03:03Z
sequence: 0
description: Jennifer Sleeman, Tim Finin and Anupam Joshi.
transp: OPAQUE
location: Room RBC 271
summary: Research | Topic Modeling for RDF Graphs
presents: Topic Modeling for RDF Graphs

Payam Barnaghi
holds role: Program committee member
last name: Barnaghi
first name: Payam
type: Person

Cong Liu
holds role: Paper author
last name: Liu
first name: Cong
made: A Topic-Sensitive Model for Salient Entity Linking
type: Person

Craig Knoblock
holds role: Program committee member
last name: Knoblock
first name: Craig
type: Person

Paul Buitelaar
holds role: Program committee member
last name: Buitelaar
first name: Paul
type: Person