Más contenido relacionado La actualidad más candente (20) Similar a Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment (20) Are Linked Datasets fit for Open-domain Question Answering? A Quality Assessment15. Digging into the QA system
Typical IR system performances
measures
● Overall Performance
○ F1
○ Precision
○ Recall
16. Digging into the QA system
Data & Component/Module
oriented measures
● Search & retrieval module
○ Indexer
○ Retriever
● Preprocessing / Linguistic
○ NLP - POS tags, NER, etc
○ Entity linking & annotation - semantics
○ Relation extraction & annotation
● Query formulation
○ SPARQL conversion
● Datasource/knowledge base
○ Data
Typical IR system performances
measures
● Overall Performance
○ F1
○ Precision
○ Recall
17. Digging into the QA system
Data & Component/Module
oriented measures
● Search & retrieval module
○ Indexer
■ Top K words accuracy; P@10,
P@1000, etc
○ Retriever
■ Ranking, Re-ranking, MRR, etc
● Preprocessing / Linguistic
○ NLP - POS tags, NER, etc
○ Entity linking & annotation - semantics
○ Relation extraction & annotation
■ annotation accuracy/precision
■ consistency, interlinking, etc
● Query formulation
○ SPARQL conversion
■ conversion accuracy/precision
● Datasource/knowledge base
○ Completeness
○ Data diversity
○ Trust and Provenance
○ Coverage
○ Timeliness (up to date)
○ etc
Typical IR system performances
measures
● Overall Performance
○ F1
○ Precision
○ Recall
18. Digging into the QA system
Data & Component/Module
oriented measures
● Search & retrieval module
○ Indexer
■ Top K words accuracy; P@10,
P@1000, etc
○ Retriever
■ Ranking, Re-ranking, MRR, etc
● Preprocessing / Linguistic
○ NLP - POS tags, NER, etc
○ Entity linking & annotation - semantics
○ Relation extraction & annotation
■ annotation accuracy/precision
■ consistency, interlinking, etc
● Query formulation
○ SPARQL conversion
■ conversion accuracy/precision
● Datasource/Knowledge base
○ Completeness
○ Data diversity
○ Trust and Provenance
○ Coverage
○ Timeliness (up to date)
○ etc
Typical IR system performances
measures
● Overall Performance
○ F1
○ Precision
○ Recall
32. Dimension Metric DB_Rest DB_Poli DB_Film DB_Soc
Availability
EstimatedDereferenceabilityMetric 0.013 0.013 0.012 0.012
EstimatedDereferenceabilityForwardLinksMetric 0.027 0.027 0.027 0.027
NoMisreportedContentTypesMetric 0 1 1 1
RDFAvailabilityMetric 0 0 0 0
EndPointAvailabilityMetric 0 0 0 0
Interlinking
EstimatedInterlinkDetectionMetric - - - -
EstimatedLinkExternalDataProviders - - - -
EstimatedDereferenceBackLinks 0.012 0.014 0.015 0.022
Semantic
accuracy
OntologyHijacking 1 1 1 1
MisusedOwlDatatypeOrObjectProperties 1 1 1 1
Data diversity
HumanReadableLabelling 0.953 0.985 0.997 1
MultipleLanguageUsageMteric 1 2 3 3
Trust and
Provenance
Basic Provenance 0 0 0 0
Extended Provenance 0 0 0 0
Provenance Richness 0 0 0 0
DBPEDIA SLICE ASSESSMENT RESULTS
33. WIKIDATA SLICE ASSESSMENT RESULTS
Dimension Metric Wiki_Rest Wiki_Poli Wiki_Film Wiki_Soc
Availability
EstimatedDereferenceabilityMetric 0.051 0.063 0.048 0.062
EstimatedDereferenceabilityForwardLinksMetric 0.093 0.053 0.050 0.064
NoMisreportedContentTypesMetric 0 1 0 1
RDFAvailabilityMetric 0 0 0 0
EndPointAvailabilityMetric 0 0 0 0
Interlinking
EstimatedInterlinkDetectionMetric - - - -
EstimatedLinkExternalDataProviders 5 11 9 8
EstimatedDereferenceBackLinks 0.013 0.098 0.089 0.083
Semantic
accuracy
OntologyHijacking 1 1 1 1
MisusedOwlDatatypeOrObjectProperties 1 1 1 1
Data diversity
HumanReadableLabelling 0.175 0.076 0.091 0.102
MultipleLanguageUsageMteric 2 3 2 3
Trust and
Provenance
Basic Provenance 0 0 0 0
Extended Provenance 0 0 0 0
Provenance Richness 0.055 0.083 0.010 0.025