Implications of inter-rater agreement on a student information retrieval evaluation

to related objects

Abstract: This paper is about an information retrieval evaluation on three different retrieval-supporting services. All three services were designed to compensate typical problems that arise in metadata- driven Digital Libraries, which are not adequately handled by a simple tf-idf based retrieval. The services are: (1) a co-word analysis based query expansion mechanism and re-ranking via (2) Bradfordizing and (3) author centrality. The services are evaluated with relevance assessments conducted by 73 information science students. Since the students are neither information professionals nor domain experts the question of inter-rater agreement is taken into consideration. Two important implications emerge: (1) the inter- rater agreement rates were mainly fair to moderate and (2) after a data-cleaning step which erased the assessments with poor agreement rates the evaluation data shows that the three retrieval services returned disjoint but still relevant result sets. (author's abstract)

Location: Deutsche Nationalbibliothek Frankfurt am Main

Extent: Online-Ressource, 1-7 S.

Language: Englisch

Notes: Postprint
begutachtet
In: Atzmüller, Martin (Hg.), Benz, Dominik (Hg.), Hotho, Andreas (Hg.), Stumme, Gerd (Hg.): LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings. 2010. S. 1-7

Bibliographic citation: LWA 2010 : Lernen, Wissen & Adaptivität ; workshop proceedings ; Bd. 2010/5
Kasseler Informatikschriften (KIS) ; Bd. 2010/5

Classification: Bibliotheks- und Informationswissenschaft

Event: Veröffentlichung

(where): Kassel

(when): 2010

Creator: Schaer, Philipp
Mayr, Philipp
Mutschke, Peter

Contributor: Atzmüller, Martin
Benz, Dominik
Hotho, Andreas
Stumme, Gerd
Universität Kassel, Knowledge and Data Engineering Group

URN: urn:nbn:de:0168-ssoar-46667-7

Rights: Open Access unbekannt; Open Access; Der Zugriff auf das Objekt ist unbeschränkt möglich.

Last update: 15.08.2025, 7:31 AM CEST

Data provider

This object is provided by:
Deutsche Nationalbibliothek. If you have any questions about the object, please contact the data provider.

Show original at data provider

Associated

Schaer, Philipp
Mayr, Philipp
Mutschke, Peter
Atzmüller, Martin
Benz, Dominik
Hotho, Andreas
Stumme, Gerd
Universität Kassel, Knowledge and Data Engineering Group

Time of origin

2010

Other Objects (12)

Implications of inter-rater agreement on a student information retrieval evaluation

Collection article | Conference paper | Konferenzbeitrag | Sammelwerksbeitrag

Implications of inter-rater agreement on a student information retrieval evaluation

Inter-rater agreement of the triagesystem RETTS-HEV

Inter-rater agreement of the triagesystem RETTS-HEV

Detection of grey zones in inter-rater agreement studies

Detection of grey zones in inter-rater agreement studies

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-Rater Reliability: Chance-corrected Measures

Inter-Rater Reliability: Chance-corrected Measures

The inter-rater reliability of stroboscopy evaluations

The inter-rater reliability of stroboscopy evaluations

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

On inter-rater reliability for crowdsourced QoE

On inter-rater reliability for crowdsourced QoE

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Implications of inter-rater agreement on a student information retrieval evaluation

Collection article | Conference paper | Konferenzbeitrag | Sammelwerksbeitrag

Implications of inter-rater agreement on a student information retrieval evaluation

Inter-rater agreement of the triagesystem RETTS-HEV

Inter-rater agreement of the triagesystem RETTS-HEV

Detection of grey zones in inter-rater agreement studies

Detection of grey zones in inter-rater agreement studies

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-Rater Reliability: Chance-corrected Measures

Inter-Rater Reliability: Chance-corrected Measures

The inter-rater reliability of stroboscopy evaluations

The inter-rater reliability of stroboscopy evaluations

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

On inter-rater reliability for crowdsourced QoE

On inter-rater reliability for crowdsourced QoE

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Implications of inter-rater agreement on a student information retrieval evaluation

Collection article | Conference paper | Konferenzbeitrag | Sammelwerksbeitrag

Implications of inter-rater agreement on a student information retrieval evaluation

Inter-rater agreement of the triagesystem RETTS-HEV

Inter-rater agreement of the triagesystem RETTS-HEV

Detection of grey zones in inter-rater agreement studies

Detection of grey zones in inter-rater agreement studies

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Mixture polarization in inter-rater agreement analysis: a Bayesian nonparametric index

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-rater agreement of comorbid DSM-IV personality disorders in substance abusers

Inter-Rater Reliability: Chance-corrected Measures

Inter-Rater Reliability: Chance-corrected Measures

The inter-rater reliability of stroboscopy evaluations

The inter-rater reliability of stroboscopy evaluations

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Examining intra-rater and inter-rater response agreement: A medical chart abstraction study of a community-based asthma care program

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Inter-rater agreement in the scoring of abstracts submitted to a primary care research conference

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

Contrast-enhanced US Bosniak Classification: intra- and inter-rater agreement, confounding features, and diagnostic performance

On inter-rater reliability for crowdsourced QoE

On inter-rater reliability for crowdsourced QoE

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Inter-rater reliability of the Abbreviated Injury Scale scores in patients with severe head injury shows good inter-rater agreement but variability between countries. An inter-country comparison study

Cultural heritage institutions wishing to register will find more information here.

Fields marked * need to be filled in.

Username*

Please enter your username

Email*

Please enter your email address

Please do not fill this field

First name

Last name

Password*

Please enter your password

Confirm password*

Please enter the same password

I have read the terms of use and the privacy policy for the collection of personal data and accept them. *

This field is required.

I would like to subscribe to the newsletter of the Deutsche Digitale Bibliothek. See newsletter subscription info.

Account created

Your "My DDB" account has been successfully created. Before you can log in to your account, you must click the confirmation link in the message we just sent to the email address you provided.