Menu

[Q] What would be the best approach to inter-rater reliability for temporally annotated events?

Reading 0:00
15s threshold

I'm looking for advice on the most appropriate way to assess inter-rater reliability for a study involving the annotation of noun events in audio recordings.

Pairs of annotators independently listen to the same recordings and identify instances where a noun is produced. For each event, they record:

The start and end time of the event.

The noun identified

How do you think I should calculate the reliability between coders? I also wanted to allow for some error in the timestamps, so if they are within a certain of tolerance then it is considered correct. Would Cohen's Kappa be useful here?

I was thinking of something like, but not sure if it is appropritae.

Annotator A Annotator B Classification

Event at 2.1–2.5s: "ball" Event at 2.2–2.6s: "ball" Agreement

Event at 4.0–4.4s: "dog" Event at 4.1–4.5s: "cat" Disagreement

Event at 6.0–6.4s: "car" No corresponding event Disagreement

No corresponding event Event at 8.0–8.4s: "cup" Disagreement

library(tibble)

reliability.df <- tribble(

~file, ~annotator,…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.