I'm looking for advice on the most appropriate way to assess inter-rater reliability for a study involving the annotation of noun events in audio recordings.
Pairs of annotators independently listen to the same recordings and identify instances where a noun is produced. For each event, they record:
The start and end time of the event.
The noun identified
How do you think I should calculate the reliability between coders? I also wanted to allow for some error in the timestamps, so if they are within a certain of tolerance then it is considered correct. Would Cohen's Kappa be useful here?
I was thinking of something like, but not sure if it is appropritae.
Annotator A Annotator B Classification
Event at 2.1–2.5s: "ball" Event at 2.2–2.6s: "ball" Agreement
Event at 4.0–4.4s: "dog" Event at 4.1–4.5s: "cat" Disagreement
Event at 6.0–6.4s: "car" No corresponding event Disagreement
No corresponding event Event at 8.0–8.4s: "cup" Disagreement
library(tibble)
reliability.df <- tribble(
~file, ~annotator,…