Abstracts

Rationale: Reliable computerized detection of epileptiform transients (ETs), characterized by interictal spikes and sharp waves in the electroencephalogram (EEG), is a useful goal since this would assist physicians in reviewing scalp EEG recordings. It is our goal to create standardized datasets to help train automated ET detection algorithms. In this study, we used EEGnet, a distributed web-based platform for the analysis of scalp EEG recordings to compare the inter-rater reliability for marking ETs. Two different methods of scoring were compared. The first method involved labeling paroxysmal activity as either epileptiform or non-epileptiform (binary scoring). The second method involved labeling paroxysmal activity on a scale of 0-4, depending on the degree of epileptiform appearance (ordinal scoring). Methods: One hundred 30-second routine scalp EEG segments from 100 different patients were selected for analysis. Fifty of these segments were selected because they contained ETs from patients with known epilepsy and the other fifty were selected because they contained benign paroxysmal activity (exaggerated alpha activity, wicket spikes, and small sharp spikes) which could easily be misinterpreted by an inexperienced reviewer. Scoring was performed in three phases by ACNS board certified academic neurophysiologists. In the first phase, seven scorers marked all of the paroxysmal activity in the segments (including epileptiform activity, other EEG activity, and artifacts). In the second phase, eleven scorers marked each paroxysmal event as either artifact, epileptiform activity, or non-epileptiform EEG activity. In the third phase, the eleven scorers categorized all of the events marked by at least one scorer (in phase two) as epileptiform as well as some randomly-selected non-epileptiform events as either non-epileptiform, or epileptiform on a scale of 1-4. For inter-rater analysis, we examined all pair-wise rater agreement scores using the kappa statistic and generated a composite score based on the average agreement score across all pairs for both the binary and the ordinal data. using Cicchetti-Allison weights. We also calculated the reliability coefficient described in Wilson et. al (1996) using the Spearman correlation. Results: The average kappa value for the data with binary scores was 0.40 +/- 0.12 with correlations with a range of 0.12-0.59. The average kappa value for the data with ordinal scores was 0.42 +/- 0.078 with a range of 0.30-0.62. The kappa values were not significantly different between the binary and ordinal scoring, but the variability of the kappa values were greater for the binary in comparison to the ordinal scoring data. The reliability coefficient was not significantly different for the binary scoring method (0.89) compared to the ordinal scoring method (0.91). Conclusions: Inter-rater agreement for categorizing paroxysmal activity as epileptiform activity was moderate. The inter-rater agreement for binary and ordinal scoring were not significantly different, although there was decreased variability for the ordinal scoring.