JOURNALOFAPPLIEDBEHAVIORANALYSIS 2009, 42, 165–169 NUMBER1 (SPRING2009) CONTINUOUS RECORDING AND INTEROBSERVER AGREEMENT ALGORITHMS REPORTED IN THE JOURNAL OF APPLIED BEHAVIOR ANALYSIS (1995–2005) OLIVER C. MUDFORD AND SARAH ANN TAYLOR UNIVERSITYOFAUCKLAND,NEWZEALAND AND NEIL T. MARTIN TREEHOUSETRUST,LONDON Wereviewedallresearcharticlesin10recentvolumesoftheJournalofAppliedBehaviorAnalysis (JABA): Vol. 28(3), 1995, through Vol. 38(2), 2005. Continuous recording was used in the majority (55%) of the 168 articles reporting data on free-operant human behaviors. Three methodsforreportinginterobserveragreement(exactagreement,block-by-blockagreement,and time-windowanalysis)wereemployedinmorethan10ofthearticlesthatreportedcontinuous recording. Having identified these currently popular agreement computation algorithms, we explainthem toassist researchers, software writers, andotherconsumers ofJABA articles. DESCRIPTORS: computers,continuousrecording,interobserveragreement,observational data,recording andmeasurement _______________________________________________________________________________ It has been over 30 years since Kelly’s (1977) JABA, including bias of interval recording and initial review of data-collection and interob- random error of time sampling (e.g., Powell, server agreement methods in the Journal of Martindale, & Kulp, 1975) and problems with Applied Behavior Analysis (JABA). Kelly found interobserver agreement (e.g., Repp, Deitz, that the majority (76%) of research articles Boles, Deitz, & Repp, 1976). published from 1968 to 1975 used pencil-and- Discontinuousmethodsdonotallowthebasic paper methods for discontinuous recording of dimensions of behaviors to be quantified accu- behavioral observations (e.g., interval recording rately in standard scientific units (e.g., rates in and time sampling). Following Kelly’s review, responses per minute, durations, interresponse there have been a series of investigations of the times, and latencies in seconds; Hanley, Cam- merits of various methods of data collection in milleri, Tiger, & Ingvarsson, 2007; Johnston & Pennypacker, 1993). Continuous recording is SarahAnnTaylorisnowatOdysseyHouse,Auckland, required for direct measurement of the basic NewZealand. dimensions of behaviors. The availability of A portion of this paper was presented at the third handheld portable electronic data-entry and international conference of the Association for Behavior Analysis,Beijing,China,November,2005.Contributions storage devices has increased the practicality and to the funding of this research were received from the affordabilityofcontinuousrecordingforresearch Universityof Auckland(UoA) Research Committee (first and clinical purposes. The aims of this review author) and the UoA Faculty of Science Summer Scholarship Programme (second author). We are grateful were to determine the relative frequencies of for explanatory correspondence with Wayne Fisher, Greg continuousanddiscontinuousrecordingmethods Hanley,Brian Iwata,andSungWoo Kahng. inJABAarticlesover10recentvolumes(1995to AddresscorrespondencetoOliverC.Mudford,Applied Behaviour Analysis Programme, Department of Psychol- 2005) and to quantify variations in methods for ogy, University of Auckland (Tamaki Campus), Private assessment of the reliability (i.e., interobserver Bag 92019, Auckland 1142, New Zealand (e-mail: agreementandaccuracy)ofcontinuouslyrecord- [email protected]). doi:10.1901/jaba.2009.42-165 ed behavioral data. 165 166 OLIVER C. MUDFORD et al. METHOD these articles were again scrutinized to ascertain what algorithms were used to assess the Selection of Articles for Review reliability of the data (e.g., block-by-block All research articles published in 10 years of agreement; Bailey & Bostow, cited in Page & JABA—Vol. 28(3), 1995, to Vol. 38(2), Iwata, 1986; Bailey & Burch, 2002), exact 2005—were examined by the first author. agreement (Repp et al., 1976), time-window Reviews, discussion articles, and reports (abbre- analysis (MacLean, Tapp, & Johnson, 1985; viated research articles) were not included. The Tapp & Wehby, 2000), or others. second author acted as an independent reviewer throughout for the purposes of estimating Interobserver Agreement of Review interobserver agreement (described below). The interobserver agreement for the review process was assessed using a stratified procedure Review Procedure in which approximately 20% of articles at each Review was conducted in four stages. First, level were selected randomly for independent all articles were retained for closer examination examination by the second author. Percentage that reported at least some direct observation agreement at the first three levels of review was data, either in vivo or from video, of free- calculated by dividing number of agreements operant human behavior. Thus, papers that between reviewers by number of articles exam- included only automatically (mechanically or ined by both reviewers and converting this ratio electronically) recorded data were excluded, as to a percentage. Mean interobserver agreements were articles that reported on restricted-operant were 100%, 96%, and 95% on 51, 36, and 21 behaviors only (e.g., data from trial-by-trial articles subjected to assessment, respectively. teaching or from researcher-controlled bite-by- A different procedure was used for checking bitebehaviorsineating-relatedstudies)andtwo the first author’s identification of interobserver research articles concerning animal behaviors. agreement calculation methods, the fourth level This resulted in a total of 168 articles. of review. Although the time-window analysis Second,theretainedarticleswerereviewedto was identifiable with 100% agreement, pub- determine whether they reported continuous or lished descriptions of the block-by-block and discontinuous data. Continuous data collection exact agreement algorithms differed across was identified by applying the following articles such that agreement assessment was definition: Researchers described observational not considered as a valid surrogate for accuracy records that contained second-by-second rec- of identification of these algorithms. Therefore, ords of occurrences of discrete behaviors or the senior authors whose work had used one or onsets and offsets of behaviors with duration, both of these algorithms and had been and the results were reported in standard units previously published in JABA were contacted of measurement or their derivatives (e.g., by the first author. Confirmation of the responsesperminute,percentageofobservation calculation methods was obtained from each session). Discontinuous methods were defined author. Although it was an unconventional as data-collection procedures that recorded methodforassessing interobserveragreementof behaviors in time samples or intervals of more review, this provided an accurate means of than 1 s. The 93 articles that included confirming our identification of the algorithms. continuous data were reviewed further. Third, the articles remaining were examined RESULTS AND DISCUSSION to determine whether obtained continuous data were analyzed to produce frequency (or rate) All 256 JABA research articles published measures, duration measures, or both. Fourth, frommid-1995tomid-2005werereviewed.Of CONTINUOUS RECORDING AND AGREEMENT 167 Figure 1. Cumulative frequencies of JABA research articles from 1995 to 2005, reporting data on free-operant human behaviors that were recorded by observers using only discontinuous recording procedures and those reporting continuouslyrecorded data. these, 168 reported direct observational data predominated (i.e., were reported over 10 from free-operant human behavior. Of these times) in the articles reviewed: block-by-block 168, 93 articles (55%) reported continuously agreement, exact agreement, and time-window recorded data. Discontinuous methods for analysis. Figure 2 shows the cumulative fre- recording such behaviors have been superseded quency of articles that reported using the three in published research applications of behavior methods. The data should not be interpreted to analysis. Figure 1 shows the rates of use of suggest that one method is preferable because it continuous and discontinuous methods for was used more often than another (e.g., the observational recording in research articles block-by-block method was reportedly used 46 across the 10 years reviewed. Among the 93 times, three times more often than the time- articles reporting continuously recorded free- windowanalysismethod).Thefrequencyofuse operant human behaviors, 88 (95%) reported may be indicative of the publication rates of frequency measures (usually rate of responding research groups that chose to employ the per minute). Duration measures were reported different methods. It was noted during review in 33 articles (36%). that computational methods for interobserver All articles that contained continuously agreement were not always fully described or recorded data reported interobserver agreement consistently named. Therefore, we provide a data;nonereportedobserveraccuracymeasures. detailed explanation of the three most popular Thus,interobserver agreementhas continued to algorithms identified during our review. be the method by which the quality of The exact and block-by-block agreement behavioral data is assessed (as in Kelly, 1977). methods were developed for use with discon- Three methods for computing agreement tinuouslyrecordeddata.Theyaresimilarinthat 168 OLIVER C. MUDFORD et al. Figure 2. Cumulative frequencies of the three commonly reported interobserver computation methods for continuous recording inJABA research articlesfrom1995through 2005. the second-by-second data streams from two In the block-by-block method, the smaller of observershave10-sintervalssuperimposed.The the two observers’ totals in a 10-s interval is level of analysis for discrete data (events) is the divided by the larger. This provides a score numberofoccurrencesofthebehaviorrecorded between 0 and 1 for every interval. For the in a 10-s interval. With duration measures, the purposes of calculating agreement when both number of seconds within a 10-s interval that observers scored no occurrences during an the behavior was recorded as occurring is interval, such intervals are scored as 1. Scores counted for each observer (e.g., Hagopian, aresummedacrossallintervalsanddividedbythe Contrucci-Kuhn, Long, & Rush, 2005; Rapp, numberofintervals,andtheresultisconvertedto Vollmer, St. Peter, Dozier, & Cotnoir, 2004). a percentage to provide a percentage agreement Theexactagreementmethodisdescribedfully index. This description of calculating block-by- in Piazza, Hanley, and Fisher (1996, p. 440): block agreementappliestoall research articlesin ourreviewidentifiedasusingthemethod,andit Exact agreement coefficients were calculated by partitioningeachsessioninto10-sintervals.Ineach wasconfirmedbyusersofthealgorithm.Asused, interval, two observers could agree on the exact the method deviates from the computation number of behaviors that occurred, agree that explained by Bailey and Burch (2002), in which behavior did not occur, or disagree about the exact intervals of agreement on nonoccurrence (zero number of behaviors that occurred (disagreement). … Coefficients were calculated by dividing the divided by zero) are ignored. The method in number of agreements by the sum of agreements commonusecouldbelabeledblock-by-block(all plus disagreements andmultiplying by100%. intervals) method to differentiate it from similar Repp et al.(1976) identified this formula as the algorithms. exact agreement (all intervals) method because Computation of percentage agreement using itincludesagreementsonnonoccurrenceinthe time-window analysis was devised for continu- calculation. ously recorded data. One-second intervals are CONTINUOUS RECORDING AND AGREEMENT 169 imposed on two observers’ data streams, and Hollenbeck, A. R. (1978). Problems of reliability in observational research. In G. P. Sackett (Ed.), second-by-second comparisons are made be- Observingbehavior:Vol.2.Datacollectionandanalysis tween them. When both records show an event methods (pp. 79–98). Baltimore: University Park (for discrete behaviors) or a second of ongoing Press. occurrence (for behaviors measured with dura- Johnston, J. M., & Pennypacker, H. S. (1993). Strategies and tactics of behavioral research (2nd ed.). Hillsdale, tion), this is counted as an agreement. Any NJ: Erlbaum. second in which only one record contains an Kelly, M. B. (1977). A review of the observational data- event or occurrence of behavior is a disagree- collection and reliability procedures reported in the JournalofAppliedBehaviorAnalysis.JournalofApplied ment. Percentage agreement is calculated by Behavior Analysis,10,97–101. dividing the number of agreements by the Lalli, J. S., Mauro, B. C., & Mace, F. C. (2000). number of agreements plus disagreements. Preference for unreliable reinforcement in children MacLean et al. (1985) recognized that their with mental retardation: The role of conditioned reinforcement. Journal of Applied Behavior Analysis, algorithm was overly stringent for data on 33,533–544. discrete events. Consequently, they recom- MacLean,W.E.,Tapp,J.T.,&Johnson,W.L.(1985). mended allowing tolerance for counting agree- Alternate methods and software for calculating interobserver agreement for continuous observation ments by expanding the definition of an data. Journal of Psychopathology and Behavioral agreement to include observations when one Assessment,7,65–73. observer recorded an event within 6 t seconds Page, T. J., & Iwata, B. A. (1986). Interobserver of the other observer. In research articles agreement: History, theory and current methods. In A.Poling&R.W.Fuqua(Eds.),Researchmethodsin sampled, t has varied from 1 s (e.g., Romaniuk appliedbehavioranalysis:Issuesandadvances(pp.99– etal.,2002) to5 s(e.g., Lalli,Mauro,& Mace, 126). New York:Plenum. 2000, Experiment 3). Piazza, C. C., Hanley, G. P., & Fisher, W. W. (1996). Functional analysis and treatment of cigarette pica. Compared with the extensive methodological Journal ofAppliedBehavior Analysis, 29,437–450. studies on discontinuous recording, there has Powell, J., Martindale, A., & Kulp, S. (1975). An been little research effort to comprehend, evalu- evaluation of time-sample measures of behavior. ate,orguideselectionofmethodsforassessment Journal ofAppliedBehavior Analysis, 8,463–469. Rapp,J.T.,Vollmer,T.R.,St.Peter,C.,Dozier,C.L.,& ofdataqualitywithcontinuousrecording.There Cotnoir, N. M. (2004). Analysis of response have been recommendations for evaluating allocation in individuals with multiple forms of interobserver agreement with continuous data stereotyped behavior. Journal of Applied Behavior Analysis,37,481–501. (e.g., Hollenbeck, 1978; MacLean et al., 1985) Repp,A.C.,Deitz,D.E.D.,Boles,S.M.,Deitz,S.M., but no methodological studies have compared & Repp,C. F. (1976). Technical article: Differences differentmethodsinuse.Theresultsofthisreview amongcommonmethodsforcalculatinginterobserv- suggestthatcontinuousrecordingisatimelytopic er agreement. Journal ofApplied Behavior Analysis, 9, 109–113. formethodologicalstudy. Romaniuk,C.,Miltenberger,R.,Conyers,C.,Jenner,N., Jurgens,M.,&Ringenberg,C.(2002).Theinfluence of activity choice on problem behaviors maintained REFERENCES byescapeversusattention.JournalofAppliedBehavior Bailey,J.S.,&Burch,M.R.(2002).Researchmethodsin Analysis,35,349–362. applied behavioranalysis.Thousand Oaks, CA:Sage. Tapp,J.,&Wehby,J.H.(2000).Observationalsoftware Hagopian, L. P., Contrucci-Kuhn, S. A., Long, E. S., & forlaptopcomputersandopticalbarcodereaders.In Rush, K. S. (2005). Schedule thinning following T. Thompson, D. Felce, & F. J. Symons (Eds.), communicationtraining:Usingcompetingstimulito Behavioral observation: Technology and applications in enhancetolerancetodecrementsinreinforcerdensity. developmental disabilities (pp. 71–81). Baltimore: JournalofAppliedBehavior Analysis,38,177–193. Brookes. Hanley, G. P., Cammilleri, A. P., Tiger, J. H., & Ingvarsson, E. T. (2007). A method for describing Received January 18,2007 preschoolers’ activity preferences. Journal of Applied Final acceptanceJanuary 25, 2008 BehaviorAnalysis,40,603–618. Action Editor,Mark Dixon