STATISTICALMETHODSFORTHEANALYSISOFDESIGNSINCLUDING TREATMENTSDEUVEREDTOGROUPSANDTOINDIVIDUALS:AN ANALYTICANDMONTECARLOSTUDY ; " 4.i:» . -' •-. • 'j" j-'i : » J By STEPHANIEBILLERWEHRY ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL OFTHEUNIVERSITYOFFLORIDAINPARTIALFULHLLMENT OFTHEREQUIREMENTSFORTHEDEGREEOF DOCTOROFPHILOSOPHY UNIVERSITYOFFLORIDA 2001 ACKNOWLEDGEMENTS Iwouldliketothankmydissertationcommitteechairman,committee members,thefacultyandstaffoftheSchoolofTeachingandLearning,Dr.Adel Boules,oftheUniversityofNorthFloridaandmyfamilyfortheirsupportand encouragementinmypursuitofthisdegree. First,Ithankmydissertationchairperson.Dr.JamesJ.Algina,forsuggesting thetopic,guidingmethroughtheprocess,andnotgivingupwhenthetopicappeared toprematurelyterminate. Additionally,IthankDr.Alginaforcaringenoughabout theartofteachingtoapproacheachlessonandassessmentopportunityina thoughtful,well-preparedmanner~themarkofanexcellentteacher. Likewise,IalsothankdissertationcommitteemembersDr.LindaCrockerand Dr.DavidMillerforcarefullyandthoughtfullyreadingthemanuscript,forexcellent classes,andforthefinancialsupportofgraduateassistantships.Furthermore,Ithank committeememberDr.DonaldBernardforhissupportofmydegreeprogramfrom myinitialvisitwithhiminthesummerof19%throughthecompletionofthe program. Dr.Bernard,Dr.MaryG.Kantowski,andDr.BenNelmshavealsoprovided financialsupportthroughgraduateteachingassistantshipsintheSchoolofTeaching andLearning. FacultymembersandstaffintheSchoolofTeachingandLearning supportedmyteachingeffortsandprovidedopportunitiesformetocontinue interactingwithpre-serviceelementaryschoolteachersandwithprofessional teachingandeducationalresearchorganizations. u IacknowledgetremendousgratitudetoDr.AdelBoulesoftheUniversityof NorthFloridaDepartmentofMathematicsandStatistics. Ithankhimforhis assistancewiththenumericalmethodsnecessaryforthecompletionofthisresearch. RnallyIthankmyfamilyfortheirpatienceandsupport TABLEOFCONTENTS ACKNOWLEDGEMENTS ii ABSTRACT vi CHAPTERS 1 INTRODUCTION 1 MethodologicalConcerns 3 StatementoftheProblem 17 2METHODOLOGY 21 Quasi-FTestforTreatmentEffectivenessComparingWeightedMeans . . 22 EstimationofVarianceComponents-WeightedMeans 24 VarianceComponentEstimates 26 TheTestStatistic 27 NegativeVarianceComponentEstimates 28 ApproximateDegreesofFreedom 30 VariablesManipulatedintheMonteCarloStudy 35 Summary 41 3 REPLICATIONOFTHESTUDYOFMYERSETAL.(1981)Models 42 VerificationofDataandEquivalenceofStatisticalModels 42 TheoryofScarianoandDavenport 1 7* 46 ResultsoftheSimulation 60 4ANALYTICSTUDYOFBALANCEDDATA..... . 61 FactorsoftheStudy 61 ResultsoftheStudy 62 SummaryoftheResults 69 5MONTECARLOSTUDYOFBALANCEDDATE 71 FactorsoftheMonteCarloStudy 71 ResultsoftheMonteCarloStudy 72 Summary 81 6 CONCLUSIONS 84 Conclusions 85 ExplanationoftheResults 90 SuggestionsforFutureResearch 92 REFERENCES 99 BIOGRAPHICALSKETCH 102 V AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFlorida inPartialFulfillmentoftheRequirementsfortheDegreeofDoctorofPhilosophy STATISTICALMETHODSFORTHEANALYSISOFDESIGNSINCLUDING TREATMENTDELIVEREDTOGROUPSANDTOINDIVIDUALS:AN ANALYTICANDMONTEDCARLOSTUDY By StephanieBillerWehry May,2001 Chain JamesJ.Algina MajorDepartment:EducationalPsychology ; Comparingtreatmenteffectivenessingroupsversusindividualresearch designs,designsinwhichtreatmentdeliveredtoindividualsiscomparedtotreatment deliveredtoindividualsnestedingroups,isconmioninsocialscienceresearch. However,ithasbeenshownthewidespreadpracticeofignoringanypossiblegroup dependencyinthedatacanresultinhypothesistestingusinganinflatedTypeIerror rate. Aquasi-Fteststatisticwithasyntheticmeansquareerrorhasbeensuggested foruseingroupsversusindividualdesignshavingequalsizedgroupsnestedinthe grouptreatmentlevel.Thechisquaredistributionoftheerrortermhasdegreesof freedomapproximatedusingatwo-momentapproach. Inthisstudy,thequasi-Fteststatisticwasextendedtoincludeafour-moment andanarithmeticaverageofthetwo-andfour-momentapproachesforapproximating thedegreesoffreedomofthesyntheticerrorterm. Aquasi-Fteststatisticwasalso vi developedtoincludedesignsthathaveunequalsizedgroups.TheTypeIerrorrates oftheresultingquasi-FtestswerestudiedusingbothMonteCarlosimulationsand analyticmethods. TheFactorsconsideredinthestudywerethesizeandnumberof thegroups,thedegreeofgroupdependency,thedegreeofhomoscedasticityofthe treatmentlevelvariances,andtherateofsubjectattrition. Theresultsofthestudysuggestedfordesignsinvolvingtwogroupsnestedin thegrouptreatmentlevel,noneoftheapproachestoapproximatingtheerrorterm degreesoffreedomresultedinteststhatcontroltheTypeIerrorrate.Increasingthe sizeofthetwogroupsfurtherinflatedtheTypeIerrorrate.Indesignsinvolvingthree groups,theaverage-momentquasi-Ftestisrecommended,and,indesignsinvolving fourormoregroups,theeithertwo-momentoraverage-momentquasi-Ftestis recommended. SubjectattritionratehadnosignificanteffectontheTypeIerrorrate ofanyofthequasi-Ftestsintheconditionsstudied. CHAPTERONE INTRODUCTION Researchersinthesocialsciencesoftenuseresearchdesignsthatcomparethe meansofmeasurementsobtainedfromindividualsnestedingroupstothoseof individualsnotnestedingroups.Thegroupsversusindividualsresearchdesigncan arisefromefficacystudiesthatusewaitlistornulltreatmentcontrolgroupsaswellas fromcomparativestudieshavingthespecificintenttocomparetheeffectivenessof treatmentsdeliveredtoindividualsnestedingroupstothosedeliveredtoindividuals. Researchersstudyingtheefficacyoftreatmentsdeliveredtosmallgroupsoftenuse waitlistcontrolgroups(Burlingame,Kircher,&Taylor,1994). Infact,themost commonexperimentalresearchdesigninpsychotherapyinvolvestheuseofa randomlyassignedcontrolconditionwhichcanfeatureavarietyofno-treatment controlschemes(Clarke,1998). Inthesedesigns,notreatmentisdeliveredtothe controlindividuals;therefore,individualresponsesareindependent. Inparticular,the useofwaitlistcontrolgroupsisethicallypleasingbecausecontrolgroupparticipants receivetreatmentaftersomespecifiedperiodoftime. Incomparativestudies,when themeasuredoutcomesoftreatmentsdeliveredtoindividualsnestedingroupsare comparedtooutcomesoftreatmentsdeliveredtoindividuals,theresultisagroups versusindividualsdesign. Theorymightsuggestgroupinteractionenhances,inhibits, orhasnoeffectonthetreatment. , ' ' ^^^ Somerecentexamplesofresearchthatusedthegroupsversusindividuals researchdesignarebyWebb(1999),byBolingandRobinson(1999),andbyBates, Thompson,andFlanagan(1999).Inherunpublisheddissertation,Webbstudiedthe effectsofagroupcounselinginterventiononhighschoolstudentsdiagnosedwith Attention-DeficitHyperactivityDisorder.Theoryincounseloreducationsuggested groupinterventionwouldbeeffective;therefore,thestudycomparedtheoutcome measurementsofindividualsnestedinsmallgroupstothoseoftheindividualsina nulltreatmentcontrolgroup.Theuseofanulltreatmentcontrolgroupresultedina groupsversusindividualsresearchdesign.BolingandRobinsoninvestigatedthe effectsofstudyenvironmentonameasureofknowledgefollowingadistance learninglecture. Thethreelevelsofstudyenvironmentincludedaprintedstudy guideaccessedbyindividuals,aninteractivemulti-mediastudyguideaccessedby individuals,andaprintedstudyguideaccessedbycooperativestudygroups.This investigationisanexampleofresearchspecificallydesignedtocomparethegroup andindividualenvironments. Intheresearch,thesamestudyguidewastailoredto accommodateeachenvironment.Batesetal.comparedtheeffectivenessofamood inductionprocedureadministeredtogroupstotheeffectivenessofthesame procedureadministeredtoindividuals. Theorysuggesteddeliveringtheprocedureto groupsshouldbemoreefficientandprovidemorehomogeneityintheadministration oftheprocedure.Thisstudyexemplifiesresearchdesignedspecificallytocompare identicaltreatmentsdeliveredtoindividualsandtoindividualsnestedingroups.Even thoughtheorysuggestedsubjectsparticipatingingroupswouldlikelybeinhibited. andconsequentlythemoodinductionprocedurenotaseffective,theresearchers hopedtofindnodifferenceintheeffectivenessoftheprocedures. o Researchingrouppsychotherapyalsoutilizesthegroupsversusindividuals researchdesign.Burlingame,Kircher,andTaylor(1994)reportedthatsevenmeta- analysesofgrouptherapyconductedinthe1980scomparedtheeffectivenessof grouptoindividuallydeliveredtherapy. Somestudiesinthemeta-analysesused treatmentsspecificallydesignedforindividualtreatmentwithbothindividualsand individualsnestedingroupsinordertocomparetreatmenteffectivenessandto endorsethecostbenefitandefficiencyofgroupdelivery.Otherstudiesused treatmentsspecificallydesignedtocapitalizeonthetheorizedeffectsofgroup behavior.Insummary,bothhistoricalandcurrentreviewsofresearchsupporttheuse andcontinueduseofgroupsversusindividualsresearchdesignsinbothefficacyand comparativestudiesoftreatmenteffectiveness. . '•> > MethodologicalConcerns CookandCampbell(1979)usedtheterm"statisticalconclusionvalidity"to expressconcernsaboutthevalidityofconclusionsformedfromstatistical relationshipsestablishedthroughdataanalysis.Proceduresthatfalselyhideorinflate statisticalrelationshipsraiseissuesofstatisticalconclusionvalidity. Lowstatistical validitycanresultfromeitherusingresearchdesignsthathaveinadequatestatistical powerorusingdesignsthatviolatetheassumptionsofthemathematicalmodels. Violatingtheassumptionofindependentobservationsacrossexperimentalunitsisa majorconcerningroupsversusindividualsresearchdesigns.