EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH (CERN) CERN-EP-2017-240 2017/10/08 CMS-BPH-15-008 Measurement of angular parameters from the decay √ B0 → K∗0µ+µ− in proton-proton collisions at s = 8TeV 7 1 The CMS Collaboration∗ 0 2 t c O 8 Abstract ] x e Angular distributions of the decay B0 → K∗0µ+µ− are studied using a sample of - √ p proton-proton collisions at s = 8TeV collected with the CMS detector at the LHC, e h corresponding to an integrated luminosity of 20.5fb−1. An angular analysis is per- [ formedtodeterminethe P and P(cid:48) parameters,wherethe P(cid:48) parameterisofparticu- 1 5 5 0 larinterestbecauseofrecentmeasurementsthatindicateapotentialdiscrepancywith 1 the standardmodel predictions. Based ona sample of 1397signal events, the P and 1 1 P(cid:48) parameters are determined as a function of the dimuon invariant mass squared. 1 5 3 Themeasurementsareinagreementwithpredictionsbasedonthestandardmodel. 0 2 / t i SubmittedtoPhysicsLettersB m b u s : v i X r a (cid:13)c 2017CERNforthebenefitoftheCMSCollaboration.CC-BY-4.0license ∗SeeAppendixAforthelistofcollaborationmembers 1 1 Introduction Phenomenabeyondthestandardmodel(SM)ofparticlephysicscanbecomemanifestdirectly, viatheproductionofnewparticles,orindirectly,bymodifyingtheproductionanddecayprop- ertiesofSMparticles. Analysesofflavor-changingneutralcurrentdecaysareparticularlysen- sitive to the effects of new physics because these decays are highly suppressed in the SM. An exampleisthedecayB0 → K∗0µ+µ−,whereK∗0 indicatestheK∗0(892)meson,withthecharge conjugate reaction implied here and elsewhere in this Letter unless otherwise stated. An an- gularanalysisofthisdecayasafunctionofthedimuoninvariantmasssquared(q2)allowsits propertiestobethoroughlyinvestigated. The differential decay rate for B0 → K∗0µ+µ− can be written in terms of q2 and three angular variablesasacombinationofsphericalharmonics,weightedby q2-dependentangularparam- eters. These angular parameters in turn depend upon complex decay amplitudes, which are described by Wilson coefficients in the relevant effective Hamiltonian [1]. There can be dif- ferent formulations of the angular parameters, in this Letter we present measurements of the so-called P and P(cid:48) [2,3]. 1 5 New physics can modify the values of these angular parameters [1, 2, 4–18] relative to the SM[1,19–25]. WhilepreviousmeasurementsofsomeoftheseparametersbytheBaBar, Belle, CDF, CMS, and LHCb experiments were found to be consistent with the SM predictions [26– 31],theLHCbCollaborationrecentlyreportedadiscrepancylargerthan3standarddeviations with respect to the SM predictions for the P(cid:48) parameter [32, 33], and the Belle Collaboration 5 reportedadiscrepancyalmostaslarge[34]. The new measurements presented in this Letter of the P and P(cid:48) angular parameters in B0 → 1 5 K∗0µ+µ− decays are performed using a sample of events collected in proton–proton (pp) col- lisions at a center-of-mass energy of 8TeV with the CMS detector at the CERN LHC. The data correspondtoanintegratedluminosityof20.5±0.5fb−1 [35]. TheK∗0 mesonisreconstructed throughitsdecaytoK+π−,andtheB0mesonbyfittingtoacommonvertexthetracksfromtwo oppositely charged muon candidates and the tracks from the K∗0 decay. The values of P and 1 P(cid:48) are measured by fitting the distributions of events as a function of three angular variables: 5 the angle between the µ+ and the B0 in the dimuon rest frame, the angle between the K+ and theB0intheK∗0 restframe,andtheanglebetweenthedimuonandtheKπ decayplanesinthe B0 restframe. Themeasurementsareperformedinthe q2 rangefrom1to19GeV2. Datainthe ranges 8.68 < q2 < 10.09GeV2 and 12.90 < q2 < 14.18GeV2 correspond to B0 → J/ψK∗0 and B0 → ψ(cid:48)K∗0 decays, respectively, and are used as control samples, since they have the same finalstateasthenonresonantdecaysofinterest. Here,ψ(cid:48) denotestheψ(2S)meson. CMSpreviouslyexploitedthesamedatasetusedinthisanalysistomeasuretwootherangular parameters in the B0 → K∗0µ+µ− decay as function of q2: the forward-backward asymmetry ofthemuons, A ,andtheK∗0 longitudinalpolarizationfraction, F ,aswellasthedifferential FB L branchingfraction[31]. Afterasimplificationofthetheoreticaldecayrateexpression,thispre- viousmeasurementwasperformedusingtwooutofthethreeangularvariables. Theanalysis presentedinthisLettershareswiththepreviousanalysis,togetherwiththedataset,thecriteria usedforselectingsignalevents,whicharereportedinSection3forcompleteness. 2 The CMS detector AdetaileddescriptionoftheCMSdetector, togetherwith thecoordinatesystemandthestan- dardkinematicvariables,canbefoundinRef.[36]. Themaindetectorcomponentsusedinthis 2 3 Reconstruction,eventselection,andefficiency analysisarethesilicontrackerandthemuondetectionsystems. Thesilicontracker,positioned within a superconducting solenoid that provides an axial magnetic field of 3.8T, consists of three pixel layers and ten strip layers (four of which have a stereo view) in the barrel region, accompanied by similar pixel and strip detectors in each endcap region, for a total pseudo- rapidity coverage of |η| < 2.5. For tracks with transverse momenta 1 < p < 10GeV and T |η| < 1.4, the resolutions are typically 1.5% in p and 25–90 (45–150)µm in the transverse T (longitudinal) impact parameter [37]. Muons are measured in the range |η| < 2.4 with detec- tion planes made using three technologies: drift tubes, cathode strip chambers, and resistive plate chambers. The probability for a pion, kaon, or proton to be misidentified as a muon is less than 2.5×10−3, 0.5×10−3, and 0.6×10−3, respectively, for p > 4GeV and |η| < 2.4. T The muon identification efficiency is greater than 0.80 (0.98) for p > 3.5GeV and |η| < 1.2 T (1.2 < |η| < 2.4) [38]. In addition to the tracker and muon detectors, CMS is equipped with electromagneticandhadroniccalorimeters. Eventsareselectedusingatwo-leveltriggersystem[39]. Thefirstlevelconsistsofspecialized hardware processors that use information from the calorimeters and muon systems to select events of interest at a rate of around 90kHz. A high-level trigger processor farm further de- creasestheeventratetolessthan1kHzbeforedatastorage. 3 Reconstruction, event selection, and efficiency Thecriteriausedtoselectthecandidateeventsduringdatataking(trigger)andafterfullevent reconstruction(offline)makeuseoftherelativelylonglifetimeofB0mesons,whichleadsthem to decay an average of about 1mm from their production point. The trigger uses only muon information to select events, while the offline selection includes the full reconstruction of all decayproducts. All events used in this analysis were recorded with the same trigger, requiring two identi- fied muons of opposite charge to form a vertex that is displaced from the pp collision region (beamspot). Multipleppcollisionsinthesameornearbybeamcrossings(pileup)causemulti- pleverticesinthesameevent. Thebeamspotposition(most-probablecollisionpoint)andsize (theextentoftheluminousregioncovering68%ofthecollisionsineachdimension)werecon- tinuouslymeasuredthroughGaussianfitstoreconstructedpileupverticesaspartoftheonline dataqualitymonitoring. Thetriggerrequiredeachmuontohave p > 3.5GeV, |η| < 2.2,and T topasswithin2cmofthebeamaxis. Thedimuonsystemwasrequiredtohave p > 6.9GeV,a T vertexfitχ2probabilitylargerthan10%,andaseparationofthevertexrelativetothebeamspot inthetransverseplaneofatleast3σ,whereσ includesthecalculateduncertaintyinthevertex position and the measured size of the beamspot. In addition, the cosine of the angle in the transverseplanebetweenthedimuonmomentumvectorandthevectorfromthebeamspotto thedimuonvertexwasrequiredtobegreaterthan0.9. The offline reconstruction requires at least two oppositely charged muons and at least two oppositely charged hadrons. The muons are required to match those that triggered the event. Thematchingisperformedbyrequiringanofflinemuontomatchatrigger-levelmuonwithin √ ∆R = (∆η)2+(∆φ)2 < 0.1, where ∆η and ∆φ are the pseudorapidity and azimuthal angle differences,respectively,betweenthedirectionsofthetrigger-levelandofflinemuons. Offline muons must, in addition, satisfy general muon identification requirements. For example, the muon track candidate from the silicon tracker must match a track segment from the muon detector, the χ2 per degree of freedom in a global fit to the silicon tracker and muon detector hits must be less than 1.9, there must be at least six silicon tracker hits, including at least two 3 fromthepixeldetector,andthetransverse(longitudinal)impactparameterwithrespecttothe beamspot must be less than 3 (30)cm. The dimuon system at the offline level is required to satisfythesamerequirementsasspecifiedaboveforthetriggerlevel. Thechargedhadroncandidatesarerequiredtofailthemuonidentificationcriteria,have p > T 0.8GeV, and an extrapolated distance d of closest approach to the beamspot in the transverse planegreaterthantwicethesuminquadratureoftheuncertaintyindandthebeamspottrans- versesize. Foratleastoneofthetwopossibleidentityassignments—thatthepositivelycharged hadronisakaonandthenegativelychargedhadronapion,orviceversa—theinvariantmassof thehadronpairmustliewithin90MeVofthenominalK∗0mass[40]. Toremovecontamination fromφ(1020) → K+K−decays,wetemporarilyassignthekaonmasstobothchargedhadrons, andtheneliminatethecandidateiftheresultinginvariantmassofthehadronpairislessthan 1.035GeV. TheB0 candidatesareobtainedbyfittingthefourchargedtrackstoacommonver- tex,andapplyingavertexconstrainttoimprovetheresolutionofthetrackparameters. TheB0 candidates must have p > 8GeV, |η| < 2.2, vertex fit χ2 probability larger than 10%, vertex T transverseseparation L fromthebeamspotgreaterthan12timesthesuminquadratureofthe uncertaintyin Landthebeamspottransversesize,andcosα > 0.9994,whereα istheangle xy xy in the transverse plane between the B0 momentum vector and the line-of-flight between the beamspotandtheB0vertex. TheinvariantmassmoftheB0candidatemustliewithin280MeV of the nominal B0 mass (m ) [40] for either the K−π+µ+µ− or K+π−µ+µ− possibility. The B0 selection criteria are optimized using signal event samples from simulation and background event samples from sideband data in m. The sideband includes both a low- and a high-mass regionandisdefinedby3σ < |m−m | < 280MeV,whereσ istheaveragemassresolution m B0 m (≈45MeV)obtainedfromfittingasumoftwoGaussianfunctionswithacommonmeantosim- ulated signal events. After applying the selection criteria, about 5% of the events have more thanonecandidate. AsinglecandidateischosenbasedonthebestB0vertexχ2 probability. For each of the selected events, the dimuon invariant mass q and its uncertainty σ are calcu- q lated. We define B0 → J/ψK∗0 and B0 → ψ(cid:48)K∗0 control samples through the requirements |q − mJ/ψ| < 3σq and |q − mψ(cid:48)| < 3σq, respectively, where mJ/ψ and mψ(cid:48) are the nominal masses[40]oftheindicatedmeson. Theaveragevalueofσ isabout26MeV. q TheremainingeventsamplestillcontainscontributionsfromB0 → J/ψK∗0 andB0 → ψ(cid:48)K∗0 de- cays,mainlybecauseofunreconstructedsoftphotonsinthecharmoniumdecay,i.e.,J/ψorψ(cid:48) → µ+µ−γ. These events have a low value of q and fall outside the control sample selection de- scribed above. They also have a low value of m and can be selectively removed using a com- binedrequirementonqandm. Forq < m (q > m ),werequire|(m−m )−(q−m )| > J/ψ J/ψ B0 J/ψ 160(60)MeV. For q < mψ(cid:48) (q > mψ(cid:48)), we require |(m−mB0)−(q−mψ(cid:48))| > 60(30)MeV. Using Monte Carlo (MC) simulation, these requirements were set so that less than 10% of the backgroundeventsoriginatefromthecontrolchannels. The selection criteria are such that they do not depend upon the choice of the primary vertex, and their optimization procedure makes use of both MC simulated signal events generated withthesamepileupdistributionasindata,andsidebanddata. Afterapplyingtheserequire- ments,3191eventsremain. The selected four-track vertex is identified as a B0 or B0 candidate depending on whether the K+π− or K−π+ invariant mass is closest to the nominal K∗0 mass. The fraction of candidates assignedtotheincorrectstateisestimatedfromsimulationtobe12–14%,dependingonq2. Theglobalefficiency, (cid:101), istheproductoftheacceptanceandthecombinedtrigger, reconstruc- tion, and selection efficiencies, all of which are obtained from MC simulated event samples. 4 3 Reconstruction,eventselection,andefficiency TheppcollisionsaresimulatedusingthePYTHIA[41]eventgenerator,version6.424,withpar- ticledecaysdescribedbythe EVTGEN [42]generator,version9.1,inwhichfinal-stateradiation is generated using PHOTOS [43]. The default matrix element in PYTHIA is used to describe the events. The simulated particles are propagated through a detailed model of the detector basedonGEANT4[44]. Thereconstructionandselectionofthegeneratedeventsproceedasfor the data. Separate samples of events are generated for B0 decays to K∗0µ+µ− → K+π−µ+µ−, J/ψ K∗0 → µ+µ−K+π−, and ψ(cid:48) K∗0 → µ+µ−K+π−. Thedistributionofppcollisionverticesin eachsampleisadjustedtomatchtheobserveddistribution. The acceptance is obtained from generator-level events, i.e., before the particle propagation with GEANT4, and is defined as the fraction of events with pT(B0) > 8GeV and |η(B0)| < 2.2 that satisfy the single-muon requirements p (µ) > 3.3GeV and |η(µ)| < 2.3. These criteria T arelessrestrictivethanthefinalselectioncriteriainordertoaccountforfinitedetectorresolu- tion,sincetheyareappliedtogenerator-levelquantities. Onlyeventssatisfyingtheacceptance criteria are processed through the GEANT4 simulation, the trigger simulation, and the recon- structionsoftware. Thecombinedtrigger,reconstruction,andselectionefficiencyisgivenbytheratioofthenum- ber of events that satisfy the trigger and selection requirements and have a reconstructed B0 candidatecompatiblewithageneratedB0 meson, relativetothenumberofeventsthatsatisfy the acceptance criteria. The generated and reconstructed B0 are considered to be compatible if the reconstructed K+ candidate appears within a distance ∆R of the generated K+ meson, and analogously for the π−, µ+, and µ−, where ∆R = 0.3 for the hadrons and ∆R = 0.004 for the muons. Requiring all four particles in the B0 decay to be matched results in an efficiency of 99.6% (0.4% of the events have a correctly reconstructed B0 candidate that is not matched toageneratedB0 meson)andapurityof99.5%(0.5%ofthematchedcandidatesdonotcorre- spondtoacorrectlyreconstructedB0candidate). Efficienciesaredeterminedforbothcorrectly tagged(theKandπhavethecorrectcharge)andmistagged(theKandπchargesarereversed) candidates. Using simulation, we search for possible backgrounds that might peak in the B0 mass region. TheeventselectionisappliedtoinclusiveMCsamplesofB0,B ,B+,andΛ decaystoJ/ψXand s b ψ(cid:48)X,whereXdenotesallpossibleSMparticlesrequiredtocompletetheknownexclusivedecay channels,andwiththeJ/ψandψ(cid:48) decayingtoµ+µ−. Noevidenceforapeakingstructurenear the B0 mass is found. The distributions of the few events that satisfy the selection criteria are similar to the shape of the combinatorial background. As an additional check, we generate events with B → K∗0(K+π−)µ+µ− decays, using the same branching fraction as for B0 → s K∗0(K+π−) µ+µ−. About 70 such events, integrated over q2, are found to cluster near the B mass. This background is considered negligible since it should be rescaled by the ratio of s branchingfractionsB(B → J/ψK∗0)/B(B0 → J/ψK∗0) ≈ 10−2 [40]. Possiblebackgroundsfrom s eventswithhadronsmisidentifiedasmuonsorwithmuonsmisidentifiedashadrons,e.g.,from randomD mesonsassociatedwithrandomstablechargedhadronsorfromB0 → DXorB0 → J/ψK∗decays,areconsiderednegligiblebecauseofthegoodmuonidentificationcapabilitiesof theCMSdetector[38]. WealsoinvestigatedpossiblebackgroundfromeventsinwhichaB+ → K+µ+µ−decayiscombinedwitharandompion,andfromeventswithaΛ → pKµ+µ−decay b in which the proton is assigned the pion mass. Both these potential sources of background are found to be negligible. The low-mass sideband might be affected by background events that have a different origin with respect to the combinatorial background that characterizes the signal region, e.g., partially reconstructed multibody B decays. We address this possible backgroundcontaminationinSection5. 5 4 Analysis method This analysis measures the P and P(cid:48) values in B0 → K∗0µ+µ− decays as a function of q2. Fig- 1 5 ure 1 illustrates the angular variables needed to describe the decay: θ is the angle between (cid:96) the positive (negative) muon momentum and the direction opposite to the B0 (cid:0)B0(cid:1) momen- tum in the dimuon rest frame, θ is the angle between the kaon momentum and the direction K opposite to the B0 (cid:0)B0(cid:1) momentum in the K∗0 (cid:0)K∗0(cid:1) rest frame, and ϕ is the angle between the plane containing the two muons and the plane containing the kaon and the pion in the B0 rest frame. Although the K+π− invariant mass is required to be consistent with that of a K∗0 meson,therecanbeacontributionfromspinless(S-wave)K+π−combinations[25,45–47]. Thisis parametrizedwiththree terms: F , which isrelated totheS-wave fraction, and A and S S A5,whicharetheinterferenceamplitudesbetweentheS-andP-wavedecays. Includingthese S components,theangulardistributionofB0 → K∗0µ+µ−decayscanbewrittenas[25]: 1 d4Γ = 9 (cid:26)2 (cid:20)(F +A cosθ )(cid:0)1−cos2θ (cid:1) dΓ/dq2dq2 dcosθ dcosθ dϕ 8π 3 S S K (cid:96) (cid:96) K (cid:21) (cid:112) (cid:112) + A5 1−cos2θ 1−cos2θ cosϕ S K (cid:96) (cid:104) +(1−F ) 2F cos2θ (cid:0)1−cos2θ (cid:1) S L K (cid:96) + 1 (1−F )(cid:0)1−cos2θ (cid:1)(cid:0)1+cos2θ (cid:1) L K (cid:96) 2 1 + P (1−F )(1−cos2θ )(1−cos2θ )cos2ϕ 1 L K (cid:96) 2 (cid:113) (cid:112) (cid:112) (cid:105)(cid:27) + 2P(cid:48) cosθ F (1−F ) 1−cos2θ 1−cos2θ cosϕ , 5 K L L K (cid:96) (1) where F denotes the longitudinal polarization fraction of the K∗0. This expression is an exact L simplification of the full angular distribution, obtained by folding the ϕ and θ angles about (cid:96) zeroandπ/2,respectively. Specifically,if ϕ < 0,then ϕ → −ϕ,andthenew ϕdomainis[0,π]. Ifθ > π/2,thenθ → π−θ ,andthenewθ domainis[0,π/2]. Weusethissimplifiedversion (cid:96) (cid:96) (cid:96) (cid:96) oftheexpressionbecauseofdifficultiesinthefitconvergencewiththefullangulardistribution duetothelimitedsizeofthedatasample. Thissimplificationexploitstheoddsymmetryofthe angular variables with respect to ϕ = 0 and θ = π/2 in such a manner that the cancellation (cid:96) aroundtheseangularvaluesisexact. Thiscancellationremainsapproximatelyvalidevenafter accountingfortheexperimentalacceptancebecausetheefficiencyissymmetricwithrespectto thefoldingangles. Foreachq2bin,theobservablesofinterestareextractedfromanunbinnedextendedmaximum- likelihood fit to four variables: the K+π−µ+µ− invariant mass m and the three angular vari- ablesθ ,θ ,and ϕ. Theunnormalizedprobabilitydensityfunction(pdf)ineachq2 binhasthe (cid:96) K followingform: (cid:20) pdf(m,θ ,θ ,ϕ) = YC SC(m)Sa(θ ,θ ,ϕ)(cid:101)C(θ ,θ ,ϕ) K (cid:96) S K (cid:96) K (cid:96) fM (cid:21) (2) + SM(m)Sa(−θ ,−θ ,ϕ)(cid:101)M(θ ,θ ,ϕ) 1− fM K (cid:96) K (cid:96) +YBBm(m)BθK(θK)Bθ(cid:96)(θ(cid:96))Bϕ(ϕ), 6 4 Analysismethod K+ φ K+ μ− μ− θ PB0 PB0 K θℓ μ+ μ+ μ+μ− rest frame π− B0 rest frame π− K*0 rest frame Figure 1: Illustration of the angular variables θ (left), θ (middle), and ϕ (right) for the decay (cid:96) K B0 → K∗0(K+π−)µ+µ−. where the three terms on the righthand side correspond to correctly tagged signal events, mistagged signal events, and background events. The parameters YC and Y are the yields S B of correctly tagged signal events and background events, respectively, and are determined in thefit. Theparameter fM isthefractionofsignaleventsthataremistaggedandisdetermined fromsimulation. Itsvaluerangesfrom0.124to0.137dependingontheq2 bin. ThesignalmassprobabilityfunctionsSC(m)andSM(m)areeachthesumoftwoGaussianfunc- tions,withacommonmeanforallfourGaussianfunctions,anddescribethemassdistribution for correctly tagged and mistagged signal events, respectively. In the fit, the mean, the four Gaussian function’s width parameters, and the two fractions specifying the relative contribu- tionof the twoGaussian functions in SC(m) and SM(m) are determined from simulation. The function Sa(θ ,θ ,ϕ) describes the signal in the three-dimensional (3D) space of the angular K (cid:96) variablesandcorrespondstoEq.(1). ThecombinationBm(m)BθK(θK)Bθ(cid:96)(θ(cid:96))Bϕ(ϕ)isobtained from the B0 sideband data in m and describes the background in the space of (m,θ ,θ ,ϕ), K (cid:96) where Bm(m)isanexponentialfunction, BθK(θK)and Bθ(cid:96)(θ(cid:96))aresecond-tofourth-orderpoly- nomials, depending on the q2 bin, and Bϕ(ϕ) is a first-order polynomial. The factorization assumption of the background pdf in Eq. (2) is validated by dividing the range of an angular variable into two at its center point and comparing the distributions of events from the two halvesintheotherangularvariables. Thefunctions(cid:101)C(θ ,θ ,ϕ)and(cid:101)M(θ ,θ ,ϕ)aretheefficienciesinthe3Dspaceof|cosθ | ≤ 1, K (cid:96) K (cid:96) K 0 ≤ cosθ ≤ 1, and0 ≤ ϕ ≤ π forcorrectlytaggedandmistaggedsignalevents, respectively. (cid:96) The numerator and denominator of the efficiency are separately described with a nonpara- metric technique, which is implemented with a kernel density estimator [48, 49]. The final efficiency distributions used in the fit are obtained from the ratio of 3D histograms derived from the sampling of the kernel density estimators. The histograms have 40 bins in each di- mension. A consistency check of the procedure used to determine the efficiency is performed bydividingthesimulateddatasampleintotwoindependentsubsets,andextractingtheangu- larparametersfromthefirstsubsetusingtheefficiencycomputedfromthesecondsubset. The efficiencies for both correctly tagged and mistagged events peak at cosθ ≈ 0, around which (cid:96) they are rather symmetric for q2 < 10GeV2, and are approximately flat in ϕ. The efficiency for correctly tagged events becomes relatively flat in cosθ for larger values of q2, while it has (cid:96) a monotonic decrease for increasing cosθ values for q2 < 14GeV2. For larger values of q2 a K decreaseintheefficiencyisalsoseennearcosθ = −1. Theefficiencyformistaggedeventshas K aminimumatcosθ ≈ 0forq2 > 10GeV2,whileitismaximalnearcosθ = 0forq2 < 10GeV2. (cid:96) K 7 Forlargevaluesofq2 amildmaximumalsoappearsnearcosθ = 1. K Thefitisperformedintwosteps. Theinitialfitdoesnotincludeasignalcomponentanduses the sideband data in m to obtain the Bm(m), BθK(θK), Bθ(cid:96)(θ(cid:96)), and Bϕ(ϕ) distributions. The distributions obtained in this step are then fixed for the second step, which is a fit to the data over the full mass range. The fitted parameters in the second step are the angular parameters P , P(cid:48), and A5, and the yields YC and Y . To avoid difficulties in the convergence of the fit 1 5 S S B related to the limited number of events, the angular parameters F , F , and A are fixed to L S S previousmeasurements[31]. The expression describing the angular distribution of B0 → K∗0µ+µ− decays, Eq. (1), and also itsmoregeneralforminRef.[25],canbecomenegativeforcertainvaluesoftheangularparam- eters. Inparticular,thepdfinEq.(2)isonlyguaranteedtobepositiveforaparticularsubsetof the P , P(cid:48), and A5 parameter space. The presence of such a boundary greatly complicates the 1 5 S numerical maximization process of the likelihood by MINUIT [50] and especially the error de- termination by MINOS [50], in particular near the boundary between physical and unphysical regions. Therefore,thesecondfitstepisperformedbydiscretizingthe P , P(cid:48) two-dimensional 1 5 space and by maximizing the likelihood as a function of the nuisance parameters YC, Y , and S B A5 at fixed values of P and P(cid:48). Finally, the distribution of the likelihood values is fit with a S 1 5 bivariate Gaussian distribution. The position of the maximum of this distribution inside the physicalregionprovidesthemeasurementsof P and P(cid:48). 1 5 The interference terms A and A5 must vanish if either of the two interfering components S S √ vanish. Theseconstraintsareimplementedbyrequiring|A | < 12F (1−F )F f and|A5| < √ S S S L S 3F (1−F )(1−F )(1+P ) f, where f is a ratio related to the S- and P-wave line shapes, S S L 1 calculated to be 0.89 near the K∗0 meson mass [25]. The constraint on A is naturally satisfied S since F , F ,and A aretakenfrompreviousmeasurements[31]. S L S Toensurecorrectcoveragefortheuncertainties,theFeldman–Cousinsmethod[51]isusedwith nuisance parameters. Two main sets of pseudo-experimental samples are generated. The first (second) set, used to compute the coverage for P (P(cid:48)), is generated by assigning values to the 1 5 otherparametersasobtainedbyprofilingthebivariateGaussiandistributiondescriptionofthe likelihooddeterminedfromdataatfixed P (P(cid:48))values. Whenfittingthepseudo-experimental 1 5 samples,thesamefitprocedureasappliedtothedataisused. The fit formalism and results are validated through fits to pseudo-experimental samples, MC simulationsamples,andcontrolchannels. Additionaldetails,includingthesizeofthesystem- aticuncertaintiesassignedonthebasisofthesefits,aredescribedinSection5. 5 Systematic uncertainties ThesystematicuncertaintystudiesaredescribedbelowandsummarizedinTable1inthesame order. Theadequacyofthefitfunctionandtheproceduretodeterminetheparametersofinterestare validated in three ways. First, a large, statistically precise MC signal sample with approxi- mately 400 times the number of events as the data is used to verify that the fitting procedure producesresultsconsistentwiththeinputvaluestothesimulation. Thedifferencebetweenthe input and output values in this check is assigned as a simulation mismodeling systematic un- certainty. Itisalsoverifiedthatfittingasamplewithonlyeithercorrectlytaggedormistagged eventsyieldsthecorrectresults. Second,200subsamplesareextractedrandomlyfromthelarge MC signal sample and combined with background events obtained from the pdf in Eq. (2) to 8 5 Systematicuncertainties mimic independent data sets of similar size to the data. These are used to estimate a fit bias by comparing the average values of the results obtained by fitting the 200 samples to the re- sultsobtainedusingthefullMCsignalsample. Muchoftheobservedbiasisaconsequenceof the fitted parameters lying close to the boundaries of the physical region. Third, 200 pseudo- experiments,eachwiththesamenumberofeventsasthedatasample,aregeneratedineachq2 binusingthepdfinEq.(2),withparametersobtainedfromthefittothedata. Fitstothese200 samplesdonotrevealanyadditionalsystematicuncertainty. Table1: SystematicuncertaintiesinP andP(cid:48). Foreachsource,therangeindicatesthevariation 1 5 overthebinsinq2. Source P (×10−3) P(cid:48)(×10−3) 1 5 Simulationmismodeling 1–33 10–23 Fitbias 5–78 10–120 Finitesizeofsimulatedsamples 29–73 31–110 Efficiency 17–100 5–65 Kπ mistagging 8–110 6–66 Backgrounddistribution 12–70 10–51 Massdistribution 12 19 Feed-throughbackground 4–12 3–24 F , F , A uncertaintypropagation 0–210 0–210 L S S Angularresolution 2–68 0.1–12 Total 100–230 70–250 Because the efficiency functions are estimated from a finite number of simulated events, there isacorrespondingstatisticaluncertaintyintheefficiency. Alternativestothedefaultefficiency functionareobtainedbygenerating100newdistributionsforthenumeratorandthedenomi- natoroftheefficiencyratiobasedonthedefaultkerneldensityestimatorsaspdfs,andrederiv- ingnewkerneldensityestimatorsforeachtrial. Theeffectofthesedifferentefficiencyfunctions onthefinalresultisusedtoestimatethesystematicuncertainty. The efficiency determination is checked by comparing efficiency-corrected results obtained from the control channels with the corresponding world average values. The B0 → J/ψK∗0 control sample contains 165000 events, compared with 11000 events for the B0 → ψ(cid:48)K∗0 sample. Because of its greater statistical precision, we rely on the B0 → J/ψK∗0 sample to perform the check of the efficiency determination for the angular variables. We do this by measuring the longitudinal polarization fraction F in the B0 → J/ψK∗0 decays. We find L F = 0.537±0.002(stat),comparedwiththeworldaveragevalue0.571±0.007(stat+syst)[40]. L The difference of 0.034 is propagated to P and P(cid:48) by taking the root-mean-square (RMS) 1 5 of the respective distributions resulting from refitting the data 200 times, varying F within L a Gaussian distribution with a standard deviation of 0.034. As a cross-check that the effi- ciency is not affected by a q2-dependent offset, we measure the ratio of branching fractions B(B0 → ψ(cid:48)K∗0)/B(B0 → J/ψK∗0) = 0.480±0.008 (stat)±0.055 (Rµµ), bymeansofefficiency- ψ correctedyieldsincludingbothcorrectlyandwronglytaggedevents(thesamecentralvalueis obtained also separately for the two subsets of events), where Rµµ refers to the ratio B(J/ψ → ψ µ+µ−)/B(ψ(cid:48) → µ+µ−) of branching fractions. This is compared to the world average value 0.484±0.018(stat)±0.011(syst)±0.012(Ree) [40], where Ree refers to the corresponding ratio ψ ψ ofbranchingfractionstoe+e−. Thetworesultsareseentoagreewithintheuncertainties. To evaluate the uncertainty in the mistag fraction fM, we allow this fraction to vary in a fit to the events in the B0 → J/ψK∗0 control sample. We find fM = (14.5±0.5)%, compared to