Fisher consistency for prior probability shift Dirk Tasche∗ We introduce Fisher consistency in the sense of unbiasedness as a criterion to distinguish potentiallysuitableandunsuitableestimatorsofpriorclassprobabilitiesintestdatasetsunder prior probability and more general dataset shift. The usefulness of this unbiasedness concept 7 is demonstrated with three examples of classifiers used for quantification: Adjusted Classify 1 & Count, EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and EM- 0 algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is not Fisher 2 consistentand,therefore,cannotbetrustedtodeliverreliableestimatesofclassprobabilities. n a Keywords: Classification, quantification, class distribution estimation, Fisher consistency. J 9 1 1. Introduction ] L Theapplicationofaclassifiertoatestdatasetforpredictingtheclasslabelsofthetestinstancesoftenis M basedontheassumptionthatthedatafortrainingtheclassifierwasrepresentativeofthetestdata.While . t this assumption might be true sometimes or even most of the time, there may be circumstances when a t the distributions of the classes or the features or both in the test set and the training set are genuinely s [ different.Spamemailsrepresentafamiliarexampleofthissituation:Thetypicalcontentsofspamemails and their proportion of the total daily number of emails received may significantly vary over time. Spam 1 v email filters developed on the set of emails received last week may become worthless this week due to 2 change in the composition of the email traffic. In the machine learning community, this phenomenon is 1 called dataset shift or population drift. The problem of how to learn a model appropriate for a test set 5 on a differently composed training set is called domain adaptation. 5 0 The simplest type of dataset shift occurs when training set and test set differ only in the distribution . 1 of the classes of the instances. As in this case only the prior (or unconditional) class probabilities (or 0 class prevalences) change between the training set and the test set, this type of dataset shift is called 7 1 prior probability shift. In the context of supervised or semi-supervised learning, the class prevalences : in the training set are always known. In contrast, the class prevalences in the test set may be known or v i unknown at the time of the application of the classifier to the test set, depending on the cause of the X dataset shift. For example, in a binary classification exercise the class prevalence of the majority class in r a the training set might deliberately be reduced by removing instances of that class at random, in order to facilitate the training of the classifier. If the original training set were a random sample of the test set then the test set class prevalences would be equal to the original training set class prevalences and therefore known. The earlier mentioned spam email filter problem represents an example of the situation where the test set class prevalences are unknown due to possible dataset shift. ∗E-mail:[email protected] TheauthorcurrentlyworksattheSwissFinancialMarketSupervisoryAuthority(FINMA).Theopinionsexpressedin thisnotearethoseoftheauthoranddonotnecessarilyreflectviewsofFINMA. 1 Inthispaper,weprimarilystudythequestionofhowtoestimatetheunknownclassprevalencesinatest dataset when the training set and the test set are related by prior probability shift. This problem was coined quantification by Forman (2008). Solutions to the problem are known at least since the 1960s (Gart and Buck, 1966) but research for more and better solutions has been ongoing ever since. It seems, however, as if the criteria of what should be deemed a ‘good’ solution are not fully clear. For this reason, we argue that as a minimum it should be required that class prevalence estimators (or quantifiers) be unbiased or, more precisely, Fisher consistent. Accordingly, focussing on binary classification and quantification, we pursue two goals with this paper: • We introduce the concept of unbiasedness expressed as Fisher consistency, in a prior probability shiftcontextandmoregenerallyforquantificationofclassprevalencesinthepresenceofanydataset shift. • WeillustratetheusefulnessofthenotionofFisherconsistency,bydemonstratingwiththreepopular quantificationapproachesthatFisherconsistencymaybeusedasacriteriontocullinaptquantifiers. Inotherwords,Fisherconsistencycanserveasan‘elktest’(see,e.g.,Wikipediacontributors,2017) for quantifiers. As a further contribution to quantification research, we suggest a new type of dataset shift, called ‘in- variant density ratio’-type dataset shift, which extends prior probability shift and is conceptually close to covariate shift. We also propose a method for generating non-trivial examples of this type of dataset shift. The plan for the paper is as follows: • In Section 2, we recall the concepts from binary classification and quantification that are relevant for the subject of the paper. These concepts include Bayes classifiers and some types of dataset shift, including prior probability shift. In addition, we motivate and define the notion of Fisher consistency. • In Section 3, we describe three approaches to class prevalence estimation which serve to illustrate the role of Fisher consistency for this task. The three approaches are Adjusted Classify & Count (Forman, 2008), the EM-algorithm (Saerens et al., 2001) and CDE-Iterate (Xue and Weiss, 2009). • In Section 4, we explore by numerical examples under which circumstances the three approaches discussed in Section 3 cease to be Fisher consistent for the estimation of class prevalences. The most important finding here is that CDE-Iterate is not Fisher consistent under prior probability shift. Hence there is no guarantee that CDE-Iterate will find the true class prevalences even in the presence of that quite benign type of dataset shift. • We conclude in Section 5 with some comments on the findings of the paper. • Appendix A presents some tables with computation results for additional information while Ap- pendixBprovidesamathematicallyrigorousderivationoftheequationthatcharacterisesthelimit of CDE-Iterate. 2. Estimating prior probabilities In this section, we recall some basic concepts from the theory of binary classification and quantification to build the basis for the discussion of prevalence estimation approaches in Section 3 and the numerical examples in Section 4. The concepts discussed include Bayes classifiers and different types of dataset shift including prior probability shift and its extension called here ‘invariant density ratio’-type dataset shift.Inaddition,weintroducethenotionofunbiasednessofprevalenceestimatorsintheshapeofFisher consistency which is most appropriate in the quantification context central for this paper. 2 2.1. Binary classification and Bayes error at population level The basic model we consider is a random vector (X,Y) with values in a product X ×{0,1} of sets. Typically, it holds that X = Rd for some positive integer d. A realisation of X ∈ X is interpreted as the vector of features of an observed instance. Y ∈ {0,1} is the class of the observed instance. For the purpose of this paper, 0 is the interesting (positive) class, as in Hern´andez-Orallo et al. (2012). Classification problem. On the basis of the observed features X of an instance, make a guess (pre- diction) g(X) ∈ {0,1} of the instance’s class Y such that the probability of an error is minimal. In mathematical terms: Find g∗ :X →{0,1} such that P[g∗(X)(cid:54)=Y] = minP[g(X)(cid:54)=Y]. (2.1a) g The functions g in (2.1a) used for predicting Y are called (crisp) classifiers. The value of the minimum in (2.1a) is called Bayes error. (2.1a) accounts for two possibilities to make classification errors: ‘Predict 0 if the true class is 1’={g(X)=0,Y =1}, and ‘Predict 1 if the true class is 0’={g(X)=1,Y =0}. Cost-sensitive errors. In practice, the consequences of these two erroneous predictions might have different severities. Say the cost related to ‘predict 0 if the true class is 1’ is c ≥0, and the cost related 1 to ‘predict 1 if the true class is 0’ is c ≥0. To deal with the issue of different severities, a cost-sensitive 0 version of (2.1a) can be studied: c P[g∗(X)=0, Y =1]+c P[g∗(X)=1, Y =0]= 1 0 minc P[g(X)=0, Y =1]+c P[g(X)=1, Y =0]. (2.1b) 1 0 g To make this problem non-trivial, of course one has to assume that c +c >0. 0 1 Bayes classifier. A solution g∗ to (2.1b) and therefore also to (2.1a) (case of c =c =1) exists and is 0 1 well-known (see Section 2.2 of van Trees (1968) or Section 1.3 of Elkan (2001)): (cid:40) 0, if P[Y =0|X]> c0 , g∗(X) = c0+c1 (2.2) 1, if P[Y =0|X]≤ c0 . c0+c1 In this equation, P[Y = 0|X] denotes the (non-elementary) conditional probability of the event Y = 0 givenX,asdefinedinstandardtextbooksonstatisticallearningandprobabilitytheory(seeAppendixA.7 ofDevroyeetal.(1996)andSection4.1ofDurrett(1996)respectively).BeingafunctionofX,P[Y =0|X] is also a non-constant random variable whenever X and Y are not stochastically independent. In the following, we also call P[Y = 0|X] feature-conditional class probability. The function g∗(X) as defined in (2.2) is called Bayes classifier. Aproofof (2.2)is alsoprovidedinAppendixBbelow(seeLemmaB.2).Thatproofshows,inparticular, that the solution g∗ to (2.1b) is unique in the sense of P[g∗(X)=g˜(X)]=1 for any other minimiser g˜of (2.1b), as long as the distribution of the ratio of the class-conditional feature densities is continuous (see Section 2.4 for the definition of the density ratio). 2.2. Binary classification and Bayes error at sample level In theory, the binary classification problem with cost-sensitive errors is completely solved in Section 2.1. In practice, however, there are issues that can make the Bayes classifier (2.2) unfeasible: 3 • Typically, the joint probability distribution of (X,Y) is not exactly known but has to be inferred from a finite sample (called ‘training set’) (x ,y ),...,(x ,y )∈X ×{0,1}. If X is high- 1,tr 1,tr m,tr m,tr dimensional this may be difficult and require a large sample size to reduce errors due to random variation. • As the Bayes classifier is explicitly given by (2.2), one can try and avoid estimating the entire distribution of (X,Y) and, instead, only estimate the feature-conditional probability P[Y = 0|X] (alsocalledposteriorprobability).However,thistaskisnotsignificantlysimplerthanestimatingthe distributionof(X,Y)–asillustratedbythefactthatmethodsfortheestimationofnon-elementary conditional probabilities constitute a major branch of applied statistics. Bystructuralassumptionsonthenatureoftheclassificationproblem(2.1b),theapproachbasedondirect estimation of the feature-conditional probability can be rendered more accessible. Logistic regression provides the possibly most important example for this approach (see, e.g., Cramer, 2003). But the price of the underlying assumptions may be high and include significant deterioration of goodness of fit. That is why alternative approaches based on direct implementation of the optimisation of the right-hand side of (2.1b) are popular. Lessmann et al. (2015) give a survey of the variety of methods available, just for application to credit scoring. 2.3. Dataset shift Even if one has succeeded in estimating a Bayes classifier or at least a reasonably good approximate Bayes classifier, issues may arise that spoil its effective deployment. Quite often, it is assumed that any instance with known features vector x but unknown class y that is presented for classification has been drawn at random from the same population as the training set. However, for a number of reasons this assumptioncanbewrong(see,e.g.,Quin˜onero-Candelaetal.,2009;KullandFlach,2014;Moreno-Torres et al., 2012; Dal Pozzolo et al., 2015). A lot of research has been undertaken and is ongoing on methods todealwiththisproblemofso-calleddatasetshift.Inthispaper,weconsiderthefollowingvariantofthe problem: • There is a training dataset (x ,y ),...,(x ,y ) ∈ X ×{0,1} which is assumed to be an 1,tr 1,tr m,tr m,tr independent and identically distributed sample from the population distribution P(X,Y) of the random vector (X,Y) as described in Section 2.1. • There is another dataset, the test dataset, (x ,y ),...,(x ,y ) ∈ X ×{0,1} which is as- 1,te 1,te n,te n,te sumedtobeanindependentandidenticallydistributedsamplefromapossiblydifferentpopulation distribution Q(X,Y). Moreover, training and test data have been independently generated. • For the instances in the training dataset, their class labels are visible and can be made use of for learning classifiers (i.e. solving optimisation problem (2.1b)). • For the instances in the test dataset, their class labels are invisible to us or become visible only with large delay. This assumption implies that test instances arrive batch-wise, not as a stream. We also assume that the sizes m of the training set and n of the test set are reasonably large such that trying to infer properties of the respective population distributions makes sense. Asfarasthetheoryforthispaperisconcerned,wewillignoretheissuescausedbythefactthatweknow the training dataset distribution P(X,Y) and the test dataset distribution Q(X,Y) only by inference from finite samples. Instead we will assume that we can directly deal with the population distributions P(X,Y)andQ(X,Y).Throughoutthewholepaper,wemaketheassumptionthattherearebothpositive and negative instances in both of the populations, i.e. it holds that 0<P[Y =0]<1 and 0<Q[Y =0]<1. (2.3) 4 The problem that P(X,Y) and Q(X,Y) may not be the same is known under different names in the literature (Moreno-Torres et al., 2012): dataset shift, domain adaptation, population drift and others. There are several facets of the problem: • Classifiers may have to be adapted to the test set or re-trained. • The feature-conditional probabilities may have to be adapted to the test set or re-estimated. • The unconditional class probabilities (also called prior probabilities or prevalences) may have to be re-adjusted for the test set or re-estimated. Quantification. In this paper, we focus on the estimation of the prevalences Q[Y = 0] and Q[Y = 1] in the test set, as parameters of distribution Q(X,Y). This problem is called quantification (Forman, 2008) and of own interest beyond its auxiliary function for classification and estimation of the feature- conditional probabilities (Gonz´alez et al., 2016). 2.4. Quantification in the presence of prior probability shift In technical terms, the quantification problem as presented in Section 2.3 can be described as follows: • Weknowthejointdistributionoffeaturesandclasslabels(X,Y)underthetrainingsetprobability distribution P. • We know the distribution of the features X under the test set probability distribution Q. • How can we infer the prevalences of the classes under Q, i.e. the probabilities Q[Y = 0] and Q[Y =1]=1−Q[Y =0], by making best possible use of our knowledge of P(X,Y) and Q(X)? ThisquestioncannotbeansweredwithoutassumingthatthetestsetprobabilitydistributionQ‘inherits’ some propertiesof the training set probabilitydistribution P. This means tomakemore specific assump- tionsaboutthestructureofthedatasetshiftbetweentrainingsetandtestset.Intheliterature,avariety of different types of dataset shift have been discussed. See Moreno-Torres et al. (2012), Kull and Flach (2014) or Hofer (2015) for a number of examples, including ‘covariate shift’ and ‘prior probability shift’ which possibly are the two most studied types of dataset shift. This paper focusses on prior probability shift. Prior probability shift.Thistypeofdatasetshiftalsohasbeencalled‘globalshift’(HoferandKrempl, 2013). The assumption of prior probability shift is, in particular, appropriate for circumstances where the features of an instance are caused by the instance’s class membership (Fawcett and Flach, 2005). Technically, prior probability shift can be described as ‘the class-conditional feature distributions of the training and test sets are the same’, i.e. Q[X ∈S|Y =0]=P[X ∈S|Y =0] and Q[X ∈S|Y =1]=P[X ∈S|Y =1], (2.4) for all measurable sets S ⊂X. Note that (2.4) does not imply Q[X ∈S]=P[X ∈S] for all measurable S ⊂X because the training set class distribution P(Y) and the test set class distribution Q(Y) still can be different. In this paper, we will revisit three approaches to the estimation of the prevalences in population Q in the presence of prior probability shift as defined by (2.4). Specifically, we will check both in theory and by simple examples if these three approaches satisfy the basic estimation quality criterion of Fisher consistency. For this purpose, a slight generalisation of prior probability shift called ‘invariant density ratio’-type dataset shift will prove useful. Before we introduce it, let us briefly recall some facts on conditional probabilities and probability densities. Feature-conditional class probabilities and class-conditional feature densities. Typically, the class-conditional feature distributions P(X|Y = i), i = 0,1, of a dataset have got densities f and f 0 1 5 respectively with respect to some reference measure like the d-dimensional Lebesgue measure. Then also the unconditional feature distribution P(X) has a density f which can be represented as f(x) = P[Y =0]f (x)+(1−P[Y =0])f (x) for all x∈X. (2.5a) 0 1 Moreover,thefeature-conditionalclassprobabilityP[Y =0|X]canbeexpressedintermsofthedensities f and f : 0 1 P[Y =0]f (x) P[Y =0|X](x) = 0 , x∈X. (2.5b) P[Y =0]f (x)+(1−P[Y =0])f (x) 0 1 Conversely, assume that there is a density f of the unconditional feature distribution P(X) and the feature-conditional class probability P[Y = 0|X] is known. Then the class-conditional feature densities f and f are determined as follows: 0 1 P[Y =0|X](x) f (x)= f(x), x∈X, 0 P[Y =0] (2.6) 1−P[Y =0|X](x) f (x)= f(x), x∈X. 1 1−P[Y =0] Invariant density ratio. Assume that there are densities f and f for the class-conditional feature 0 1 distributionsP(X|Y =i),i=0,1,ofthetrainingsetpopulation.Wethensaythatthedatasetshiftfrom population P to Q is of ‘invariant density ratio’-type if there are also densities h and h respectively for 0 1 the class-conditional distributions Q(X|Y =i), i=0,1, of the test set, and it holds that f (x) h (x) 0 = 0 for all x∈X. (2.7) f (x) h (x) 1 1 Note that by (2.6), the density ratio f0 can be rewritten as f1 f (x) P[Y =0|X](x) 1−P[Y =0] 0 = , x∈X, (2.8) f (x) 1−P[Y =0|X](x) P[Y =0] 1 aslongasP[Y =0|X](x)<1.Hencethedensityratiocanbecalculatedwithoutknowledgeofthevalues of the densities. Obviously, ‘invariant density ratio’ is implied by prior probability shift if all involved class-conditional distributions have got densities. ‘Invariant density ratio’-type dataset shift was discussed in some detail by Tasche (2014). It is an interesting type of dataset shift for several reasons: 1) ‘Invariant density ratio’ extends the concept of prior probability shift. 2) Conceptually,‘invariantdensityratio’isclosetocovariateshiftwhichmaybedefinedbytheproperty that the feature-conditional class probabilities of the training and test sets are the same (Moreno- Torres et al., 2012, Section 4.1): P[Y =0|X] = Q[Y =0|X]. (2.9a) Iftherearedensitiesoftheclass-conditionalfeaturedistributionsunderPandQ,likefor(2.7),this can be rewritten as P[Y =0] f (x) Q[Y =0] h (x) 0 = 0 for all x∈X. (2.9b) 1−P[Y =0] f (x) 1−Q[Y =0] h (x) 1 1 Hence, covariate shift also can be described in terms of the ratio of the class-conditional densities, like ‘invariant density ratio’. The two types of dataset shift coincide if P[Y =0]=Q[Y =0], i.e. if the class prevalences in the training and test sets are the same. 6 3) The maximum likelihood estimates of class prevalences under prior probability shift actually are maximumlikelihoodestimatesunder‘invariantdensityratio’too,andhencecanbecalculatedwith the EM (expectation-maximisation) algorithm as described by Saerens et al. (2001). 4) In terms of the dataset shift taxonomy of Moreno-Torres et al. (2012), ‘invariant density ratio’ is a non-trivial but manageable ‘other’ shift, i.e. it is neither a prior-probability shift, nor a covariate shift, nor a concept shift. By properties 1) and 2), ‘invariant density ratio’ is in the neighbourhood of both prior probability shift and covariate shift. In Section 4 below, we will make use of its manageability property 4) to construct an example of dataset shift that reveals the limitations of some common approaches to the estimation of class prevalences. 2.5. Fisher consistency as important requirement The Wikipedia summary of Fisher consistency is that “in statistics, Fisher consistency, named after Ronald Fisher, is a desirable property of an estimator asserting that if the estimator were calculated using the entire population rather than a sample, the true value of the estimated parameter would be obtained” (Wikipedia contributors, 2016). In this sense, we apply the notion of Fisher consistency to quantification. In practice, estimators are often Fisher consistent and asymptotically consistent (weakly or strongly, see Section 10.4 of van der Vaart, 1998, for the definitions) at the same time. Gerow (1989) discusses the questions of when this is the case and what concept Fisher (1922) originally defined. Our focus on Fisher consistencyisnotmeanttoimplythatasymptoticconsistencyisalessimportantproperty.Inthecontext of statistical learning, asymptotic consistency seems to have enjoyed quite a lot of attention, as shown for instance by the existence of books like Devroye et al. (1996). In addition, the convergence aspect of asymptotic consistency often can be checked empirically by observing the behaviour of large samples. However, in some cases it might be unclear if the limit of a seemingly asymptotically consistent large sample is actually the right one. This is why, thanks to its similarity to the concept of unbiasedness, Fisher consistency becomes an important property. Definition 2.1 (Fisher consistency) In the dataset shift setting of this paper as described in Sec- tion 2.3, we say that an estimator T(Q), applied to the elements Q of a family Q of possible population distributions of the test set, is Fisher consistent in Q for the prevalence of class 0 if it holds that T(Q) = Q[Y =0] for all Q∈Q. (2.10) This definition of Fisher consistency is more restrictive than the above definition quoted from Wikipedia as it requires the specification of a family of distributions to which the parameter ‘recovery’ property applies. The family Q of most interest for the purpose of this paper is the set of distributions Q that are related to one fixed training set distribution P by prior probability shift, i.e. by (2.4). 3. Three approaches to estimating class prevalences under prior probability shift In this section, we study three approaches to the estimation of binary class prevalences: • Adjusted Classify & Count (ACC) (Forman, 2008), called ‘confusion matrix approach’ by Saerens et al. (2001), but in use since long before (Gart and Buck, 1966), 7 • the EM (expectation maximisation) algorithm by Saerens et al. (2001), described before as maxi- mum likelihood approach by Peters and Coberly (1976), and • CDE-Iterate by Xue and Weiss (2009). A variety of other approaches have been and are being studied in the literature (see the discussion in Hofer,2015,forarecentoverview).Theselectionofapproachestobediscussedinthispaperwasdrivenby findings of Xue and Weiss (2009) and more recently Karpov et al. (2016). According to that research, for theestimationofbinaryclassprevalencestheCDE-Iterateapproachseemstoperformequallywelloreven stronger than ACC which by itself was found to outperform the popular EM-algorithm. In this section, we recall the technical details of the approaches which are needed to implement the numerical examples ofSection4below.Inaddition,wecheckthethreeestimatorsonatheoreticalbasisforFisherconsistency. In particular with regard to CDE-Iterate, the theory is inconclusive with regard to its possible Fisher consistency. However, the example of Section 4.2 below shows that CDE-Iterate is not Fisher consistent for class 0 prevalence under prior probability shift, in contrast to both ACC and the EM-algorithm. 3.1. Adjusted Classify & Count (ACC) Let g :X →{0,1} be any classifier. Under prior probability shift as described by (2.4), we then obtain Q[g(X)=0]=Q[Y =0]Q[g(X)=0|Y =0]+(1−Q[Y =0])Q[g(X)=0|Y =1] =Q[Y =0]P[g(X)=0|Y =0]+(1−Q[Y =0])P[g(X)=0|Y =1]. (3.1) If P[g(X)=0|Y =0](cid:54)=P[g(X)=0|Y =1], i.e. if g(X) and Y are not stochastically independent, (3.1) is equivalent to Q[g(X)=0]−P[g(X)=0|Y =1] Q[Y =0] = . (3.2) P[g(X)=0|Y =0]−P[g(X)=0|Y =1] Equation (3.2) is called the ACC approach to the quantification of binary class prevalences. Let us recall some useful facts about ACC: • (3.2) has been around for a long time, at least since the 1960s. In this paper, the quantification approachrelatedto(3.2)iscalled‘adjustedclassify&count’asinForman(2008)becausethisterm nicely describes what is done. • Q[g(X)=0]istheproportionofinstancesinthetestset(measuredbycounting)thatareclassified (predicted) positive by classifier g(X). • P[g(X)=0|Y =1] is the ‘false positive rate’, as measured for classifier g(X) on the training set. • P[g(X)=0|Y =0] is the ‘true positive rate’, as measured for classifier g(X) on the training set. • Forman (2008) discusses ACC in detail and provides a number of variations of the theme in order to account for its deficiencies. • Possibly, the main issue with ACC is that in practice the result of the right-hand side of (3.2) can turn out to be negative or greater than 1. This can happen for one or both of the following two reasons: 1) The dataset shift in question actually is no prior probability shift, i.e. (2.4) does not hold. 2) The estimates of the true positive and true negative rates are inaccurate. • In theory, if the dataset shift is indeed a prior probability shift the result of the right-hand side of (3.2) should be same, regardless of which admissible (i.e. such that the denominator is not zero) classifier is deployed for determining the proportion of instances in the test set classified positive. Hence, whenever in practice different classifiers give different results, that could suggest that the assumption of prior probability shift is wrong. 8 • As long as g(X) and Y are at least somewhat dependent, possible lack of power of the classifier g should not be an issue for the applicability of (3.2). For a training set distribution P denote by Q = Q (P) the family of distributions Q that are prior prior related to P by prior probability shift in the sense of Definition 2.1, i.e. Q = {Q:Q is probability measure satisfying (2.4)}. (3.3) prior Then, for fixed training set distribution P and fixed classifier g(X) such that g(X) and Y are not independent under P, the ACC approach is Fisher consistent in Q for the prevalence of class 0 by prior construction: Define the operator T =T by g,P Q[g(X)=0]−P[g(X)=0|Y =1] T(Q) = . P[g(X)=0|Y =0]−P[g(X)=0|Y =1] Then (3.2) implies (2.10) for Q∈Q . However, denote – again for some fixed training set distribution prior P – by Q =Q (P) the family of distributions Q that are related to P by ‘invariant density invariant invariant ratio’-type dataset shift in the sense of (2.7), i.e. Q = {Q:Q is probability measure satisfying (2.7)}. (3.4) invariant Then in general the ACC approach is not Fisher consistent in Q as it is shown in Section 4.3 invariant below that there are a classifier g∗, a distribution P∗ and a related distribution Q∗ ∈Q (P∗) such invariant that T (Q∗)(cid:54)=Q∗[Y =0]. g∗,P∗ 3.2. EM-algorithm Saerens et al. (2001) made the EM-algorithm popular for the estimation of class prevalences, as a neces- sarystepforthere-adjustmentofthresholdsofsoftclassifiers.AcloserinspectionofthearticlebyPeters andCoberly(1976)showsthattheyactuallyhadstudiedthesamealgorithmandprovidedconditionsfor its convergence. In particular, this observation again draws attention to the fact that the EM-algorithm, deployedondatasamples,shouldresultinuniquemaximumlikelihoodestimatesoftheclassprevalences. As noticed by Du Plessis and Sugiyama (2014), the population level equivalent of sample level maximum likelihood estimation of the class prevalences under an assumption of prior probability shift is minimisa- tionoftheKullback-Leiblerdistancebetweentheestimatedtestsetfeaturedistributionandtheobserved test set feature distribution. Moreover, Tasche (2013, 2014) observed that the EM-algorithm finds the true values of the class prevalences not only under prior probability shift but also under ‘invariant den- sity ratio’-type dataset shift. In other words, the EM-algorithm is Fisher consistent both in Q and prior Q , as defined in (3.3) and (3.4) respectively, for the prevalence of class 0. See Proposition 3.1 invariant below for a formal proof. In the case of two classes, the maximum-likelihood version of the EM-algorithm in the sense of solving the likelihood equation is more efficient than the EM-algorithm itself. This statement applies even more to the population level calculations. In this paper, therefore, we describe the result of the EM-algorithm as the unique solution of a specific equation, as described by Tasche (2014). Calculating the result of the EM-algorithm. In the population setting of Section 2.3 with training set distribution P(X,Y) and test set distribution Q(X,Y), assume that the class-conditional feature distributions P(X|Y = i), i = 0,1, of the training set have got densities f and f . Define the density 0 1 ratio R by f (x) R(x) = 0 , for x∈X. (3.5) f (x) 1 9 WethendefinetheestimationoperatorT (Q)fortheprevalenceofclass0astheuniquesolutionq ∈(0,1) R of the equation (Tasche, 2014) (cid:20) (cid:21) R(X)−1 0 = E , (3.6a) Q 1+q(R(X)−1) where E denotes the expectation operator with respect to Q. Unfortunately, not always a solution of Q (3.6a) exists in (0,1). However, there exists a solution in (0,1) if and only if (cid:20) (cid:21) 1 E [R(X)]>1 and E >1, (3.6b) Q Q R(X) and if there is a solution in (0,1) it is unique (Tasche, 2014, Remark 2(a)). Proposition 3.1 The operator T (Q) (EM-algorithm), as defined by (3.6a), is Fisher consistent in R Q and Q for the prevalence of class 0. prior invariant Proof. We only have to prove the claim for Q because Q is a subset of Q . Let any invariant prior invariant Q ∈ Q be given and denote by h , i = 0,1, its class-conditional feature densities. By (2.7) and invariant i (2.5b), it then follows that (cid:20) (cid:21) (cid:20) (cid:21) R(X)−1 h −h E =E 0 1 Q 1+Q[Y =0](R(X)−1) Q h +Q[Y =0](h −h ) 1 0 1 (cid:2) (cid:3) (cid:2) (cid:3) E Q[Y =0|X] E Q[Y =1|X] Q Q = − Q[Y =0] Q[Y =1] =1−1=0. Hence, the prevalence Q[Y = 0] of class 0 is a solution of (3.6a) and, therefore, the only solution. As a consequence, T (Q) is well-defined and satisfies T (Q)=Q[Y =0]. (cid:50) R R Inpractice,forfixedqtheright-handsideof (3.6a)couldbeestimatedonatestsetsample(x ,y ),..., 1,te 1,te (x ,y ) as in Section 2.3 by the sample average n,te n,te n 1 (cid:88) R(xi,te)−1 , n 1+q(R(x )−1) i,te i=1 where R could be plugged in as a training set estimate of the density ratio by means of (2.8). 3.3. CDE-Iterate In order to successfully apply the ACC quantification approach as described in Section 3.1, we must get hold of reliable estimates of the training set true and false positive rates of the classifier deployed. If the positive class is the minority class, the estimation of the true positive rate can be subject to large uncertainties and, hence, may be hard to achieve with satisfactory accuracy. Similarly, if the negative class is the minority class the estimation of the false positive rate can be rather difficult. Application of the EM-algorithm as introduced by Saerens et al. (2001) or described in Section 3.2 requires reliable estimation of the feature-conditional class probabilities or the density ratio. Again, such estimates in general are hard to achieve. That is why alternative methods for quantification are always welcome. In particular,methodsthatarebasedexclusivelyonlearningoneormorecrispclassifiersarepromising.For learning classifiers is a well-investigated problem that essentially is considered to have been solved. Xue and Weiss (2009) proposed ‘CDE-Iterate’ (CDE for class distribution estimation) which is appro- priately summarised by Karpov et al. (2016) as follows: “The main idea of this method is to retrain a classifier at each iteration, where the iterations progressively improve the quantification accuracy of 10