Outliers and related problems 7 1 0 2 Lev B. Klebanov∗, Jaromir Antoch†, Andrea Karlova‡ n and Ashot V. Kakosyan§ a J 3 2 ] Abstract R P We define outliers as a set of observations which contradicts the . proposed mathematical (statistical) model and we discuss the fre- h t quentlyobservedtypesoftheoutliers. Furtherweexplorewhatchanges a m in the model have to be made in order to avoid the occurance of the outliers. We observe that some variants of the outliers lead to clas- [ sical results in probability, such as the law of large numbers and the 1 v concept of heavy tailed distributions. 2 Key words: outlier; the law of large numbers; heavy tailed dis- 4 tributions; model rejection. 6 6 0 . 1 Introduction and suggestive reflections 1 0 7 In this paper we revise the concept of the outliers. We found the con- 1 : temporary notion rather vague, which motivates us to carefuly dispute its v i meaning. Let us start by closely looking at the definion of outlier provided X by the widely popular free internet encyclopedia Wikipedia. The outliers r a are defined there as follows: “In statistics, an outlier is an observation point that is distant from other observations. An outlier may be due to variability in the measurement or it may indicate experimental error; the latter are ∗Department of Probability and Mathematical Statistics, MFF, Charles University, Czech Republic. e-mail: lev.klebanov@mff.cuni.cz †Department of Probability and Mathematical Statistics, MFF, Charles University, Czech Republic. ‡Institute of Information Theory and Automation, CAS, Prague, Czech Republic. §Yerevan State University, Yerevan, Armenia. 1 sometimes excluded from the data.” Obviously, the definition is given nei- ther in mathematically nor statistically correct way. In particular, we found the description of ”the point being distant” from other observations rather confusing.1 In our opinion, it is essential to specify some measurement unit of the considered distance and mainly the definition of the corresponding considered distance. Therefore we wish to conclude that the term outlier in such a setup is highly depended on the choice of topology and geometry of the space in which we consider our experiment. In the same manner, we found the term ”experimental error” equally misleading. Say, outlier is an observation which is not connected to the particular experiment, and so this observation will not appear in the next experiment. However, the statistics is devoted to repeating the experiments, and such observations will be automatically excluded from further experiments and study. Now consider the possibility that such ”distant” observations remain appearing in the repetitions of our experimental study. In that case, we need to keep the observations attributed to the experiment. Therefore, it is misleading to label the observations as ”errors”. For example, the trigerring event of occurance of such observations can be caused by the design of the particular experiment, i.e. the way how the experiment is designed does not capture the nature of corresponding applied problem. As a result, some observations may appear as a natural phenomena seamlessly to the considered problem. However, there are no mathematical or statistical tools to recognize such a situation and so we are left with concluding that: such observations are in contradiction with mathematical model choosen to describe the practical model under study. Of course, if some observations are in contradiction with one model, they may be in a good agreement with another model. And so we conclude that the notion of outliers is a model sensitive, i.e. the outlier needs to be associated with the concrete mathematical or statistical model. Based on our initial discussion, let us give the following definition. Definition 1.1. Consider a mathematical model of some real phenomena experiment. We say that an observation is the outlier for this particular model if it is ”in contradiction” with the model, i.e. it is either impossible to 1 A little bit better seems to be a definition given on NISTA site:“An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. Inasense,thisdefinitionleavesituptotheanalyst(oraconsensusprocess)to decidewhatwillbeconsideredabnormal. Beforeabnormalobservationscanbesingledout, it is necessary to characterize normal observations.” However, it has similar drawbacks. 2 have such an observation under the assumption that the model holds, or the probability to obtain such observation for the case of true model is extremely low. If the probability is very small yet non-zero, we denoted the probability as β, we will call relevant observation the β-outlier. Definition 1.1 gives precise sense to the second part of the Wikipedia definition. However, it provides no connection to the first part. In the following sections of this paper we provide the arguments and explanations that some typical cases of the outliers appearance in the statistical modelling are closely connected with the properly defined ”the distant character” of them. These ”properdefinitions” providemeaningful suggestionstoposssible model modifications in order to include the outliers as an element of the new model. Note that some ideas of the modification of outliers definitions were already considered in [1]. 2 First definition of distant outliers 2.1 Outliers of the first kind In this section we explore the situation when some observations observation are ”distant” from the others. What is the ”unit of measurement” for such a distance? The natural way to start is to measure the distance of the observations to their mean value in terms of sample variance. Suppose that X ,X ,...,X is a sequence of independent identically dis- 1 2 n tributed (i.i.d.) random variables. Denote by n n 1 1 x¯ = X , s2 = (X −x¯)2 n n j n n j X X j=1 j=1 their empirical mean and empirical variance correspondingly. Let k > 0 be a fixed number. Namely, let us estimate the following probability p = IP{|X −x¯ |/s > k}, (2.1) n n n Definition 2.1. We say that the distribution of X produces outliers of the first kind if the probability (2.1) is high (say, higher than for normal distribution). 3 Really, if one has a model based on Gaussian distribution then the presence of many observations with p greater that for normal case contradicts n to the model, and the observations appears to be outliers in the sense of our Definition 1.1. Such approach was used in financial mathematics to show the Gaussian distribution provides bad model for corresponding data (see, for example, [2, 3]). The observations X for which the inequality |X − x¯ |/s > k holds j j n n appears to be outliers for Gaussian model. In some financial models the presence of them were considered as an argument for the existence of heavy tails for real distributions. Unfortunately, this is not so (see [4, 5, 6]). Theorem 2.1. (see [6]) Suppose that X ,X ,...,X is a sequence of i.i.d. 1 2 n r.v.s belonging to a domain of attraction of strictly stable random variable with index of stability α ∈ (0,2). Then lim p = 0. (2.2) n n→∞ Proof. Since X , j = 1,...,n belong to the domain of attraction of strictly j stable random variable with index α < 2, it is also true that X2,...,X2 1 n belong to the domainof attractionof one-sided stable distribution with index α/2. 1) Consider at first the case 1 < α < 2. In this case, x¯ −→ a = IEX n 1 n→∞ and s −→ ∞. We have n n→∞ IP{|X −x¯ | > ks } = IP{X > ks +x¯ }+IP{X < −ks +x¯ } = 1 n n 1 n n 1 n n = IP{X > ks +a+o(1)}+IP{X < −ks +a+o(1)} −→ 0. 1 n 1 n n→∞ 2) Suppose now that 0 < α < 1. In this case, we have x¯ ∼ n1/α−1Y n as n → ∞. Here Y is α-stable random variable, and the sign ∼ is used for asymptotic equivalence. Similarly, n 1 s2 = X2 −x¯2 ∼ n2/α−1Z(1+o(1)), n n j n X j=1 where Z has one-sided positive stable distribution with index α/2. We have IP{|X −x¯ | > ks } = IP{(X −x¯ )2 > ks2} = 1 n n 1 n n 4 = IP{X2 > n2/α−1Z(1+o(1))} −→ 0. 1 n→∞ 3) In the case α = 1 we deal with Cauchy distribution. The proof for this case is very similar to that in the case 2). We omit the details. From this Theorem it follows that (for sufficiently large n) many heavy- tailed distributions will not produce any outliers of the first kind. Moreover, nowweseethethepresenceofoutliersofthefirstkindisincontradictionwith many models having heavy tailed distributions, particularly, with models involved stable distributions. By the way, word variability is not defined precisely, too. It shows, that high variability may denote something different than high standard deviation. We will discuss this in Section 3, but now let us continue the study of distributions with high probability p . n 2.2 How to obtain more outliers of the first kind? Here we discuss a way of constructing from a distribution another one having a higher probability to observe outliers. We call this procedure ”put tail down”. Let F(x) be a probability distribution function of random variable X having finite second moment σ2 and such that F(−x) = 1 − F(x) for all x ∈ IR1. Take a parameter p ∈ (0,1) and fix it. Define a new function F (x) = (1−p)F(x)+pH(x), p where H(x) = 0 for x < 0, and H(x) = 1 for x > 0. It is clear that F (x) p is probability distribution function for any p ∈ (0,1). Of course, F also has p finite second moment σ2, and F (−x) = 1−F (x). However, σ2 = (1−p)σ2. p p p p Let Y be a random variable with probability distribution function F . Then p p IP{|Y | > k 1−pσ} = 2IP{Y > k 1−pσ} = p p p p = 2(1−p) 1−F(k 1−pσ) . (cid:0) p (cid:1) Denoting F¯(x) = 1−F(x) rewrite previous equality in the form IP{|Y | > k 1−pσ} = 2(1−p)F¯(k 1−pσ). (2.3) p p p For Y to have more outliers than X it is sufficient that p (1−p)F¯(k 1−pσ) > F¯(kσ). (2.4) p 5 There are many cases in which inequality (2.4) is true for sufficiently large values of k. Let us mention two of them. 1. Random variable X has exponential tail. More precisely, F¯(x) ∼ Ce−ax, as x → ∞, for some positive constants C and a. In this case, inequality (2.4) is equivalent for sufficiently large k to (1−p) > Exp{−a·k ·σ ·(1− 1−p)}, p which is obviously true for large k. 2. F has power tail, that is F¯(x) ∼ C/xα, where α > 2 in view of existence of finite second moment. Simple calculations show that (2.4) is equivalent as k → ∞ to (1−p)1−α/2 < 1. The last inequality is true for α > 2. Let us note that the function F has a jump at zero. However, one can p obtainsimilar effect without such jump byusing asmoothing procedure, that is by approximating F by smooth functions. p ”Put tail down” procedure allows us to obtain more outliers in view of two its elements. First element consists in changing the tail by smaller, but proportional to previous with coefficient 1−p. The second element consist in moving a part of mass into origin (or into a small neighborhood of it), which reduces the variance. The procedure described above shows us thatthe presence of outliers may have no connection with existence of heavy tails of underlying distribution or with experimental errors. 2.3 On extremal and related distributions with outliers of the first kind In the case of finite variance it is possible to find a distribution maximizing the probability p(k) = IP{|X| > kσ} (2.5) 6 for the case IEX = 0, k > 1. Corresponding boundary is given by Selberg inequality (see, for example, [7]). Namely, if X is a random variable such that IEX = 0, IEX2 = σ2 < ∞ then for any k > 1 1 p(k) ≤ . (2.6) k2 The equality in (2.6) is attended on a distribution concentrated at 3 points: −σ, 0 and σ. In the case of k = 3 the boundary in (2.6) is 1/k2 = 1/9, which shows that for extremal distribution one may have many outliers of the first kind. However, corresponding distribution has a compact support. It is clear that extremal distribution has very specific form and rarely appears in applications. Therefore, it is of essential interest to find out which properties of a distribution lead to the presence of rather high probability for outliers of the first kind. The form of extremal distribution tells us that a part of it has to be concentrated near mean value, while other part must be not too close to the mean. Suppose that X ,X ,...,X are i.i.d. random 1 2 n variables. Denote by 0 ≤ X ≤ X ≤ ... ≤ X ordered values of absolute (1) (2) (n) values ofobservations |X |, j = 1,...,n. It seems tobetruethat to get many j outliersof thefirst kindoneneeds tohave rather highprobabilityof theevent IP{X > ρX }, where ρ > 1. Let us verify this statement. (2) (1) To this aim calculate the probability of X to be greater than ρX (2) (1) for a fixed ρ > 1. Suppose that X ,...,X are i.i.d. random variables, 1 n and X ,...,X are the observations ordered in its absolute values. Sup- (1) (n) pose that random variable |X | has absolute continuous distribution function 1 F(x), andp(x) is its density. Denote by p (x,y) the common density of X 1,2 (1) and X . We have (see, for example, [8]) (2) p (x,y) = n(n−1)Fn−2(y)p(x)p(y), (2.7) 1,2 for x ≤ y. Therefore the probability of the event that X ≥ ρX is (2) (1) IP{X ≥ ρX } = (2) (1) ∞ n−1 n−1 = n 1−F(x) − 1−F(ρx) p(x)dx = Z (2.8) (cid:16) (cid:17) 0 (cid:0) (cid:1) (cid:0) (cid:1) ∞ = 1−n (1−F(ρx))n−1p(x)dx. Z 0 7 Let us try to study limit behavior of the probability (2.8) for large values of sample size n. We have ∞ p(x) IP{X ≥ ρX } = 1−n (1−F(ρx))n−1ρp(ρx) dx. (2.9) (2) (1) Z ρp(ρx) 0 Assume that p(x) n lim 1−F(ρx) = 0. (2.10) x→∞ ρp(ρx) (cid:0) (cid:1) Integrating by parts in (2.9) gives us ∞ p(x) d p(x) IP{X ≥ ρX } = lim − (F(ρx))n dx (2.11) (2) (1) x→0 ρp(ρx) Z dx(cid:16)ρp(ρx)(cid:17) 0 If the function d p(x) dxρp(ρx) is absolute integrable over (0,∞) then ∞ d p(x) (F(ρx))n dx → 0 Z dxρp(ρx) 0 as n → ∞. Therefore, p(x) lim IP{X ≥ ρX } = lim , (2.12) (2) (1) n→∞ x→0 ρp(ρx) assuming that the limit in right-hand side of (2.12) exists. Finally, we obtain the following result. Theorem 2.2. Suppose that X ,...,X are i.i.d. random variables, and 1 n X ,...,X are the observations ordered in its absolute values. Let random (1) (n) variable |X | has absolute continuous distribution function F(x), and let p(x) 1 be its density. Suppose that p(x) is regularly varying function of index α−1 at zero, the function d p(x) dxρp(ρx) exists and is integrable over (0,∞), and p(x) n lim 1−F(ρx) = 0. x→∞ ρp(ρx) (cid:0) (cid:1) 8 Then 1 lim IP{X ≤ ρX } = 1− . (2.13) n→∞ (2) (1) ρα Proof. ThestatementoftheTheoremfollowsfromconsiderationsgivenabove and from the definition of regularly varying function (see, for example [9]). Let usconsider theprobability IP{X ≤ ρX }asa functionofρ. Under (2) (1) conditions of Theorem 2.2 this probability represents cumulative distribution function of Pareto law with parameter α. Is it possible to find a distribution function for which the equality holds not only in limit, but for all values of n? The answer to this question is affirmative. Theorem 2.3. Let X ,...,X be i.i.d. random variables taking values in 1 n interval (0,1). Suppose additionally that: 1. Distribution function F(x) of X is absolute continuous, strictly mono- 1 tone on (0,1), and p(x) is its density. 2. p(x) is regularly varying function of index α−1 at zero. 3. p(x) is differentiable on (0,1) and the function d p(x) dxρp(ρx) is integrable on (0,1). Then the equality IP{X ≤ ρX } = 1−1/ρα (2.14) (2) (1) holds for all positive integer n and all ρ > 1 if and only if 1/X has Pareto 1 distribution with parameter α and initial point 1. Proof. Let us suppose that 1/X has Pareto distribution with parameter α 1 and initial point 1. Then F(x) = xα for x ∈ (0,1). In this case, p(x) = αxα−1 for x ∈ (0,1) and p(x) = 0 otherwise. It is easy to calculate that d p(x) = 0. From (2.11) it follows dxρp(ρx) (3.8). 9 Suppose now that (3.8) holds. From (2.11) we see that necessarily 1/ρ d p(x) (F(ρx))n dx = 0 Z dxρp(ρx) 0 for all positive integers n. In view of compactness of the interval (0,1/ρ) and strictly monotone character of F the problem of moments has unique solution. Therefore, d p(x) = 0 dxρp(ρx) for all x ∈ (0,1/ρ). This implies that p(x) = A(ρ), ρp(ρx) whereA(ρ)dependsonρonly. Inotherwords, wehavethefollowingequation p(x) = B(ρ)p(ρx), (2.15) for all ρ > 1 and all x ∈ (0,1/ρ), and B(ρ) = ρA(ρ). Passing to logarithms transforms (2.15) to well-known Cauchy functional equation, which leads to p(x) = αxα−1. Let now Y be a symmetric random variable such that 1/|Y| has Pareto distribution with parameter α and initial point 1. We expect that Y has outliers of thefirst kind with rather high probability althoughits distribution is not too close to extremal one and has a compact support. Really, simple calculations give us that α α α/2 IEY = 0, σ2 = IEY2 = , IP{|Y| ≥ 3σ} = 1−3α . 2+α (cid:16)2+α(cid:17) For α = 0.089115 this probability is approximately 0.0417598, which is greater than for Gaussian distribution. As a conclusion of this section, we can say the presence of outliers of the first kind is not connected to tails of a distribution. It is associated with the behavior of the density near mean value. 10

Outliers and related problems

Similar Outliers and related problems