SubmittedtotheAnnalsofProbability ∗ PRODUCT OF RANDOM STOCHASTIC MATRICES By Behrouz Touri and Angelia Nedic´ University of Illinois at Urbana-Champaign Itiswell-knownthatforanyaperiodicandirreduciblestochastic matrix A, limk→∞Ak exists and it is a rank one stochastic matrix. We show that a generalization of this result not only holds for in- 1 homogeneous chains of stochastic matrices but also for independent 1 random chains of such matrices. 0 2 t 1. Introduction. Let (Ω,F,Pr) be a probability space and let {W(k)} be a chain of m×m c randomrowstochastic matrices, i.e. forallk ≥ 1,thematrix W(k)isarow stochastic almostsurely O and W (k) : Ω → R is a Borel-measurable function for all i,j ∈ [m], where [m] = {1,...,m}. ij 8 Throughout this paper, we denote random chains by last alphabet letters such as {W(k)} and ] {U(k)}, and we use the first alphabet letters such as {A(k)} and {B(k)} to denote deterministic R stochastic chains. We will exclusively deal with row stochastic chains, so we will refer to them as P stochastic chains, or just simply as chains. . h Let {W(k)} be an independent random chain. Then, we say that {W(k)} is strongly aperiodic if t a there exists a γ ∈ (0,1] such that m [ E[W (k)W (k)] ≥ γE[W (k)] for all i 6= j ∈ [m] and all k ≥ 1. ii ij ij 1 v Note that if Wii(k) ≥ γ almost surely for all i ∈ [m] and all k ≥ 1, then such a chain is strongly 1 aperiodic. Also, note that by summing both sides of the above inequality over j 6= i, we obtain 5 7 E[W (k)] ≥ E[W (k)(1−W (k))] ≥ γ(1−E[W (k)]). 1 ii ii ii ii . 0 Hence, E[W (k)] ≥ γ for all i ∈ [m] and all k ≥ 1. Thus, for a strongly aperiodic chain {W(k)}, 1 ii 1−γ 1 the expected chain {E[W(k)]} is strongly aperiodic. It follows that a deterministic chain {A(k)} is 1 strongly aperiodic if and only if A (k) ≥ γ˜ for some γ˜ > 0, and for all i ∈ [m] and k ≥1. ii : v For the subsequent use, for an m ×m random (or deterministic) matrix W and a non-trivial Xi index set S ⊂ [m] (i.e. S 6= ∅ and S 6= [m]), we define the quantity WSS¯ = i∈S,j∈S¯Wij, where S¯ r is the complement of the index set S. a P We say that an independent random chain {W(k)} is balanced if there exists some α > 0 such that (1) E[W (k)] ≥ αE[W (k)] for all nontrivial S ⊂ [m] and all k ≥ 1. SS¯ S¯S From this definition, it can be seen that α ≤ 1. Finally, with a given random chain {W(k)}, let us associate a random graph G∞ = ([m],E∞) with the vertex set [m] and the edge set E∞ given by ∞ E∞(ω) = {i,j} | (W (k,ω)+W (k,ω)) = ∞ . ij ji ( ) k=1 X ∗ This research was supported by theNational ScienceFoundation underCAREER grant CMMI 07-42538. AMS 2000 subject classifications: Primary 60F99, 60B20 1 2 We refer to G∞ as the infinite flow graph of {W(k)}. By the Kolmogorov’s 0-1 law, the infinite flow graph of an independent random chain {W(k)} is almost surely equal to a deterministic graph. It has been shown that this determinstic graph is equal to the infinite flow graph of the expected chain {E[W(k)]} ([8], Theorem 5). For a matrix W, let W and Wj denote the ith row vector and the jth column vector of W, i respectively. Also, for a chain {W(k)}, let W(k : t ) = W(k)···W(t +1), where k > t ≥ 0 and 0 0 0 let W(k : k) = I for all k ≥ 0. With these preliminary definitions and notation in place, we can state the main result of the current study. Theorem 1. Let {W(k)} be an independent random stochastic chain which is balanced and strongly aperiodic. Then, for any t ≥ 0 the product W(k :t ) = W(k)···W(t +1) converges to a 0 0 0 random stochastic matrix W(∞ :t ), almost surely. Furthermore, for all i,j in the same connected 0 component of the infinite flow graph of {W(k)}, we have W (∞ :t )= W (∞ :t ) almost surely. i 0 j 0 As an immediate consequence of this result, it follows that W(∞ :t ) has rank at most τ where 0 τ is the number of the connected components of the infinite flow graph G∞ of {W(k)}. Thus, if G∞ is a connected graph, the limiting random matrix W(∞ :t ) = lim W(k : t ) is a rank-one 0 k→∞ 0 random stochastic matrix almost surely, i.e. W(∞ : t )= evT(t ) almost surely for some stochastic 0 0 vector v(t ). This and Theorem 1 imply that: if an independent random chain {W(k)} is balanced 0 and strongly aperiodic, then {W(k)} is almost surely strongly ergodic (as defined in [3]) if and only if the infinite flow graph of {W(k)} is connected. 2. The Dynamics System Perspective. In order to prove Theorem 1, we establish some intermediate results, some of which are applicable to a more general category of random stochastic chains, namely adapted random chains. For this, let {W(k)} be a random chain adapted to a filtration {F }. For an integer t ≥ 0 and a vector v ∈ Rm, consider the trivial random vector k 0 x(t ): Ω → Rm defined by x(t ,ω) = v for all ω ∈ Ω. Now, recursively define: 0 0 (2) x(k+1) = W(k+1)x(k) for all k ≥ t . 0 Notethatx(t )ismeasurablewithrespecttothetrivialσ-algebra{∅,Ω}andhence,itismeasurable 0 with respect to F . Also, since {W(k)} is adapted to {F }, it follows that for any k > t , x(k) is t0 k 0 measurablewith respecttoF .We referto {x(k)} as arandomdynamics drivenby {W(k)} started k at the initial point (t ,v) ∈Z+×Rm. We say that a given property holds for any dynamics {x(k)} 0 driven by {W(k)} if that property holds for any initial point (t ,v) ∈Z+×Rm. 0 Note that if lim W(k : t ) = W(∞ : t ) exists almost surely, then for any initial point k→∞ 0 0 (t ,v) ∈ Z+×Rm, the random dynamics {x(k)} converges to W(∞ : t )v almost surely. Also, note 0 0 that for any i,j ∈ [m], we have lim kW (k,t )−W (k,t )k = 0 almost surely if and only if k→∞ i 0 j 0 lim (x (k)−x (k)) = 0 almost surely for any initial point. When verifying the latter relation, k→∞ i j due to linearity of the dynamics, it suffices to check that lim (x (k)−x (k)) = 0 for all the k→∞ i j initial points of the form (t ,e ) with ℓ ∈ [m], where {e ,...,e } is the standard basis for Rm. 0 ℓ 1 m In order to study the limiting behavior of the products W(k :t ), we study the limiting behavior 0 of the dynamics {x(k)} driven by {W(k)}. This enables us to use the dynamics system’s tools and its stability theory to draw conclusions about the limiting behavior of the matrix products W(k : t ). 0 2.1. Why the Infinite Flow Graph. In this section we provide a result showing the relevance of the infinite flow graph to the study of the product of stochastic matrices. Let us consider a deterministic chain {A(k)} of stochastic matrices and let usdefinemutual ergodicity and an ergodic index as follows. PRODUCTOF RANDOMSTOCHASTICMATRICES 3 Definition 1. For an m×m chain {A(k)} of stochastic matrices, we say that an index i ∈ [m] is ergodic if lim A (k : t ) exists for all t ≥ 0. Also, we say that two disctinct indices i,j ∈ [m] k→∞ i 0 0 are mutually ergodic if lim kA (k :t )−A (k :t )k = 0. k→∞ i 0 j 0 From the definition it immediately follows that an index i ∈ [m] is ergodic for a chain {A(k)} if and only if lim x (k) exists for any dynamics {x(k)} driven by {A(k)}. Similarly, indices k→∞ i i,j ∈ [m] are mutually ergodic if and only if lim (x (k)−x (k)) = 0 for any dynamics driven k→∞ i j by {A(k)}. Thefollowingresultillustratestherelevanceoftheinfiniteflowgraphtothestudyoftheproducts of stochastic matrices. Lemma 1. ([8], Lemma 2) Two distinct indices i,j ∈ [m] are mutually ergodic only if i and j belong to the same connected component of the infinite flow graph G∞ of {A(k)}. Generally, if i and j are mutually ergodic indices, it is not necessarily true that they are ergodic indices. As an example, consider the 4×4 stochastic chain {A(k)} defined by: 1 0 0 0 1 0 0 0 1 0 0 0 0 0 0 1 A(2k) =  , A(2k+1) =   for all k ≥ 1. 1 0 0 0 0 0 0 1  0 0 0 1   0 0 0 1          It can be verified that for any starting time t ≥ 0 and any k > t , we have A(k :t )= A(k). Thus, 0 0 0 it follows that indices 2 and 3 are mutually ergodic, while lim A (k :t ) and lim A (k :t ) k→∞ 2 0 k→∞ 3 0 do not exist. The following result shows that under special circumstances, we can assert that some indices are ergodic if we know that a certain mutual ergodicity pattern exists in a chain. Lemma 2. Let S be a connected component of the infinite flow graph G∞ of a chain {A(k)}. Suppose that indices i and j are mutually ergodic for all distinct i,j ∈ S. Then, every index i ∈ S is ergodic. Proof. Without loss of generality let us assume that S = {1,...,i∗} for some i∗ ∈ [m]. Let S¯ be the complement of S. For the given chain {A(k)} and the connected component S, let the chain {B(k)} be defined by: A (k) if i6= j and i,j ∈ S or i,j ∈ S¯, ij 0 if i6= j and i ∈ S,j ∈ S¯ or i ∈ S¯,j ∈ S, Bij(k) =  A (k)+ A (k) if i= j ∈ S,  ii ℓ∈S¯ iℓ  A (k)+ A (k) if i= j ∈ S¯. ii Pℓ∈S iℓ  Then, B(k) has the block diagonPal structure of the following form B (k) 0 B(k)= 1 for all k ≥ 1. 0 B (k) 2 (cid:20) (cid:21) By construction the chain {B(k)} is stochastic. It can be verified that ∞ |A (k)−B (k)| < ∞ k=1 ij ij foralli,j ∈ [m].Thus,{B(k)}isanℓ -approximationof{A(k)}asdefinedin[8].Then,byLemma1 1 P in[8],itfollowsthatindicesiandj aremutuallyergodicforthechain{B(k)}foralldistincti,j ∈ S. By the block diagonal form of {B(k)}, it follows that i and j are mutually ergodic for the |S|×|S| 4 chain {B (k)} and all i,j ∈ S. This, however, implies that the chain {B (k)} is weakly ergodic 1 1 (as defined in [3]) and, as proven in Theorem 1 in [3], this further implies that {B (k)} is strongly 1 ergodic, i.e. any index i ∈ S is ergodic for {B (k)}. Again, by the application of Lemma 1 in [8], 1 we conclude that any index i ∈ S is ergodic for {A(k)}. Q.E.D. 2.2. Comparison Functions. Here, we show that under general conditions, a rich family of com- parison functions exists for the dynamics {x(k)} driven by a random chain {W(k)}. Letus definean absolute probability process foran adapted chain {W(k)}, which is an extension of the concept of the absolute probability sequence introduced by A. Kolmogorov for deterministic chains in [4]. Definition 2. We say that a random (vector) process {π(k)} is an absolute probability process for a random chain {W(k)} adapted to {F } if k 1. the random process {π(k)} is adapted to {F }, k 2. the vector π(k) is stochastic almost surely for all k ≥ 1, and 3. the following relation holds almost surely E πT(k+1)W(k+1) | F = πT(k) for all k ≥ 0. k (cid:2) (cid:3) Whenanabsoluteprobability processexists forachain,wesay thatthechain admitsanabsolute probability process. For a deterministic chain of stochastic matrices {A(k)}, Kolmogorov showed in [4] that there exists a sequence of stochastic vectors {v(k)} such that vT(k +1)A(k +1) = vT(k) for all k ≥ 0. Note that, for an independent random chain, any absolute probability sequence for the expected chain is an absolute probability process for the random chain. Thus, the existence of an absolute probability process for an independent random chain of stochastic matrices follows immediately from the Kolmogorov’s existence result. As another non-trivial example of random chains that admit an absolute probability process, one may consider an adapted random chain {W(k)} that is doubly stochastic almost surely. In this case, the static sequence {1e} is an absolute probability m process for {W(k)}, where e∈ Rm is the vector with all components equal to 1. Now, suppose that we have an adapted chain {W(k)} which admits an absolute probability sequence {π(k)}. Also, let g : R → R be an arbitrary convex function. Let us define the function V :Rm×Z+ → R, as follows: g,π m (3) V (x,k) = π (k)g(x )−g(πT(k)x) for all x ∈Rm and all k ≥ 0. g,π i i i=1 X From the definition of an absolute probability process, it follows that V (x(k),k) is measurable g,π with respect to F for any dynamics {x(k)} driven by a chain {W(k)} that is adapted to {F }. k k Also, since π(k) is almost surely stochastic vector and g is a convex function, it follows that for any x ∈ Rm, we have V (x,k) ≥ 0 almost surely for all k ≥ 0. g,π Next, we show that V is a comparison function for the dynamics (2) for any convex function g,π g. In particular, we prove that {V (x(k),k)} is a super-martingale sequence irrespective of the g,π initial point for the dynamics {x(k)}. Theorem 2. Let {W(k)} be an adapted chain that admits an absolute probability process {π(k)}. Then, for the dynamics (2) started at any initial point (t ,v) ∈Z+×Rm, we have 0 E[V (x(k+1),k+1) | F ]≤ V (x(k),k) for all k ≥ t . g,π k g,π 0 PRODUCTOF RANDOMSTOCHASTICMATRICES 5 Proof. By the definition of V in (3), we have almost surely g,π m V (x(k+1),k+1) = π (k+1)g(x (k+1))−g(πT(k+1)x(k+1)) g,π i i i=1 X m = π (k+1)g([W(k+1)x(k)] )−g(πT(k+1)x(k+1)) i i i=1 X m m (4) ≤ π (k+1) W (k+1)g(x (k))−g(πT(k+1)x(k+1)), i ij j i=1 j=1 X X where in the second equality we use [·] to denote the ith component of a vector, while the in- i equality is obtained by using the convexity of g(·) and the fact that matrix W(k) is stochas- tic almost surely. Since {π(k)} is an absolute probability process for {W(k)}, it follows that E πT(k+1)W(k+1) |F = πT(k). Also, since x(k) is measurable with respect to F , by tak- k k ing the conditional expectation with respect to F on both sides of Eq. (4), we obtain almost k (cid:2) (cid:3) surely m E[V (x(k+1),k+1) |F ] ≤ π (k)g(x (k))−E g(πT(k+1)x(k+1)) | F g,π k j j k j=1 X (cid:2) (cid:3) m ≤ π (k)g(x (k))−g(E πT(k+1)x(k+1) |F ), j j k j=1 X (cid:2) (cid:3) where the last inequality follows by the convexity of g and Jensen’s inequality. The result follows by using x(k+1) = W(k+1)x(k) and the definition of absolute probability process. Q.E.D. Theorem 2 shows that the dynamics (2) admits infinitely many comparison functions, provided that {W(k)} admits an absolute probability process. Since V (x(k),k) ≥ 0 almost surely for all k ≥ 0, it follows that {V (x(k),k)} is a bounded g,π g,π super-martingale. Hence, it is convergent almost surely irrespective of the initial point of the dy- namics {x(k)} and the choice of the convex function g. Corollary 1. Let {W(k)} be an adapted chain that admits an absolute probability process {π(k)}. Then, for any dynamics {x(k)} driven by {W(k)} and for any convex function g :R → R, the limit lim V (x(k),k) exists almost surely. k→∞ g,π 2.3. Quadratic Comparison Function. Inthesequel,wefocusontheparticularchoiceoffunction g(s) = s2 in relation (3). For convenience, we let m m V (x,k) = π (k)(x −πT(k)x)2 = π (k)x2 −(πT(k)x)2. π i i i i i=1 i=1 X X For this particular choice of the convex function g, we can provide a lower bound for the decrease of the conditional expectations E[V (x(k+1),k+1) |F ], which is exact under certain conditions. g k Theorem 3. Let {W(k)} be an adapted random chain with an absolute probability process {π(k)}. Then, for any dynamics {x(k)} driven by {W(k)}, we have almost surely E[V (x(k+1),k+1) | F ]≤ V (x(k),k)− H (k)(x (k)−x (k))2 for all k ≥ t , π k π ij i j 0 i<j X 6 where H(k) = E WT(k+1)diag(π(k+1))W(k+1) | F withdiag(v) denoting the diagonal matrix k induced by a vector v (i.e., with components v on the main diagonal), and = m m . (cid:2) i (cid:3) i<j i=1 j=i+1 Furthermore, if πT(k+1)W(k+1) =πT(k) almost surely, then the inequality holds as an equality. P P P Proof. We have for all k ≥t , 0 m (5) V (x(k),k) = π (k)x2(k)−(πT(k)x(k))2 =xT(k)diag(π(k))x(k)−(πT(k)x(k))2. π i i i=1 X Thus, by letting ∆(x(k),k) = V (x(k),k)−V (x(k+1),k+1) and using x(k+1) = W(k+1)x(k), π π we obtain for all k ≥t , 0 ∆(x(k),k) = xT(k)diag(π(k))x(k)−(πT(k)x(k))2 − xT(k+1)diag(π(k+1))x(k+1)−(πT(k+1)x(k+1))2 = xT(k(cid:8)) diag(π(k))−WT(k+1)diag(π(k+1))W(k+1) x(k) (cid:9) + ((cid:2)πT(k+1)x(k+1))2 −(πT(k)x(k))2 (cid:3) = xT(k(cid:8))L(k)x(k)+ (πT(k+1)x(k+1))2 −(cid:9)(πT(k)x(k))2 , where L(k) = diag(π(k))−WT(k+1)diag(cid:8)(π(k+1))W(k+1). (cid:9) Note that the sequence {πT(k)x(k)} is a martingale, implying that {−(πT(k)x(k))2} is a super- martingale. Thus, by taking the conditional expectation on both sides of the preceding equality and noticing that x(k) is measurable with respect to F , we have almost surely k (6) E[∆(x(k),k) | F ]≥ E xT(k)L(k)x(k) |F = xT(k)E[L(k) |F ]x(k) for all k ≥ t . k k k 0 Further, letting e ∈ Rm be(cid:2)the vector with all co(cid:3)mponents equal to 1, from the definition of L(k) we almost surely have for all k ≥ t : 0 E[L(k) |F ]e= E diag(π(k))e−WT(k+1)diag(π(k+1))W(k+1)e |F k k = π(cid:2)(k)−E WT(k+1)π(k+1) | Fk = 0, (cid:3) (cid:2) (cid:3) which holdssinceW(k)is stochastic almost surelyand{π(k)} is an absoluteprobability processfor {W(k)}. Thus, the random matrix E[L(k) |F ] is symmetric and E[L(k) | F ]e = 0 almost surely. k k Itcan beshownthat forasymmetricmatrix Awith Ae= 0, wehave xTAx = − A (x −x )2. i<j ij i j Then, it follows that almost surely P xT(k)E[L(k) | F ]x(k) = − H (k)(x (k)−x (k))2, k ij i j i<j X where H(k) = E WT(k+1)diag(π(k+1))W(k+1) |F . Using this relation in inequality (6), we k conclude that almost surely (cid:2) (cid:3) (7) E[V (x(k+1),k+1) | F ]≤ V (x(k),k)− H (k)(x (k)−x (k))2 for all k ≥ t . π k π ij i j 0 i<j X In the proof of inequality (7), the inequality appears due to relation (6) only. If πT(k+1)W(k+ 1) = πT(k)W(k) almost surely, then we have πT(k+1)x(k+1) = πT(k)x(k) almost surely. Thus, relation (6) holds as an equality, and consequently so does relation (7). Q.E.D. PRODUCTOF RANDOMSTOCHASTICMATRICES 7 One of the important implications of Theorem 3 is the following result. Corollary 2. Let {W(k)} be an adapted random chain that admits an absolute probability process {π(k)}. Then, for any random dynamics {x(k)} driven by {W(k)}, we have for all t ≥ 0, 0 ∞ E L (k)(x (k)−x (k))2 ≤ E[V (x(t ),t )] < ∞, ij i j π 0 0   kX=t0Xi<j   where L(k) = WT(k+1)diag(π(k+1))W(k+1). Proof. By taking expectation on both sides of the relation in Theorem 3, we obtain for all k ≥ t : 0 (8) E[V (x(k+1),k+1)] ≤ E[V (x(k),k)]−E E[L (k) |F ](x (k)−x (k))2 . π π ij k i j   i<j X   Since x(k) is measurable with respect to F , it follows that k E[L (k) | F ](x (k)−x (k))2 = E L (k)(x (k)−x (k))2 |F = E L (k)(x (k)−x (k))2 . ij k i j ij i j k ij i j Using this relation in Eq. (8), we see(cid:2)that for all k ≥ t : (cid:3) (cid:2) (cid:3) 0 E[V (x(k+1),k+1)] ≤ E[V (x(k),k)]−E L (k)(x (k)−x (k))2 . π π ij i j   i<j X   Hence, ∞ E L (k)(x (k)−x (k))2 ≤ E[V (x(t ),t )] for any t ≥ 0. Q.E.D. k=t0 i<j ij i j π 0 0 0 h i 3. CPlass P∗.PIn this section, we introduce a class of random chains which we refer to it as the class P∗ and prove one of the central results of this work. In particular, we show that the claim of Theorem 1 holds for any chain that is in the class P∗ and satisfies some form of aperiodicity. Definition 3. The class P∗ is the class of random adapted chains that admit an absolute probability process {π(k)} that is unformly bounded away from zero almost surely, i.e., π (k) ≥ p∗ i almost surely for some scalar p∗ > 0, and for all k ≥ 0 and all i ∈ [m]. We write this concisely as {π(k)} ≥ p∗ >0. It may appear that the definition of the class P∗ is rather a restrictive requirement. Later on, we show that in fact the class P∗ contains a broad family of deterministic and random chains. To establish the main result of this section, we make use of the following intermediate result. Lemma 3. Let {A(k)} be a deterministic chain with the infinite flow graph G∞ = ([m],E∞). Let (t ,v) ∈ Z+×Rm be an initial point for the dynamics driven by {A(k)}. If 0 lim (x (k)−x (k)) 6= 0, i0 j0 k→∞ for some i ,j belonging to the same connected component of G∞, then we have 0 0 ∞ [(A (k+1)+A (k+1))(x (k)−x (k))2] = ∞. ij ji i j kX=t0Xi<j 8 Proof. Leti ,j beinthesameconnectedcomponentofG∞ withlimsup (x (k)−x (k)) = 0 0 k→∞ i0 j0 α > 0. Without loss of generality we may assume that x(t ) ∈ [−1,1]m, for otherwise we can con- 0 sider the dynamics started at y(t ) = 1 x(t ). Let S be the vertex set of the connected 0 kx(t0)k∞ 0 component in G∞ containing i ,j , and without loss of generality assume that S = {1,2,...,q} for 0 0 some q ∈ [m], q ≥ 2. Then, by the definition of the infinite flow graph, there exists a large enough K ≥ t such that 0 ∞ α A (k+1) ≤ , S 32q k=K X where AS(k +1) = ASS¯(k +1)+AS¯S(k +1). Furthermore, since limsupk→∞(xi0(k)−xj0(k)) = α > 0, there exists a time instance t ≥ K such that x (t )−x (t ) ≥ α. 1 i0 1 j0 1 2 Let σ : [q] → [q] be a permutation such that x (t ) ≥ x (t ) ≥ ··· ≥ x (t ), i.e. σ is an σ(1) 1 σ(2) 1 σ(q) 1 ordering of {x (t ) | i ∈ [q]}. Since x (t )−x (t ) ≥ α, it follows that x (t )−x (t ) ≥ α i 1 i0 1 j0 1 2 σ(1) 1 σ(q) 1 2 and, therefore, there exists ℓ ∈ [q] such that x (t )−x (t ) ≥ α. Let σ(ℓ) 1 σ(ℓ+1) 1 2q t α T = argmin (A (k+1)+A (k+1)) ≥ . 1 t>t1 σ(i)σ(j) σ(j)σ(i) 32q kX=t1 i,Xj∈[q] i≤ℓ,ℓ+1≤j Since S is a connected component of the infinite flow graph G∞, we must have T < ∞; otherwise, 1 S could bedecomposedintotwo disconnected components{σ(1),...,σ(l)} and{σ(l+1),...,σ(q)}. Now, let R = {σ(1),...,σ(l)}. We have for any k ∈ [t ,T ]: 1 1 T1−1 T1−1 AR(k+1) =  (Aσ(i)σ(j)(k+1)+Aσ(j)σ(i)(k+1)) kX=t1 kX=t1 i,Xj∈[q] i leqℓ,ℓ+1≤j  + A (k+1)+ A (k+1) σ(i)j iσ(j)  i≤Xℓ,j∈S¯ i∈XS¯,j≤l T1−1  ∞ α ≤ (A (k+1)+A (k+1))+ A (k+1) ≤ , σ(i)σ(j) σ(j)σ(i) S 16q kX=t1 i,Xj∈[q] kX=K i≤ℓ,ℓ+1≤j which follows by the definition of T and the choice of t ≥ K. By Lemma 1 in [9], it follows that 1 1 for k ∈[t ,T ], 1 1 α α maxx (k) ≤ maxx (t )+2 , min x (k) ≥ min x (t )−2 . i i 1 i i 1 i∈R i∈R 16q i∈S\R i∈S\R 16q Thus, for any i,j ∈ [q] with i≤ l and j ≥ l+1, and for any k ∈ [t ,T ], we have 1 1 α α x (k)−x (k) ≥ 2 2 = . σ(i) σ(j) 16q 4q (cid:18) (cid:19) PRODUCTOF RANDOMSTOCHASTICMATRICES 9 Therefore, T1 (A (k+1)+A (k+1))(x (k)−x (k))2 σ(i)σ(j) σ(j)σ(i) σ(i) σ(j) kX=t1 i,Xj∈[q] i≤ℓ,ℓ+1≤j α T1 α 2 α ≥ ( )2 (A (k+1)+A (k+1)) ≥ = β > 0. 4q σ(i)σ(j) σ(j)σ(i) 4q 32q kX=t0 i,Xj∈[q] (cid:18) (cid:19) i≤l,j≥l+1 Further, it follows that: T1 (A (k+1)+A (k+1))(x (k)−x (k))2 ij ji i j kX=t1Xi<j T1 ≥ (A (k+1)+A (k+1))(x (k)−x (k))2 ≥ β. σ(i)σ(j) σ(j)σ(i) σ(i) σ(j) kX=t1 i,Xj∈[q] i≤ℓ,ℓ+1≤j Since limsup (x (k)−x (k)) = α > 0, there exists a time t > T such that x (t ) − k→∞ i0 j0 2 1 i0 2 x (t ) ≥ α. Then, using the above argument, there exists T > t such that T2 (A (k+ j0 2 2 2 2 k=t2 i<j ij 1)+A (k+1))(x (k)−x (k))2 ≥ β. Hence, using the induction, we can find time instances ji i j P P ··· > T > t > T > t > T > t > ··· > T > t ≥ t , ξ+1 ξ+1 ξ ξ ξ−1 ξ−1 1 1 0 such that Tξ (A (k +1)+A (k +1))(x (k)−x (k))2 ≥ β for any ξ ≥ 1. The intervals k=tξ i<j ij ji i j [t ,T ] are non-overlapping subintervals of [t ,∞), implying that ξ ξ P P 0 ∞ (A (k+1)+A (k+1))(x (k)−x (k))2 = ∞. ij ji i j kX=t0Xi<j Q.E.D. For our main result, let us define the weak aperiodicity for an adapted random chain. Definition 4. We say that an adapted random chain {W(k)} is weakly aperiodic if for some γ > 0, and for all distinct i,j ∈ [m] and all k ≥ 0, E WiT(k+1)Wj(k+1) |F ≥ γE[W (k+1)+W (k+1) | F ]. k ij ji k (cid:2) (cid:3) Now, we establish the main result of this section. Theorem4. Let{W(k)} ∈ P∗ beanadapted chainthatisweaklyaperiodic. Then,lim W(k : k→∞ t ) = W(∞ :t ) exists almost surely for any t ≥ 0. Moreover, the event under which W (∞ :t )= 0 0 0 i 0 W (∞ : t ) for all t ≥ 0 is almost surely equal to the event that i,j are belonging to the same j 0 0 connected component of the infinite flow graph of {W(k)}. Proof. Since{W(k)} is inP∗,{W(k)} admitsan absoluteprobability process {π(k)} suchthat {π(k)} ≥ p∗ >0 almost surely. Thus, it follows that p∗E WT(k+1)W(k+1) | F ≤ E WT(k+1)diag(π(k+1))W(k+1) | F = H(k+1). k k (cid:2) (cid:3) (cid:2) (cid:3) 10 On the other hand, by the weak aperiodicity, we have γE[W (k+1)+W (k+1) |F ] ≤ E WiT(k+1)Wj(k+1) | F , ij ji k k forsomeγ ∈ (0,1]andforalldistincti,j ∈ [m].Thus,w(cid:2)ehavep∗γE[W (k+1)+W(cid:3) (k+1) | F ] ≤ ij ji k H (k+1). By Corollary 2, for the random dynamics {x(k)} driven by {W(k)} and started at ar- ij bitrary (t ,v) ∈ Z+×Rm, it follows that 0 ∞ p∗γ E (W (k)+W (k))(x (k)−x (k))2 ≤ E[V (x(t ),t )]. ij ji i j π 0 0   kX=t0 Xi<j   As a consequence, ∞ (W (k)+W (k))(x (k)−x (k))2 < ∞ almost surely. ij ji i j kX=t0Xi<j Therefore, by Lemma 3, we conclude that lim (x (k,ω)−x (k,ω)) = 0 for any i,j belonging to k→∞ i j the same connected component of G∞(ω), for almost all ω ∈ Ω. By Lemma 2 it follows that every indexi ∈ [m]isergodicforalmostallω ∈ Ω.Byconsideringtheinitialconditions(t ,e )∈ Z+×Rm 0 ℓ for all ℓ ∈[m], the assertion follows. Q.E.D. Theorem 4 shows that the dynamics in (2) is convergent almost surely for aperiodic chains {W(k)} ∈ P∗. Moreover, the theorem also characterizes the limiting points of such a dynamics as well as the limit matrices of the products W(k : t ) as k → ∞. 0 4. Balanced Chains. In this section, we characterize a subclass of P∗ chains, namely the class of strongly aperiodic balanced chains. We firstshow that this class includes many of the chains that have been studied in the existing literature. Then, we prove that any aperiodic balanced chain belongs to the class P∗. We also show that a balanced independent random chain is strongly aperiodic, thus concluding Theorem 1. Before continuing our analysis on balanced chains, let us discuss some of the well-known sub- classes of such chains: 1. Balanced Bidirectional Chains: We say that an independent chain {W(k)} is a balanced bidirectional chain if there exists some α > 0 such that E[W (k)] ≥ αE[W (k)] for all k ≥ 1 ij ji and i,j ∈ [m]. These chains are in fact balanced, since for any S ⊂ [m] we have: E[W (k)] = E W (k) ≥ E αW (k) = αE[W (k)]. SS¯  ij   ji  S¯S i∈XS,j∈S¯ i∈XS,j∈S¯     Examples of such chains are bounded bidirectional deterministic chains, which are the chains such that A (k) > 0 implies A (k) > 0 for all i.j ∈ [m] and all k ≥ 1, and the positive ij ji entries are uniformly bounded from below by some γ > 0 (i.e., A (k) >0 implies A (k) ≥ γ ij ij for all i,j ∈ [m] and all k ≥ 1). In this case, for A (k) > 0, we have A (k) ≥ γ ≥ γA (k) ij ij ji and for A (k) = 0, we have A (k) = 0 and, hence, in either of the cases A (k) ≥ γA (k). ij ji ij ji Therefore, bounded bidirectional chains are examples of balanced bidirectional chains. Such chains have been considered in [6, 1, 2].