Hilbert Spaces Joseph Muscat 2014-5-27 (A revised and expanded version of these notes are now published by Springer.) 1 Inner Product Spaces 1.1 Introduction Definition An inner-product on a vector space X is a map , : X X C h i × → such that x,y +z = x,y + x,z , h i h i h i x,λy = λ x,y , h i h i y,x = x,y , h i h i x,x > 0; x,x = 0 x = 0. h i h i ⇔ In words, it is said to be a positive-definite sesquilinear form. The simplest examples are RN and CN with x,y = N x¯ y ; the h i n=1 n n square matrices of size N N also have an inner-product given by A,B = N A¯ B . × P h i i,j=1 ij ij PProposition 1.1 1. x,x is real (and positive) and is denoted by x 2 h i k k 2. x,y = 0 x y = 0 (put x = y) h i ∀ ⇒ ¯ 3. λx,y = λ x,y (anti-linear); x+y,z = x,z + y,z ; h i h i h i h i h i 1 1.1 Introduction J Muscat 2 4. λx = λ x . k k | |k k Definition Two vectors x,y are orthogonal when x,y = 0, written h i as x y. The angle between two vectors is given by cosθ = x,y / x y . ⊥ h i k kk k Proposition 1.2 1. x+y 2 = x 2 +2Re x,y + y 2. k k k k h i k k 2. (Pythagoras) If z = x+y and x,y = 0 then z 2 = x 2+ y 2. h i k k k k k k 3. For any orthogonal vectors x,y, x 6 x+y . k k k k 4. For any non-zero vectors x,y, there is a unique vector z and a unique scalar λ such that x = z +λy and z y. ⊥ Proof. The first three statements follow immediately from the axioms and the first proposition. For the last statement, let z = x λy as required; − then z y holds λ = y,x / y,y (check). ⊥ ⇔ h i h i (cid:3) Proposition 1.3 Cauchy-Schwarz x,y 6 x y |h i| k kk k Proof. Decompose x into orthogonal parts x = (x λy)+λy where λ = hy,xi − y,y h i (assuming y = 0); nowusePythagoras’ theorem, anddeduce that λy 6 x . 6 | | k k (cid:3) Note that Pythagoras’ theorem and Cauchy-Schwarz’s inequality are still valid even if the ‘inner-product’ is not positive definite but just semi-definite, as long as y = 0. k k 6 Corollary x+y 6 x + y k k k k k k The proof is simply an application of the Cauchy-Schwarz inequality to the expansion of x+y 2. k k (cid:3) 1.1 Introduction J Muscat 3 Hence x is a norm, and all the facts about normed vector spaces apply k k to inner-product spaces. In particular they are metric spaces with distance d(x,y) = x y , convergence of sequences makes sense as x x n k − k → ⇔ x x 0, continuity and dual spaces also make sense. Inner-product n k − k → spaces are special normed spaces which not only have a concept of length but also of angle. Definition A Hilbert space is an inner-product space which is com- plete as a metric space. Proposition 1.4 The closure of a linear subspace remains a linear subspace. ¯ Proof. Ifx ,y A, thenlim x +lim y = lim (x +y ) A. n n n n n n n n n ∈ →∞ →∞ →∞ ∈ (cid:3) Recall that finite dimensional subspaces are always closed. (exercise) From the theory of Banach spaces we know that addition and scalar multiplication are continuous operations. Proposition 1.5 +,λ , , are continuous in each variable. · h i Proof. If x x < δ and y y < δ then ′ ′ k − k k − k (x+y) (x +y ) 6 x x + y y < 2δ. ′ ′ ′ ′ k − k k − k k − k Similarly, λ(x x) = λ x x < λ δ. ′ ′ k − k | |k − k | | We can consider φ := y, to be a functional mapping x y,x . y h i → h i This functional is continuous because φ (x) = y,x 6 y x . In fact y | | |h i| k kk k φ = y . y k k k k (cid:3) It follows that taking limits commutes with the inner-product: x, lim y = lim x,y . n n h n i n h i →∞ →∞ Doallnormsonvectorspacescomefrominner-products, andifnot, which normed vector spaces are in fact inner-product spaces? The answer is given by 1.1 Introduction J Muscat 4 Proposition 1.6 (Parallelogram law) A norm comes from an inner-product if, and only if, it satisfies x+y 2+ x y 2 = 2( x 2 + y 2) k k k − k k k k k Proof. Expanding and adding x + y 2 and x y 2 gives the paral- k k k − k lelogram law. Conversely, subtracting the two gives 4Re x,y . Hence, by h i noticing that Im x,y = Rei x,y = Re ix,y , we get the polarization h i − h i h i identity 1 x,y = y +x 2 y x 2 +i y +ix 2 i y ix 2 . h i 4 k k −k − k k k − k − k (cid:0) (cid:1) The converse is true but harder to prove. (cid:3) This law can be generalized further. If ωN = 1 (N > 2), then x,y = h i 1 N ωn y + ωnx 2. Even more generally, x,y = 1 y + zx 2dz. N n=1 k k h i 2πi S1k k That is, the inner product x,y is a ‘complex’ average of lengths on a ball P h i R of radius x , centred at y. k k Proposition 1.7 L2(R) is a Hilbert space with inner-product f,g = ¯ h i fg. R Proof. We assume L2(R) is a complete normed space. Verifying the parallelogram law, f +g 2 + f g 2 = ( f 2+f¯g +fg¯+ g 2 + f 2 f¯g fg¯+ g 2. | | | − | | | | | | | − − | | Z Z Z Hence find the associated inner-product from the previous proposition. (cid:3) 1.1.1 Exercises 1. Find the angle between sin and cos in the space L2[ π,π]. − 2. Show that ℓ2 is a Hilbert space with inner-product hx,yi = ∞n=1x¯nyn. Show further that l1 and l are not inner-product spaces by finding ∞ P two sequences which do not satisfy the parallelogram law. 3. Similarly show that L and L1 are not inner-product spaces. ∞ 1.1 Introduction J Muscat 5 4. Show that the Cauchy-Schwarz inequality becomes an equality if, and only if, x = αy α. ∃ 5. Let d = inf x+λy ; show that λ k k x,y 2 6 ( x 2 d2) y 2. |h i| k k − k k 6. Show that any finite-dimensional subspace of a Hilbert space is closed. Deduce that the set of polynomials of degree at most m forms a closed linear subspace of L2[a,b] with dimension m+ 1; find a basis for this space. 7. Show that the set of continuous functions C[a,b] is a non-closed linear subspace of L2[a,b]. 8. Starting from a norm . that satisfies the parallelogram law, we can k k create an inner-product using Prop. 6, from which we can define a norm . := .,. . Show that the two norms are identical. 1 k k h i 9. Show that twopinner-products, on the same vector space, are conformal i.e. give the same angles between vectors, if, and only if, x,y = h i1 λ x,y for some positive real λ. h i2 10. We have shown that an inner-product is determined by a norm that satisfies the parallelogram law. More generally, show that any bilinear form A(x,y) that satisfies A(y,x) = A(x,y), is determined by its real quadratic form q(x) = A(x,x), as A(x,y) = 1(q(x + y) + q(x y) + 4 − iq(x+iy) iq(x iy)). − − 11. Showthattherealandimaginarypartsofaninner-productgivesymmetric/anti- symmetric real-valued inner-products. Thus a complex inner-product has more structure than a real (symmetric) inner-product e.g. C2 has more structure than R4. 12. Let H H be the product of two inner-product spaces. Show that 1 2 × x ,x , y ,y := x ,y + x ,y h 1 2 1 2 i h 1 1iH1 h 2 2iH2 (cid:0) (cid:1) (cid:0) (cid:1) defines an inner-product on H H . 1 2 × 13. Show that the formula f,g := f(x)g(x)w(x)dx, where w(x) is a h i positive real function, defines an inner-product. The resulting space of R functions is called a weighted L2 space. 1.2 Orthogonality J Muscat 6 14. * Consider the vector space of holomorphic functions f : R R+ C × → which satisfy sup f 2dx < . Show that the formula f,g := y>0 | | ∞ h i lim f(x+iǫ)g(x+iǫ)dx gives an inner-product. ǫ 0+ R → R 1.2 Orthogonality Definition The orthogonal space of a set A is the set A = x : y,x = 0 y A . ⊥ { h i ∀ ∈ } 1.2.1 Exercises 1. Show that 0 = X, X = 0. ⊥ ⊥ 2. Show that if x = X then x = 0. ⊥ { } 3. Show that if x = 0 then X is one-dimensional. ⊥ { } Proposition 1.8 A is a closed linear subspace. ⊥ Proof. ThatA isalinearsubspace followsfromthelinearityoftheinner- ⊥ product. Let x A . That is, there is a sequence of vectors x A such ⊥ n ⊥ ∈ ∈ that x x. Now, for any y A, x,y = lim x ,y = lim x ,y = 0. n n n n n → ∈ h i h i h i Hence x A . ⊥ ∈ (cid:3) Proposition 1.9 1. A A 0; ⊥ ∩ ⊆ 2. A B B is a closed subspace of A ; ⊥ ⊥ ⊆ ⇒ 3. A A . ⊥⊥ ⊆ Proof. (i) is left as an easy exercise. For (ii), let x B i.e. b,x = ⊥ ∈ h i 0 b B. In particular this is true for b A so that x A . For (iii), let ⊥ ∀ ∈ ∈ ∈ x A, then y,x = 0, y A . Hence x,y = 0 y A . Hence x A . ⊥ ⊥ ⊥⊥ ∈ h i ∀ ∈ h i ∀ ∈ ∈ (cid:3) NotethatA isalwaysaclosedlinearsubspaceevenifAisn’t. Question: ⊥⊥ if M is a closed linear subspace is it necessarily true that M = M ? ⊥⊥ 1.3 Least Distance in Hilbert spaces J Muscat 7 1.3 Least Distance in Hilbert spaces Definition Aconvexset is aset Athat includes all linesegments between its points: x,y A, t [0,1], tx+(1 t)y A. ∀ ∈ ∀ ∈ − ∈ Exercise: Show that linear subspaces and balls are convex. Theorem 1.10 If M is a closed convex set and x is any point of a Hilbert space H, then there is a unique point x M which is 0 ∈ closest to x. x H !x M : d(x,x ) 6 d(x,y) y M. 0 0 ∀ ∈ ∃ ∈ ∀ ∈ Proof. Let d be the smallest distance from M to x i.e. d = inf d(x,y) y M ∈ (how do we know that d exists?) Then there is a sequence of vectors y M n ∈ such that x y = d(x,y ) d. Now, by the parallelogram law, n n k − k → y y 2 = 2 y x 2 +2 y x 2 (y +y ) 2x 2 n m n m n m k − k k − k k − k −k − k 6 2 y x 2 +2 y x 2 4d2 n m k − k k − k − 0 → as n . Hence (y ) is a Cauchy sequence. But M is closed (hence n → ∞ complete) and so y x M. It follows, by continuity of the norm, that n 0 → ∈ x x = d. 0 k − k Suppose y M is another closest point to x, i.e. d(x,y) = d. Then ∈ x y 2 = 2 x x 2 +2 y x 2 (x +y) 2x 2 0 0 0 k − k k − k k − k −k − k 6 2 x x 2 +2 y x 2 4d2 0 k − k k − k − = 0 (cid:3) Note that this theorem, which does not refer to inner-products, is not true in Banach spaces in general. For example l has a closed subspace c , ∞ 0 but there is no closest sequence in c to the sequence (1,1,1,...). In fact, 0 the smallest distance is d = 1 and this is achieved by any bounded sequence (a ) with 0 6 a 6 2. n n Exercise: prove these assertions. 1.3 Least Distance in Hilbert spaces J Muscat 8 Let us concentrate on the case when M is a closed subspace of H. Theorem 1.11 For a closed linear subspace M of a Hilbert space H, x M is the closest point to x H if, and only if, x x M . 0 0 ⊥ ∈ ∈ − ∈ Proof. Let a be any point of M. Then x x = λa+z with a z, where 0 − ⊥ λ = a,x x / a 2 and z = x (x +λa). By Pythagoras’ theorem, we get 0 0 h − i k k − z 6 x x making x +λa even closer to x than the closest point x . 0 0 0 k k k − k This is only possible if λ = 0 i.e. a,x x = 0. Since a is arbitrary we get 0 h − i x x M. 0 − ⊥ Conversely, if x x a for any a M, then by Pythagoras’ theorem, 0 − ⊥ ∈ x a 2 = x x 2 + x a 2, so that x x 6 x a , making x 0 0 0 0 k − k k − k k − k k − k k − k the closest point to x. (cid:3) Corollary If M is a closed linear subspace of a Hilbert space H, then H = M M . ⊥ ⊕ Proof. Write x = x + (x x ). These two components are in M and 0 0 − M as proved previously. Moreover M M = 0. ⊥ ⊥ ∩ (cid:3) Note that this is false for Banach spaces e.g. the space l = c M for ∞ 0 6 ⊕ any linear subspace M. Corollary If M is a closed linear subspace of a Hilbert space H, then M = M. More generally, for any set A, A = [[A]]. ⊥⊥ ⊥⊥ Proof. Let x M . Then x = a + b where a M and b M . ⊥⊥ ⊥ ∈ ∈ ∈ Then 0 = b,x = b,a + b,b = b 2, making b = 0 and x M. For the h i h i h i k k ∈ secondpart, notethat[[A]]isthesmallest closedlinearsubspace containingA. Therefore, since A [[A]], we get [[A]]⊥ A and hence A [[A]]⊥⊥ = [[A]]. ⊥ ⊥⊥ ⊆ ⊆ ⊆ (cid:3) Corollary [[A]] is dense in H if, and only if, A = 0. ⊥ ¯ Proof. Suppose [[A]] = H. Then A = A = H = 0, since A is a ⊥ ⊥⊥⊥ ⊥ ⊥ ¯ closed linear subspace. Conversely, [[A]] = A = 0 = H. ⊥⊥ ⊥ (cid:3) 1.3 Least Distance in Hilbert spaces J Muscat 9 1.3.1 Exercises 1. Find the closest points x to a closed linear subspace M from (i) a 0 point x M, (ii) a point x M . ⊥ ∈ ∈ 2. In the decomposition x = a+b with a M and b M , show that a ⊥ ∈ ∈ and b are unique. 3. Deduce that if H = M N where M is a closed linear subspace and ⊕ M N then N = M . ⊥ ⊥ 4. Show that, if M is a closed linear subspace of N, then M (M N) = ⊥ ⊕ ∩ N. 1.3.2 Projections Let us take a closer look at the map from x to x M. 0 ∈ Definition A projection is a linear map P : H H such that → P2 = P. An orthogonal projection is one such that kerP imP. ⊥ Theorem 1.12 If M is a closed linear subspace of a Hilbert space H then the map P : H H defined by Px = x is a continuous 0 → orthogonal projection with imP = M and kerP = M . ⊥ Proof. By the definition of P, for any a H, (a Pa) M and this ⊥ ∈ − ∈ property defines Pa M. ∈ P is linear since (x+y) (Px+Py) = (x Px)+(y Py) M and ⊥ − − − ∈ Px+Py M, hence P(x+y) = Px+Py. Similarly P(λx) = λPx. ∈ P is a projection since Px M implies that P2x = Px. ∈ P is onto M since for any x M, Px = x. Moreover x kerP Px = ∈ ∈ ⇔ 0 x = x Px M . ⊥ ⇔ − ∈ P is continuous since x 2 = x Px 2+ Px 2 by Pythagoras’ theorem k k k − k k k so that Px 6 x . k k k k (cid:3) 1.3.3 Exercises 1. Show that the map (x,y,z) (x,0,0) is an orthogonal projection in 7→ R3. Find a projection which is not orthogonal. 2. Let P,Q be two commuting projections. Show that PQ is also a pro- jection. 1.3 Least Distance in Hilbert spaces J Muscat 10 3. Show that if P is a projection then H = kerP imP. ⊕ 4. Show that any orthogonal projection is continuous, and imP kerP. ⊥ Hence note that there is a 1-1 correspondence between closed linear subspaces and orthogonal projections. 5. Show that if P is an orthogonal projection onto M then I P is an − orthogonal projection onto M . ⊥ 6. SupposethatH = M N. Showthat,ifP andQdenotetheorthogonal ⊕ projections onto M and N respectively, then I = P +Q and PQ = 0. Extend this by induction to the case when H = M ... M to get 1 n ⊕ ⊕ I = P +...P with P P = P δ . 1 n i j i ij 7. Consider two closed linear subspaces M and M (with associated pro- 1 2 jections P and P ). Show that the iteration x = P P x starting 1 2 n+1 2 1 n from x = y converges to the closest point y M M . 0 0 1 2 ∈ ∩ 8. Let M =< y > (a closed linear subspace). Find the orthogonal pro- jection P which maps any point x to its closest point in M. (Ans: Px = y,x y) h i 9. Let M = y ,...,y . Find P again. (Ans: Px = α y where h 1 Ni i i i α = y ,x / y 2. i i i h i k k P 1.3.4 Examples: Least Squares Approximation 1. Find the closest point in the plane 2x+y 3z = 0 to the point x = − (5,2,0). Letting n = (2,1, 3), the equation of the plane is given by − n x = 0. Therefore we are looking for a point x in the plane such 0 · that x x is perpendicular to the plane i.e. is parallel to n. Hence, 0 − x = x+λn such that it satisfies n x = 0. Substituting gives λ. 0 0 · 2. Find the best-fitting (closest in the space L2) cubic polynomial to the function sin in the region [0,2π]. The space of cubic polynomials, a+bx+cx2+dx3, is a four-dimensional closedlinearsubspaceoftheHilbertspaceL2[0,2π],withbasis1,x,x2,x3. It must therefore have an element p(x) = a+bx+cx2 +dx3 which is closest to sinx. In fact, p is characterized by the condition p sin − ⊥ q, q M. In particular we get four equations p(x) sin(x),xi = 0 ∀ ∈ h − i for i = 0,...,3, or, equivalently, p(x),xi = sin(x),xi . The right- h i h i hand sides can be worked out, giving four linear equations in the four unknowns a,b,c,d, which can be extracted by solving.