On closed convex subsets in Hilbert space
In mathematics, the Hilbert projection theorem is a famous result of convex analysis that says that for every vector
in a Hilbert space
and every nonempty closed convex
there exists a unique vector
for which
is minimized over the vectors
; that is, such that
for every ![{\displaystyle c\in C.}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Finite dimensional case
Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.
Consider a finite dimensional real Hilbert space
with a subspace
and a point
If
is a minimizer or minimum point of the function
defined by
(which is the same as the minimum point of
), then derivative must be zero at
In matrix derivative notation[1]
Since
is a vector in
that represents an arbitrary tangent direction, it follows that
must be orthogonal to every vector in
Statement
Detailed elementary proof
Proof that a minimum point
existsLet
be the distance between
and
a sequence in
such that the distance squared between
and
is less than or equal to
Let
and
be two integers, then the following equalities are true:
and
Therefore
(This equation is the same as the formula
for the length
of a median in a triangle with sides of length
and
where specifically, the triangle's vertices are
).
By giving an upper bound to the first two terms of the equality and by noticing that the middle of
and
belong to
and has therefore a distance greater than or equal to
from
it follows that:![{\displaystyle \|c_{n}-c_{m}\|^{2}\;\leq \;2\left(\delta ^{2}+{\frac {1}{n}}\right)+2\left(\delta ^{2}+{\frac {1}{m}}\right)-4\delta ^{2}=2\left({\frac {1}{n}}+{\frac {1}{m}}\right)}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
The last inequality proves that
is a Cauchy sequence. Since
is complete, the sequence is therefore convergent to a point
whose distance from
is minimal.![{\displaystyle \blacksquare }](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Proof of characterization of minimum point when
is a closed vector subspaceAssume that
is a closed vector subspace of
It must be shown the minimizer
is the unique element in
such that
for every
Proof that the condition is sufficient:
Let
be such that
for all
If
then
and so
which implies that
Because
was arbitrary, this proves that
and so
is a minimum point.
Proof that the condition is necessary:
Let
be the minimum point. Let
and
Because
the minimality of
guarantees that
Thus
is always non-negative and
must be a real number.
If
then the map
has a minimum at
and moreover,
which is a contradiction.
Thus
![{\displaystyle \blacksquare }](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Proof by reduction to a special case
It suffices to prove the theorem in the case of
because the general case follows from the statement below by replacing
with
ProofLet
be as described in this theorem and let
This theorem will follow from the following lemmas.
Lemma 2 — A sequence
satisfying the hypotheses of Lemma 1 exists.
Lemma 2 and Lemma 1 together prove that there exists some
such that
Lemma 1 can be used to prove uniqueness as follows.
Suppose
is such that
and denote the sequence
by
so that the subsequence
of even indices is the constant sequence
while the subsequence
of odd indices is the constant sequence
Because
for every
in
which shows that the sequence
satisfies the hypotheses of Lemma 1.
Lemma 1 guarantees the existence of some
such that
in
Because
converges to
so do all of its subsequences.
In particular, the subsequence
converges to
which implies that
(because limits in
are unique and this constant subsequence also converges to
). Similarly,
because the subsequence
converges to both
and
Thus
which proves the theorem. ![{\displaystyle \blacksquare }](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Consequences
Properties
Expression as a global minimum
The statement and conclusion of the Hilbert projection theorem can be expressed in terms of global minimums of the followings functions. Their notation will also be used to simplify certain statements.
Given a non-empty subset
and some
define a function
A global minimum point of
if one exists, is any point
in
such that
in which case
is equal to the global minimum value of the function
which is:
Effects of translations and scalings
When this global minimum point
exists and is unique then denote it by
explicitly, the defining properties of
(if it exists) are:
The Hilbert projection theorem guarantees that this unique minimum point exists whenever
is a non-empty closed and convex subset of a Hilbert space.
However, such a minimum point can also exist in non-convex or non-closed subsets as well; for instance, just as long is
is non-empty, if
then
If
is a non-empty subset,
is any scalar, and
are any vectors then
which implies:![{\displaystyle {\begin{alignedat}{6}\min &(sC,sx)&&=s&&\min(C,x)\\\min &(-C,-x)&&=-&&\min(C,x)\\\end{alignedat}}}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
![{\displaystyle {\begin{alignedat}{6}\min \left(C+x_{0},x+x_{0}\right)&=\min(C,x)+x_{0}\\\min \left(C-x_{0},x-x_{0}\right)&=\min(C,x)-x_{0}\\\end{alignedat}}}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
![{\displaystyle {\begin{alignedat}{6}\min &(C,-x){}&&=\min(C+x,0)-x\\\min &(C,0)\;+\;x\;\;\;\;&&=\min(C+x,x)\\\min &(C-x,0){}&&=\min(C,x)-x\\\end{alignedat}}}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Examples
The following counter-example demonstrates a continuous linear isomorphism
for which
Endow
with the dot product, let
and for every real
let
be the line of slope
through the origin, where it is readily verified that
Pick a real number
and define
by
(so this map scales the
coordinate by
while leaving the
coordinate unchanged).
Then
is an invertible continuous linear operator that satisfies
and
so that
and
Consequently, if
with
and if
then ![{\displaystyle \,\min(A(C),A\left(x_{0}\right))\neq A\left(\min \left(C,x_{0}\right)\right).}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
See also
Notes
- ^ Because the norm
is continuous, if
converges in
then necessarily
converges in
But in general, the converse is not guaranteed. However, under this theorem's hypotheses, knowing that
in
is sufficient to conclude that
converges in ![{\displaystyle H.}](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
- ^ Explicitly, this means that given any
there exists some integer
such that "the quantity" is
whenever
Here, "the quantity" refers to the inequality's right hand side
and later in the proof, "the quantity" will also refer to
and then
By definition of "Cauchy sequence,"
is Cauchy in
if and only if "the quantity"
satisfies this aforementioned condition. - ^ Technically,
means that the addition map
defined by
is a surjective linear isomorphism and homeomorphism. See the article on complemented subspaces for more details.
References
- ^ Petersen, Kaare. "The Matrix Cookbook" (PDF). Retrieved 9 January 2021.
Bibliography