The separating paraboloid of two distributions in Euclidean space

Gérard Letac^{∗}

Abstract

Given two random variables X and Y in the affine Euclidean space E of dimension n + 1, having second moments and different means, we prove the existence of an orthonormal coordinate system (x_{0}, …, x_{n}) of E and the existence of numbers (c_{1}, …, c_{n}) such that the paraboloid P

x_{0} = \frac{x_{1}^{2}}{c_{1}} + … + \frac{x_{n}^{2}}{c_{n}}

has the following property: all tangent hyperplanes H to P separate the two distributions of X and Y in the following sense: the expectations of the squared distance of X and Y to H coincide.

I Separating two distributions

Given two finite clouds X and Y of points in a Euclidean space E of respective size |X| and |Y | one can will to separate them in some manner. One way to proceed is to make strong use of the underlying Euclidean structure and to consider the points ξ ∈ E such that the mean square distances of points of X to ξ coincide with the one with Y, namely:

\frac{1}{|X|} ∑_{x∈X} ∥x − ξ∥^{2} = \frac{1}{|Y |} ∑_{y∈Y} ∥y − ξ∥^{2}.  (1.1)

Denoting by m_{X} and m_{Y} the respective barycenters of X and Y and assuming that m = m_{Y} − m_{X} ≠ 0 one easily sees that 1.1 is equivalent to

2⟨ξ, m⟩ = \frac{1}{|X|} ∑_{x∈X} ∥x∥^{2} − \frac{1}{|Y |} ∑_{y∈Y} ∥y∥^{2},  (1.2)

where ⟨ξ, m⟩ is the scalar product. Thus the set of acceptable points ξ is an affine hyperplane of E orthogonal to the difference m of the barycenters whose equation is 1.2.

At this point we make several remarks. Considering clouds of points is somewhat artificial and we may consider instead two probability distributions in E. It will be convenient for short statements to declare that they are the distributions of two random variables X and Y in E (their joint distribution being irrelevant). We need the existence of second moments of X and Y. The vectors m_{X} and m_{Y} are their expectations and the symmetric positive endomorphisms Q_{X} and Q_{Y} of E are their respective covariance operators (namely ⟨x, Q_{X}(y)⟩ = IE(⟨X −m_{X}, x⟩⟨X −m_{X}, y⟩)). Observe that IE(∥X −m_{X}∥^{2}) = trace Q_{X} and that, from a Huyghens formula, IE(∥X∥^{2}) = IE(∥X − m_{X}∥^{2}) + ∥m_{X}∥^{2}. With the above notations, still assuming m = m_{Y} − m_{X} ≠ 0 and denoting q = \frac{1}{2}(Q_{X} − Q_{Y}), then the set of ξ ∈ E such that IE(∥X − ξ∥^{2}) = IE(∥Y − ξ∥^{2}) is the affine hyperplane

∗Laboratoire de Statistique et Probabilités, Université Paul Sabatier, 31062, Toulouse, France

⟨ξ, m⟩ = \frac{1}{2}(∥m_{Y} ∥^{2} − ∥m_{X}∥^{2}) + trace q.  (1.3)

Let us particularize to the real line, ie to dim E = 1. In this case , introducing the variances σ_{Y}^{2} and σ_{X}^{2}, the unique point ξ is

ξ = \frac{m_{X} + m_{Y}}{2} + \frac{σ_{Y}^{2} − σ_{X}^{2}}{2(m_{Y} − m_{X})}.  (1.4)

Now, we introduce a different idea: instead of considering the hyperplane of points ξ such that IE(∥X − ξ∥^{2}) = IE(∥Y − ξ∥^{2}) we coin the following definition

Definition. An affine hyperplane H of the Euclidean space E is said to be separating the distributions of X and Y if the expectations of the squared distances of X and Y to H coincide.

Clearly from 1.4, choosing a vector u ∈ E with unit norm there exists one and only one affine separating hyperplane H whose normal vector is u, provided that ⟨m, u⟩ ≠ 0. Taking simply orthogonal projections on the line of direction u, its equation is given by 1.4 as the set of ξ ∈ E such that

⟨ξ, u⟩ = \frac{⟨m_{X} + m_{Y} , u⟩}{2} + \frac{q(u)}{⟨m, u⟩}.  (1.5)

Now we have too many separating hyperplanes, one for any direction u non orthogonal to m. The aim of this note is to find the enveloppe of all these separating hyperplanes. We do it in the next section.

II The separating paraboloid.

We keep the notations m and q above. The key point is to use the existence of an orthonormal basis e = (e_{0}, …, e_{n}) of E such that e_{0} is proportional to m and such that the representative matrix \frac{1}{∥m∥}[q]_{e}^{e} of q in this basis e has the following form, for 2n + 1 real numbers a, b_{1}, …, b_{n}, c_{1}, …, c_{n} :

\frac{1}{∥m∥}[q]_{e}^{e} = [ \begin{matrix}a & b_{1} & … & b_{n} \\ b_{1} & c_{1} & … & 0 \\ b_{2} & 0 & … & 0 \\ … & … & … & … \\ b_{n} & 0 & … & c_{n} \end{matrix} ] .  (2.6)

To see this, choose first an arbitrary orthonormal basis e′ = (e_{0}, e′_{1}, …, e′_{n}) such that e_{0}

is proportional to m. Thus \frac{1}{∥m∥}[q]_{e′}^{e′} is written by blocks as [ \begin{matrix}a & b \\ b^{′T} & C′ \end{matrix} ] , where b′ is a n

line vector. Now the symmetric (n, n) matrix C′ can be diagonalized and written UCU^{T} where U is an orthogonal matrix and C = diag(c_{1}, …, c_{n}). Thus there exist e such that 2.6 holds, with b = b′U.

Let us also recall a few facts about the way of computing the enveloppe of a family of hyperplanes in a Euclidean space. Suppose that the equation of each hyperplane of the family is its normal equation, ie has the form ⟨ξ, u⟩ = p(u), where u has norm 1 and p is a smooth function on the unit sphere such that p(u) = −p(−u). It is convenient to extend the definition of p to E \ \{0\} by p(x) = ∥x∥p(\frac{x}{∥x∥}). Then the enveloppe of such a family is the hypersurface parametrized by the unit sphere by the map u ↦ p′(u) where p′(u) is the gradient of p estimated in u. Here we take advantage of the fact that the space in Euclidean for identifying the differential of a real function with the gradient vector.

We now prove our theorem

Theorem. With the above notations, assume without loss of generality that (m_{X} + m_{Y})/2 = 0. Assume also that c_{i} ≠ 0 for all i = 1, …, n. Consider the paraboloid P whose equation in the basis e is

(x_{0} − a) + ∑_{i=1}^{n} \frac{1}{c_{i}} (\frac{x_{i}}{2} − b_{i})^{2} = 0.  (2.7)

Then the affine hyperplane H separates the distributions of X and Y if and only if H is tangent to P.

Proof. Since the family of separating huperplanes is given by the normal equations 1.5, with p(u) = q(u)/⟨m, u⟩, then the computation of the gradient of

p(ξ) = \frac{1}{ξ_{0}} (aξ_{0}^{2} + 2ξ_{0} ∑_{i=1}^{n} b_{i}ξ_{i} + ∑_{i=1}^{n} c_{i}ξ_{i}^{2})

where ξ = ξ_{0}e_{0}+…+ξ_{n}e_{n}, is quite easy. We get the parametric equations of the enveloppe

x_{0} = \frac{∂p}{∂ξ_{0}} = a − ∑_{i=1}^{n} c_{i} (\frac{ξ_{i}}{ξ_{0}} )^{2}  (2.8)
x_{i} = \frac{∂p}{∂ξ_{i}} = 2b_{i} + 2c_{i} (\frac{ξ_{i}}{ξ_{0}} ) for i = 1, …, n.  (2.9)

Carrying \frac{ξ_{i}}{ξ_{0}} from 2.9 to 2.8 provides 2.7.

Conversely, it is standard to check that the tangent hyperplane to P with normal direction u has equation 1.5 and thus is a separating hyperplane.

III Comments.

1. Let us consider the case where some of the c′_{i}s are 0. Without loss of generality we assume c_{1}, …, c_{k} ≠ 0 and c_{k+1} = … = c_{n} = 0. Under these circumstances from the equations 2.9 one considers the intersection with the cylindrical paraboloid P (x_{0} − a) + ∑_{i=1}^{k} \frac{1}{c_{i}} (x_{i} − \frac{b_{i}}{2} )^{2} = 0 with the n − k dimensional affine subspace F defined by x_{i} = 2b_{i} for i = k + 1, …, n. The hyperplanes H containing the intersection between F and any hyperplane which is tangent to P are the separating hyperplanes. The simplest situation for which this occurs is the cas n = 1, that is E is a Euclidean plane. Then p(ξ_{0}, ξ_{1}) = aξ_{0} + 2b_{1}ξ_{1} and the set of separating lines is the set of lines through the point (a, 2b_{1}). This occurs for instance if X and Y are standard normal in the plane with different means.

2. Coming back to the standard situation where c_{i} ≠ 0 for all i = 1, …, n one can notice that E \ P is splitted in two connected components described by f(x) = (x_{0} − a) + ∑_{i=1}^{n} \frac{1}{c_{i}} (\frac{x_{i}}{2} − b_{i})^{2} > 0 and f(x) < 0 respectively. For instance in order to see that the set S of (x_{0}, …, x_{n}) such that x_{0} > ∑_{i=1}^{k} x_{i}^{2}−∑_{i=k+1}^{n} x_{i}^{2} is connected just consider the continuous path from (1, 0, …, 0) to x ∈ S defined by x(t) = (1−t^{2}+t^{2}x_{0}, tx_{1}, …, tx_{n}) for 0 ≤ t ≤ 1. It is contained in S and this implies that the open set S is connected. The proof for the general case has the same style.

3. Note that the hyperplane 1.3 is not necessarily tangent to P. Its equation is x_{0} = a + c_{1} + … + c_{n} while the separating hyperplane orthogonal to m is x_{0} = a.

4. It should be pointed out that in general the intersection between P and the tangent hyperplane of normal direction u is not reduced to the point of tangency when P is not elliptic (ie when the sign of the c′_{i}s is not constant). This can be seen by carrying the parametric equations 2.8 and 2.9 of P in the equation 1.5 of H : we get

∑_{i=1}^{n} c_{i}(u_{i}ξ_{0} − u_{0}ξ_{i})^{2} = 0

which has other solutions than (u_{i}ξ_{0} − u_{0}ξ_{i}) = 0 for all i corresponding to the point of tangency.