Orthogonality is perfectly well-defined in Hilbert spaces. Indeed, we say that ##x\bot y## iff ##<x,y> = 0##. The idea of Diminnie orthogonality is to extend the notion of orthogonality to more general Hilbert spaces.
The Riesz representation theorem says that any continuous functional on Hilbert space ##f:X\rightarrow \mathbb{C}## has the form ##f(x) = <a,x>##.
So on Hilbert space, we get the following:
\textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|,\|b\|\leq 1\} = \|x\|\|y\|
if ##x## and ##y## are Diminnie orthogonal.
Very related is the following quantity:
\textrm{sup}\{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|^2 + \|b\|^2 \leq 1\}
This quantity is in Hilbert spaces somewhat better behaved. Indeed, we can take the Hilbert space ##X\times X## with inner product ##<(a,b),(c,d)> = <a,c> + <b,d>##. Then we can look at the following operator
\psi(a,b) = <a,x><b,y> - <b,x><a,y>
We often use the notation ##x\otimes y## for the operator ##(x\otimes y)(a,b) = <x,a><y,b>##, so we have ##\psi = x\otimes y - y\otimes x##. The quantity
\|x\otimes y - y\otimes x\|= \textrm{sup}\{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|^2 + \|b\|^2 \leq 1\}
is the norm of this functional.
Now, the space generated by all the ##x\otimes y## is called the tensor product ##X\otimes Y## and is a Hilbert space under the inner product ##<x\otimes y, z\otimes w> = <x,z><y,w>##. The associated norm is denoted as ##\|~\|_2## and we have ##\|~\|\leq \|~\|_2##.
In particular, if we have ##<x,x> = <y,y>=1##
\|x\otimes y - y\otimes x\|^2 \leq \|x\otimes y - y\otimes x\|_2^2 = <x\otimes y - y\otimes x, x\otimes y - y\otimes x> = 2 - 2<x,y><y,x>
Now, what does this have to do with our quantity
\textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|,\|b\|\leq 1\} = \|x\|\|y\|
Well, let's take ##\|x\| = \|y\|= 1## (this is the general case since we can just divide by ##\|x\|\|y\|##).
Then if we have ##\|a\|,\|b\|\leq 1##, then ##\|a\|^2 + \|b\|^2 \leq 2##. Thus we see that
<br />
\begin{eqnarray*}<br />
& &<br />
\textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|,\|b\|\leq 1\}\\<br />
& \leq & \textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|^2+\|b\|^2\leq 2\}\\<br />
& = & \frac{1}{2}\textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|^2 + \|b\|^2 \leq 1\}\\ & = & 1 - <x,y><y,x><br />
\end{eqnarray*}<br />
In fact, equality holds since we can take ##a=x## and ##b=y## and then
<a,x><b,y> - <b,x><a,y> = 1 - <y,x><x,y>
Thus we get that for ##\|x\|= \|y\| = 1## that
\textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|,\|b\|\leq 1\} = 1 - <x,y><y,x>
Thus if ##x## and ##y## are Diminnie orthogonal, then
## 1 - <x,y><y,x> = \textrm{sup} \{<a,x><b,y> - <b,x><a,y>~\vert~\|a\|,\|b\|\leq 1\} =1##
and thus easily follows that ##<x,y> = 0##.
Conversely if ##<x,y>= 0##, then we see easily that ##x## and ##y## are Diminnie orthogonal.