Things are slightly more complicated in the infinite dimensional case when unbounded operators are considered. In fact when physicists say Hermitian, the definition they often give is something like A is Hermitian if \langle u | A | v \rangle=\langle v|A|u\rangle^* for all u,v\in \mathcal{H} however they also want the spectral theorem and all it's consequences to apply to A. Strictly speaking this definition merely says that A is what mathematicians would call symmetric and the spectral theorem does NOT necessarily apply to such operators.
To give the precise definition for unbounded operators, it is necessary to pay close attention to the domains of the operators in question. So, suppose that an operator A is defined on the domain \mathcal{D}(A) which is dense in \mathcal{H}. Then the adjoint A^\dagger of A is defined on the domain consisting of all u\in \mathcal{H} such that the map v\mapsto \langle u,Av \rangle for v\in \mathcal{D}(A) extends to a bounded linear functional on all of \mathcal{H}. (Note that this extension is unique since the domain is dense.) In this case, the adjoint A^\dagger is defined to be the element of \mathcal{H} corresponding to this linear functional via the Riesz-representation theorem so that the equality \langle A^\dagger u,v\rangle=\langle u,Av\rangle as expected.
The definition of symmetric (or Hermitian as physicists often call it) is simply that \langle Au,v\rangle=\langle u,Av\rangle for all u,v\in \mathcal{D}(A). The definition of self-adjoint is that A^\dagger =A. The key point here is that in general, \mathcal{D}(A^\dagger) \supseteq \mathcal{D}(A) so a self-adjoint operator has the extra condition that it's adjoint has the same domain as the operator itself which a merely symmetric operator need not satisfy. It is often possible to extend a symmetric operator to a self-adjoint operator by extending it's domain but, contrary to what a previous poster stated, even if this is possible I believe there is no guarantee that you get a unique extension. A related class of operators, the essentially self-adjoint ones, are precisely those which have a unique extension to a self-adjoint operator but this does not include all symmetric operators.
Since the majority of physics books I've read anyway (at least the more introductory ones) do not pay very close attention to the precise domain on which unbounded operators are defined, the above distinction between self-adjoint and symmetric seems to get blurred or not mentioned at all but as I said, the spectral theorem only applies to self-adjoint operators not Hermitian(symmetric) ones so these are what are really meant if you want to be precise.