I think you can work this out with some approximations. If assume that the ground station is at sea level and that the Earth is round, then we need to compute the horizon for the satellite. Consider this figure from wikipedia:
We will use radial vectors to describe the position of the ground station and satellite. Let the position of the satellite at point O be described by a vector ##\vec{s}##, with length ##|\vec{s}| = R + h ##, where ##R## is the Earth radius and ##h## is the height above sea-level of the satellite. (Note: the figure has used ##s## to denote an arc length, please ignore that.) The vector ##\vec{g}## will describe the ground station (not on the figure), with length ##|\vec{g}| = R##.
Referring to the figure, the angle between the horizon point and the satellite vector is ##\gamma## and some easy trig tells us that
$$ \cos\gamma = \frac{R}{R+h}.$$
Note that by definition, ##0<\gamma \leq \pi/2##. For the satellite to be visible from the ground station, the angle ##\theta## between ##\vec{g}## and ##\vec{s}## must be less than ##\gamma##, else the ground station is beyond the horizon. Therefore ##\theta \leq \pi/2## as well. But we can use the dot product to compute ####
$$\cos\theta = \frac{ \vec{g}\cdot \vec{s} }{ |\vec{g}||\vec{s}|} = \frac{ \vec{g}\cdot \vec{s} }{ R(R+h)}.$$
Since ##\gamma,\theta \leq \pi/2##, the condition that ##\theta < \gamma## is that ##\cos \theta > \cos\gamma##, so the condition that the satellite be visible is that
$$ \frac{ \vec{g}\cdot \vec{s} }{ R(R+h)} > \frac{R}{R+h}$$
or
$$ \vec{g}\cdot \vec{s}> R^2 .$$
If you need to consider terrestrial obstructions or atmospheric refraction, most of the above geometry will still be useful with appropriate modifications.
