middle east, your idea of a tangent line at p as a line that meets the curve at p, and in some disc around p lies entirely on one side, is an excellent one. indeed that is euclid's original definition of a tangent line to a circle (a line that meets but does not "cut" i.e. does not cross the circle. however as fact checker points out, this definition can fail for some curves that are not convex at p, i.e. that have an "inflection point" at p.
one must also be careful if one wants to detect curves which do not have a tangent line. i.e. if a curve is known to have a tangent line at p, then any line that meets the curve at p, and lies entirely on one side of the curve near p, is the tangent line. of course there may not be any such line, as with factchecker's cubic. but there may also be more than one such line, as with the graph of the curve y = |x|. then it seems there is no tangent line. i.e. it seems that if there is a curve through p and lying entirely on one side near p, and if there is only one such line, then it seems the curve does have a tangent line at p, and that unique line is it. so for all convex curves, it seems your idea both determines if a tangent line exists, and when it does, identifies that line.
one could try to modify your idea to contain a more naive version of fresh42's definition by saying that it is a line that meets the curve, in some disc around p, only at p, but such that if we rotate it a tiny amount (in one direction or the other) it then meets the curve in another point near p. I.e. looking only at one half of the line at a time, it lies entirely on one side of the curve, but after a small rotation, no longer does so.
this would cover factchecker's cubic, and essentially any algebraic curve, but it does not work for really crinkly curves like y = x^2.sin(1/x), where the tangent line at (0,0) actually meets the curve many times near there and never lies entirely on one side of the curve in any neighborhood of (0,0). of course you may not care about such non algebraic examples.
but in fact such an example can easily be treated by this definition also as follows: I.e. if there is a unique line that meets the curve infinitely often on every disc around p, then that is the tangent line. (In particular this implies that a line is its own tangent at p.) If more than one line does so, there is no tangent line. If every line through p has some disc around p on which it meets the curve only finitely often , then the previous definition works.
note also that an elementary way to state fresh's limit definition is to mimic euclid again and just say the line makes "an angle of zero" with the curve at p. i.e. in euclid's language, "between the tangent line and the curve, no other line (through p) can be interposed", in any small region around p. I.e. if we form any angle at all from two lines intersecting at p, such that the angle is bisected by the tangent line, then in some disc around p, the curve will lie entirely within the angle formed by the two lines.
notice that the concept of limit does appear in our discussion in the form of the "universal quantifier" word "every". when we say something is true for every line, we are really taking a limit, i.e. an infinitely good approximation. so if the correct use of the word "every" may be considered as not requiring calculus, then neither does the definition of a tangent line. if one wonders whether this "limiting" language precedes calculus, note that euclid's proposition that "no other line may be interposed between the tangent and the circle", is logically equivalent to saying that "every other line fails to so interpose", hence is logically a limit statement. thus i agree with fresh42 that some form of limiting statement is needed, but simply point out that euclid already gave one.