"Cosine" is a function. "Cos(t)" is a number- the value of the cosine function at the number, t. So, yes, you can multiply by such a thing or divide by it if you are careful about values of t that make that number 0.
Here, I would row reduce that in the "standard" way. Since the first row of the first column is "5 cos(t)", divided entire first row by it to make that entry "1". The first row becomes:
[tex]\left[\begin{array}{cc}1 & \frac{sin(t)}{cos(t)}\end{array}\right| -\frac{1}{5}\right][/tex]
Now, to change that "sin(t)" in the second row of the first column to "0", subtract sin(t) times the new first row from that second row. The number in the second column of the second row becomes
[tex]-cos(t)- sin(t)\frac{sin(t)}{cos(t)}= -cos- \frac{sin^2(t)}{cos(t)}= \frac{-cos^2(t)- sin^2(t)}{cos(t)}[/tex]
and we certainly can use a "trig identity": [itex]sin^2(xt)+ cos^2(t)= 1[/itex]
to get
[tex]-\frac{1}{cos(t)}[/tex]
(moral- you can't expect to see trig identities from the start- go ahead, do the algebra and they will come!)
For the third column, then, we will need to subtract (-1/5) times sin(t) from sin(t)+ (2/5)cos(t): sin(t)+ (2/5)cos(t)+ (1/5)sin(t)= (4/5)sin(t)+ (6/5)sin(t)+ (2/5)cos(t) so that after reducing the first column you have
[tex]\left[\begin{array}{cc}1 & \frac{sin(t)}{cos(t)} \\ 0 & \frac{1}{cos(t)}\end{array}\right|\left|\begin{array}{c}-\frac{1}{5} \\ \frac{6}{5}sin(t)+ \frac{2}{5}cos(t)\end{array}\right]=\left[\begin{array}{cc}1 & tan(t) \\ 0 & sec(t)\end{array}\right|\left|\begin{array}{c}-\frac{1}{5} \\ \frac{6}{5}sin(t)+ \frac{2}{5}cos(t)\end{array}\right][/tex]