You appear to have correctly calculated the components of gravitational force acting on the two masses normal to and along the incline, though you may have gotten your right triangle confused (it is gravity itself that is "directly down", not the pull on the box: For the 8 kg mass they are F
S= 8(9.8)sin(35) (F
S is the force in the direction of the incline) and
F
N= 8(9.8)cos(35). For the 3.5 kg mass they are
F
S= 3.5(9.8)sin(35) and F
N= 3.5(9.8)cos(35).
I also know the force of friction is equal to the coefficient of friction times the normal force. I think the normal force is mass times the sine of theta times gravity, but I don't know how to find the force of friction.
You almost said it! The normal force is mg cos(35) (not just mass times sin- mg times cosine-look at your diagram) and the friction force is the coefficient of friction times that. The problem is that, here, you are not given the coefficient of friction- that's the "unknown" you are asked to find. Let's call it "c". The friction force on the 8 kg mass is 8(9.5)cos(35) and for the 3.5 kg mass it is 3.5(8.9)cos(35).
The "S" component of gravitational force on the 8 kg mass is pulling it down the incline, the "S" component of gravitational force on the 3.5 kg mass is holding it back (through the rope) and both frictional forces are holding it back. Taking the positive direction down the slope, we have for the total force
F= 8(9.8)sin(35)- 3.5(9.8)sin(35)- 8(9.8)c cos(35)- 3.5(9.8)c sin(35)
= (8-3.5)(9.8)sin(35)- (8-3.5)(9.8)c cos(35)
= 4.5(9.8)sin(35)- 4.5(9.8)c cos(35).
We are told that the 8 kg mass has acceleration 1.5 m/s
2 so we must have (F= ma) 4.5(9.8)sin(35)- 4.5(9.8)c cos(35)= 8(1.5).
Solve that equation for c.
Of the four forces, two (the "S" component of gravity on the 8 kg mass and the friction force of the 8 kg mass) are acting directly on the 8 kg mass. The other two (the "S" component of gravity on the 3.5 kg mass and the friction force on the 3.5 kg mass) have to act through the string connecting the two mass. The total of those two forces is the tension in the string.