Ive come up with this so far. It only allows to fit the rectangles in portrait or landscape mode, not a combination of both, nor diagonally.
Any triangle can be made into two right triangles (I think).
A given right triangle has bases t1,t2, and hypotenuse t3.
The number of rectangles (r1,r2) that can fit in the first row in the triangle is
[tex]N = \downarrow \frac{[(t1-r1)\frac{t2}{t1}]}{r1}[/tex]
(down arrow denotes rounding down to nearest integer)
The total number of rectangles that can fit in the triangle is then:
[tex]N = \Sigma_{n=1} \downarrow \frac{[(t1-nr1)\frac{t2}{t1}]}{r1}[/tex]
where the sum is from n=1 until the outcome of the expression is zero (I don't know how to denote this mathematically).
This needs to be done twice, the second time you switch between r1 and r2, to find out if landscape or portrait offers the bigger solution.