Hello Danielle,
The volume of the box is the product of the area of its base and its height:
$$V=bh$$
We will be cutting squares, let's let the side length of these squares be $x$, from each corner of the cardboard sheet, so that we have flaps to fold up to make the box bottom. Hence, the area of the base is:
$$b=(12-2x)(8-2x)=4(6-x)(4-x)$$
and the height of the box is $x$, and so we have:
$$V(x)=4x(6-x)(4-x)$$
We should observe that we require $$0<x<4$$ in order to get meaningful results.
Next, we need to equate the derivative of the volume function to zero, and look at critical values in the valid domain. We will use the following rules of differentiation:
$$g(x)=k\cdot f(x)\implies g'(x)=k\cdot f'(x)$$ where $k$ is a constant.
$$u(x)=f(x)g(x)h(x)\implies u'(x)=f'(x)g(x)h(x)+f(x)g'(x)h(x)+f(x)g(x)h'(x)$$
So, we find:
$$V'(x)=4\left((1)(6-x)(4-x)+x(-1)(4-x)+x(6-x)(-1) \right)=$$
$$4\left(24-10x+x^2-4x+x^2-6x+x^2 \right)=4(3x^2-20x+24)=0$$
Applying the quadratic formula, and taking the root in the domain, we find:
$$x=\frac{20\pm\sqrt{(-20)^2-4(3)(24)}}{2\cdot3}=\frac{20\pm4\sqrt{7}}{6}=\frac{2(5\pm\sqrt{7})}{3}$$
The root in the domain is:
$$x=\frac{2(5-\sqrt{7})}{3}$$
To verify this critical value is at a maximum, we may use the first derivative test, and observe that it is positive to the left of this critical value and negative to the right, hence the critical value is at a maximum.
Now, to find the volume of the box at this value of $x$, we need to evaluate:
$$V\left(\frac{2(5-\sqrt{7})}{3} \right)=\frac{64(7\sqrt{7}+10)}{27}\approx68\text{ in}^3$$
To Danielle and any other visitors viewing this topic, I encourage you to register and post other optimization problems in our http://www.mathhelpboards.com/f10/ forum.