Hi, and welcome to the forum!
You can do this either with or without trigonometry. But in either case I have not found a formula for the largest square side, only an approximate value.
Let the $AC=b$ and $BH=h$. Then the area of $\triangle ABC$ is $(1/2)bh$. It also equals the sum of areas of $\triangle KBL$, the area of the square and the combined area of $\triangle AKN$ and $\triangle CLM$. Let the square side be $x$. Then the combined area of $\triangle AKN$ and $\triangle CLM$ is $(1/2)x(b-x)$. Can you express the area of the remaining figures and write the equation?
For the future, please read the http://mathhelpboards.com/rules/, especially rule #11 (click the Expand button on top).