To rationalize the expression (5+2√6)/(7+4√3), the first step is to multiply both the numerator and denominator by the conjugate of the denominator, which is (7-4√3). This process eliminates the square root in the denominator, making it easier to express the fraction in the form a-b√3. After performing the multiplication and simplifying, the values of a and b can be determined. The discussion emphasizes the importance of starting with the rationalization technique to solve the problem effectively.