|a + b|^2 = (a + b)[tex]\bullet[/tex](a + b)
and |a + b| = sqrt(3) ==> |a + b|^2 = 3
Now, use the fact that the dot product is associative, distributive, and commutative and the two equations above to see if you can evaluate (2a - 5b) )[tex]\bullet[/tex] (b + 3a).
Work with (a + b) [itex]\cdot[/itex] (a + b) = 3. You also know that a and b are unit vectors, which means that a [itex]\cdot[/itex] a = 1 and b [itex]\cdot[/itex] b = 1.