devious gave his reasoning: it's multiplication modulo 7:
4*12= 48= 7*6+ 6 so 4*12= 6 mod 7
8* 8= 64= 7*9+ 1 so 8*8= 1 mod 7
then 5*6= 30= 4*7+ 2 so 5*6= 2 mod 7.
HOWEVER, that is not the only answer. If fact, that 12 in "4*12" as well as the 8 in "8*8" makes me suspicious of the multiplication being "mod 7". It is common (but not required) to reduce numbers to be less that the modulo number. (Strictly speaking the objects in "modulo arithmetic" are equivalence classes of integers. You can use any number in an equivalence class to "represent" it and it is common to use the smallest positive number.)
Saying 4*12= 6 (mod k) means 4*12= 48= nk+ 6 for some integer n. That is the same as nk= 42.
Saying 8*8= 1 (mod k) means 8*8= 64= mk+ 1 for some integer m. That is the same as mk= 63.
42 factors as 2*3*7 and 63 factors as 3*3*7.
It is the fact that 7 is a common factor (so 42= n(7) with n=6) that leads to the conclusion that 4*12= 6 (mod 7) and (63= m(7) with m= 9) 8*8= 1 (mod 7).
Of course, 3 is also a common factor: 42= n(3) with n= 14 and 63= m(3) with m= 21. 4*12= 48= 42+ 6= (14)(3)= 6 so 4*12= 6 (mod 3) and 8*8= 64= 63+ 1= (21)(3)+ 1 so 8*8= 1 (mod 3). Of course that "= 6 (mod 3)" looks a little strange since, as I said, we normally choose a representative less than the base. Of course, 5*6= 30= 3*10 so 5*6= 0 (mod 3).
My preference would be (mod 3*7) or (mod 21).
Since 4*12= 48= 42+ 6= 2(21)+ 6, 4*12= 6 (mod 21).
Since 8*8= 64= 63+ 1= 3(21)+ 1, 8*8= 1 (mod 21).
That gives 5*6= 30= 21+ 9 so 5*6= 9 (mod 21).
I would support 9 as the best answer.
