it is all about representing a group of bits with a wave form. If you are using 256-QAM, it means you have 256 waveforms to represent any combination of 8 bits of data. 2 to the power of 8 is 256.
obviously in you assignment, it is a 3-bit modulation. 2^3=8, so you will have to define 8 different waveforms to represent a group of 3 bits data.
Each waveform will represent 000, 001, 010, 011, 100, 101, 110, 111 respectively.
Each waveform will be different from each other based on the 3 properties of waveform: Amplitude, Phase & Frequency.
If ASK, then the amplitude is different
If PSK, then the phase is different
If FSK, then the frequency is different
If ASK + PSK, then the combination of Amplitude and phase are different.
First, define the 8 sets of waveform and bit-group respectively
Then, If you data is given as 101110111, you can split it up to 101, 110 and 111.
lastly, use the defined waveform to represent each of them...