Since you are assuming no friction, it follows that there can be no component of the collision force that is parallel to the wall. The force must all be in the perpendicular direction. You are correct that this means that any motion of the ball parallel to the wall can be ignored in computing work done.
Although kinetic energy is a scalar, that does not mean that we cannot account for the total kinetic energy as partly due to motion in the x direction, partly due to motion in the y direction and partly due to motion in the z direction. The pythagorean theorem makes this work out nicely. We know that the square of total velocity is the sum of the squares of the velocity components in the x, y and z directions. So total energy is given by one half of mass times the sum of the squared velocity components in the x, y and z direction.
However, that's largely irrelevant to the question you posed. Recall the definition of work as the [vector dot] product of force applied multiplied by distance moved. If the wall is not moving that means that no work is done on the wall.
If the wall is moving then, in the absence of friction, the work done can be computed as distance moved by the wall (in the normal direction) multiplied by force applied by the ball (which is neccessarily in the normal direction).
We normally treat a collision as if it were instantaneous. So the force applied is infinite and the duration is zero. That means that the product of force applied times distance moved is technically undefined. However, it turns out that one can still solve such problems. The product of force applied times duration of collision is equal to the "impulse" applied to the ball. If you know how much the momentum of the ball changed then that is how much impulse was involved.
impulse(ball) = m(ball) * delta-v(ball)
By Newton's third law, the impulse applied to the wall is equal and opposite.
impulse(wall) = - m(ball) * delta-v(ball)
Recall the definition of impulse as force multiplied by time.
force-on-wall * collision-duration = - m(ball) * delta-v(ball).
Multiply both sides by the velocity of the wall
force-on-wall * collision-duration * velocity(wall) = -m (ball) * delta-v(ball) * velocity(wall)
But time times velocity gives a distance
force-on-wall * distance-moved-by-wall = - m(ball) * delta-v(ball) * velocity(wall)
And that's what we're after
work-done-on-wall = - m(ball) * delta-v(ball) * velocity(wall)
[There is some lack of rigor in the above, but it can be worked out rigorously and still gives the same result]