I think the same force acts on you both ways, that is, -F. And therefore, your acceleration will be the same and so will the final velocity (assuming that the contact time is the same).
This is consistent with the law of conservation of linear momentum which states that:
The total momentum...