Depends much on how you decide to orient the reference system... anyway pay attention that initial velocities and final velocities are not necessarily equal, therefore you have better using different notations (like Va and Vb for initial velocities of the two objects and Ua and Ub for final ones)... indeed in your notation the second case you wrote would give you one no matter what happens...
As far as I understand, you want to have a number between 0 and 1 (0 being the case in which one object stops, 1 being elastic collision)... then you have to do the ratio of the final relative velocities and initial relative velocities... this order is simply used to get two positive numbers... you could also put absolute values and get the same result with any order...