You start with the greater number when adding on a number line simply for convenience.
Put it this way:
Would you rather add more to a small number or add less to a large number?
Naturally people would choose to add less to a large number because that way you do not have to do so much work. So, you would start with the greater number when adding on a number line.
