Multiple rules at the same level in a custom approval plan are evaluated in alphabetical order. The order the rules appear in when looking at the custom approval plan rules in the interface does not matter, it is the name of the rules that matter.
For example, maybe you have three rules in your approval plan, all at Level 1, like this:
C First Rule
A Second Rule
B Third Rule
The incoming order for approval will first be evaluated if it matches to 'A Second Rule', then it will be evaluated against 'B Third Rule', and then it will be evaluated against 'C First Rule', independently of the ordering of the rules in the window.
Just to be clear, the names of the rules as sorted alphabetically determine in what order they will be evaluated when an incoming order is using a custom approval plan. The approval for the order will be triggered by the first rule that the order matches to at any given level of the rules in the approval plan.