I need multiple rules at the same level in a custom approval plan, but what determines the order in which these rules are evaluated for orders?

Multiple rules at the same level in a custom approval plan are evaluated in alphabetical order. The order the rules appear in when looking at the custom approval plan rules in the interface does not matter, it is the name of the rules that matter.
 
For example, maybe you have three rules in your approval plan, all at Level 1, like this:
 
C First Rule
A Second Rule
B Third Rule
 
 
The incoming order for approval will first be evaluated if it matches to 'A Second Rule', then it will be evaluated against 'B Third Rule', and then it will be evaluated against 'C First Rule', independently of the ordering of the rules in the window.
 
Just to be clear, the names of the rules as sorted alphabetically determine in what order they will be evaluated when an incoming order is using a custom approval plan. The approval for the order will be triggered by the first rule that the order matches to at any given level of the rules in the approval plan.
 
For more information on custom approval plans, please see article Ch. 062 Approval Plans (B2B Sites Only)