I don't see any problems with supporting multiple SRV records.
If we only permit one SRV record, the typical case takes one lookup,
the worst case is two. If we permit multiple SRV, hostile senders
can make lookups arbitrarily slow, e.g., what does your implementation
do with this?
If we permit multiple RRs, we'll have to add "too slow" heuristics
like SPF does. Blech.
What are the semantics of multiple SRVs? If they all have the same
numbers and they're all 1 2 0 or all 1 2 1, we probably agree that we
take the union of the addresses in the A records as the set of
authorized addresses. But what do all of the other permutations mean?