"Doubtful cases go to a human" is easy to say and easy to get wrong. If the review queue is slow, confusing or endless, people stop trusting it, start rubber-stamping, and the accuracy you measured in the pilot quietly disappears. Here is how I would design the queue so it holds up.
What goes into the queue
Two kinds of items belong there:
Low confidence. Anything below the threshold you agreed on. In my ticket-triage demo (public data, test numbers) about 24% of tickets fall below the line, and that share is a design input, not a surprise.
Hard rules. Items that must always be checked regardless of confidence: amounts above a limit, a failed checksum, a customer flagged as sensitive.
Keep the two reasons separate in the record.…