I am building a cloud trade copier for MetaTrader. Each follower account has one broker session on one node, and that node copies the master's trades onto it. In an earlier post I described how a node leases its accounts from Postgres: a node_id and a lease_until on the account row, FOR UPDATE SKIP LOCKED to claim, and a node that stops sending orders 15 seconds before its lease ends.
That post was about keeping ownership. This one is about losing it: what happens, second by second, when a node is cut off from the database, another node takes its accounts, and both have to end up agreeing with what is actually open at the broker.
The defaults
default
lease 60 s
renewal cycle every 20 s
fence (stop sending before the lease ends) 15 s
retry after a failed cycle 2 s
These are configuration values, not measurements.
The timeline
Say node A last renewed its leases at t = 0. Then its connection to the database breaks.…