Hi! everyone, it's been a while not posting I'm looking for feedback from experienced Go developers on a concurrency and throughput problem in a distributed task-processing service.

The service processes upload tasks using Go goroutines and a semaphore to limit concurrent transfers. Each task uploads an object to S3-compatible storage, typically around 48–97 MiB.

Current implementation:

Semaphore-based concurrency limit: 16.

Each task acquires a permit before starting its upload.

The task's StartedAt timestamp is recorded before semaphore acquisition.

The task's CompletedAt timestamp is recorded after the upload and verification finish.

The semaphore permit is released after the transfer completes.

This means the reported task duration includes both semaphore queue waiting and actual transfer/verification time.

Observed production behavior:

We see two very different performance patterns:

Some 97 MiB uploads finish in approximately 3–8 seconds.

During certain bursts, similar-sized uploads take…