TL;DR: This evening I logged into the server running one of my products and found malware that had been there for months. Two services were reachable from the internet with no authentication: Redis (port 6380, no password) and OnlyOffice Document Server (port 8083, no JWT) . The Redis hole was enough for an automated worm to install a Monero miner that also scans for other Redis servers. Below: what I found, what I could not determine, how to clean up properly, and how to make sure it is not you next.
Update: "I rebuilt the server from a clean image and rotated all secrets."
How I found it
I was checking on the server for one of my products when something looked wrong. [ADD: the first thing you noticed, for example high CPU, an odd process, or the OnlyOffice failure.] When I restarted the container, the miner ran again.
That detail matters. If a restart brings the malware back, it is not just a process you can kill.…