To load an on-prem database into a cloud warehouse, read it in pages inside your network, check and mask each page there, keep the failed rows locally, and send the rest in batches of a few thousand rows per statement. In our test of 1 million rows, rows per insert set the speed. Our first checks caught 55% of the bad rows; after Claude profiled the data and rewrote the checks through the Neblex MCP server, they caught 94%, including in a half of the table they were never shown. The better checks cost about 2 minutes per million rows.
The request we hear often
Moving data from a database inside a company's network to a cloud warehouse, usually as a scheduled ETL job, is a request we hear often. The questions are always the same: how long will it take, will bad rows reach the warehouse, and does personal data leave the building?
Our measurement post covered what the runtime costs your server.…