Menu

1 / 3

Load an on-prem database into a cloud warehouse: 1 million rows, measured

DEV Community: postgres·Adan Potyondy·5 days ago
#vM2WG8PU
#devtopostgresql#devto#dev#rows#test
Reading 0:00
15s threshold

To load an on-prem database into a cloud warehouse, read it in pages inside your network, check and mask each page there, keep the failed rows locally, and send the rest in batches of a few thousand rows per statement. In our test of 1 million rows, rows per insert set the speed. Our first checks caught 55% of the bad rows; after Claude profiled the data and rewrote the checks through the Neblex MCP server, they caught 94%, including in a half of the table they were never shown. The better checks cost about 2 minutes per million rows.

The request we hear often

Moving data from a database inside a company's network to a cloud warehouse, usually as a scheduled ETL job, is a request we hear often. The questions are always the same: how long will it take, will bad rows reach the warehouse, and does personal data leave the building?

Our measurement post covered what the runtime costs your server.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.