database cloning with parallell netcat

Database cloning to several dev environments can be time consuming if running a normal restore (especially GemStone restores of some size to virtual dev servers). Time and effort can be reduced significantly if we were able to pump the full db directly to the new servers.

In GemStone's case, either first clone to an intermediate clone server and then stop the repository, or do a System suspendCheckpointsForMinutes: 5, start a snapshot on the extent filesystem and then do a System resumeCheckpoints.

Now ready to pump the database extents to the new environments, following the tricks from Tumblr's engineering team:

  • For each extent file, set up a netcat process on the last receiver listening on some port appending output to the extent file.
  • On each intermediate receiver, mkfifo a temporary local fifo, then start an nc sending to the next receiver reading and from the fifo, and another nc listening on some port, teeing output to the fifo and appending output to the extent file.
  • Finally, on your clone server or snapshot, cat each extent file to the first receiver's listen port. Add a pv -rb (pv along your pipes to watch transfer rates and progress.
  • Finally delete your snapshot and startup your cloned repositories.

VoilĂ !