This post was written for my own reference, so I can stop recreating the process each of the 2-3 times a year I need to use it

Every now and then, you have a large number of files to transfer across a really fast network, and the usual methods just have too much overhead.

At times like this, Ye Olde Skool Neckbeard Sysadmin reverts to a time-honoured technique known as the Netcat Tar Pipe:


On the receiving end do:
# netcat -l -p 7000 | tar x

And on the sending end do:
# tar cf - * | netcat otherhost 7000

The chiefest drawback of this technique is that you don't know what's happening. You know that Some Data is being transferred at A Rate, but that's about all.

Enter Pipe Viewer, aka pv:


pv - Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.

A minor tweak or two will give you fancypants progress indicators, and a file transfer mechanism that's almost certainly faster than anything else you can do.

First you need to calculate the size of the data you're sending:


[/share/polleyj] # du -sk
594409480 .

Now we use a simple modification of the command above on the receiving side:


netcat -l -p 7000 | pv -s 594409480k | tar vx

and on the sending side:


tar cf - * | pv -s 594409480k | netcat otherhost 7000

You now get a fancy progress indicator on the sending side:


959MB 0:03:18 [4.64MB/s] [> ] 0% ETA 33:13:22

and with the addition of the 'v flag on the receiving side, you can see filenames as they're unpacked as well:

Bodacious/Downloads/._.DS_Store
Bodacious/Downloads/VERY_LARGE_FILE.ISO
352MB 0:02:41 [4.85MB/s] [> ] 0% ETA 73:41:36

So there you go. May as well go make a coffee or 73 while you wait for that data to copy over.