gl2gb/content/posts/can-i-haz-ur-dataz.md

title	categories	tags	date
Can I haz ur dataz	cyber brain fart		2019-02-01 09:55:07

Remote data acquisitions over ssh

To get your data trough ssh to your local storage, you simply use pipes. It does not matter if you use cat, dd or any other command line tool which outputs the data on standard output (stdout).

Using cat

When using cat the there is no need to add any additional parameters in your command chain. A simple cat <input> will suffice.

Cat does not have any progress information during read operations.

Using dd

The program dd requires more user interaction during the setup phase. To use dd you have to give the if= argument. The use of different blocksizes (bs) will not have that much of an impact on the speed. Feel free to have a look at this this.

Wow. So scientific! Much CLI.

Examples

A simple example with no output to the terminal except of errors. The output to stderr is not suppressed.

$ # Using cat to copy /dev/sda
$ ssh <user@remotehost> 'cat /dev/sda' > sda.raw

If you want to suppress errors, use:

$ # Using cat to copy /dev/sda
$ ssh <user@remotehost> 'cat /dev/sda 2>/dev/null' > sda.raw

The next example will demonstrate the use of dd.

$ # Using dd to copy /dev/sda
$ ssh <user@remotehost> 'dd if=/dev/sda' > sda.raw

Of course you can suppress errors as well.

Speeding up the data acquisition and minor tweaks

With the basics covered, we can begin optimizing our data transfer. In the first step we speed up the transfer with gzip.

The argument -c will write the compressed data to stdout which will be piped to your local system.

Of course you can use this with cat as well.

$ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' | gunzip > sda.raw

To add some progress information, you have two options with dd.

$ # dd status
$ ssh <user@remotehost> 'dd if=/dev/sda status=progress | gzip -c' \
> | gunzip > sda.raw

The status argument writes the output to stderr and will not end up in your local copy.

$ # dd through pv
$ ssh <user@remotehost> 'dd if=/dev/sda | gzip -c' \
> | pv | gunzip > sda.raw

Pv needs to be installed separatly. Check your packet manager. 0r teh Googlez!

KTHXBYE.

Update #01

Fixed a problem within the examples. Had a pipe too much. #Fail