Dd
dd - convert and copy a file.
REALLY GOOD ARTICLE: https://www.computernetworkingnotes.com/linux-tutorials/generate-or-create-a-large-text-file-in-linux.html
Contents
How to use dd
This command will create a file of size count*bs, in which the below will create a file of 1 MB:
dd if=/dev/zero of=file.txt count=1024 bs=1024
- /dev/zero is a special file in Unix-like operating systems that provides as many null characters as are read from it.
- if=FILE -- read from FILE instead of stdin
- of=FILE -- write to FILE instead of stdout
- count=N -- copy only N input blocks
- bs=BYTES -- read and write up to BYTES bytes at a time
Generate FIles
This command will generate 1 GB file (where 1048576 bytes = 1MB):
dd if=/dev/zero of=file.txt count=1024 bs=1048576
When You Don’t Care About The Contents But Want Some Lines, as in, you don't want it to just be full of nulls, you can use /dev/urandom. This is a partner of /dev/random, which serves as a random number generator. Essentially /dev/random will eventually block unless your system has a lot of activity, /dev/urandom in non-blocking. We don’t want blocking when we’re creating our files so we use /dev/urandom. The only real difference is that /dev/urandom is actually less random but for our purposes it is random enough.
This will create a file with bs*count random bytes, in our case 2048*10 = 20KB (20480 Bytes).
dd if=/dev/urandom of=file.txt bs=2048 count=10
To generate a 100Mb file we would do using /dev/urandom:
dd if=/dev/urandom of=file.txt bs=1048576 count=100
How to duplicate data from 'source' to 'destination' volume
(From k u m o 2666 Linux - Create HVM AMI from PV source instance) 2. Check the 'source' volume and minimize the size of original filesystem to speed up the process. We do not want to copy free disk space in the next step.
e2fsck -f /dev/xvdm resize2fs -M /dev/xvdm # Output from resize command: Resizing the filesystem on /dev/xvdm to 269020 (4k) blocks. The filesystem on /dev/xvdm is now 269020 blocks long.
3. Duplicate 'source' to 'destination' volume.
dd if=/dev/xvdm of=/dev/xvdo1 bs=4K count=269020
4. Resize the filesystem on the 'destination' volume after the transfer completed.
resize2fs /dev/xvdo1
How to initialize (or pre-warm) an EBS volume restored from a snapshot
Use the dd or fio utilities to read all of the blocks on the device. The dd command is installed by default on Linux systems, but fio is considerably faster because it allows multi-threaded reads.
- NOTE: This step may take several minutes up to several hours, depending on your EC2 instance bandwidth, the IOPS provisioned for the volume, and the size of the volume.
- Run lsblk to check the block device you want:
$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 10G 0 disk ├─nvme0n1p1 259:1 0 10G 0 part / └─nvme0n1p128 259:2 0 1M 0 part
The if (input file) parameter should be set to the drive you wish to initialize. The of (output file) parameter should be set to the Linux null virtual device, /dev/null. The bs parameter sets the block size of the read operation; for optimal performance, this should be set to 1 MB.
Incorrect use of dd can easily destroy a volume's data. Be sure to follow precisely the example command below. Only the if=/dev/xvdf parameter will vary depending on the name of the device you are reading.
$ sudo dd if=/dev/xvdf of=/dev/null bs=1M
When the operation is finished, you will see a report of the read operation. Your volume is now ready for use.
Other methods besides DD
DD is not parallel, so there are other methods to create files much quicker than DD. Below is a process on how you can do so:
1) Create ten thousand small files in your local storage on your instance.
$ mkdir /tmp/efs; for each in $(seq 1 10000); do SUFFIX=$(mktemp -u EFS-XXXXXX); sudo dd if=/dev/zero of=/tmp/efs/${SUFFIX} bs=64k count=1; done
2) To test copy the files from your instance to the EFS file system:
Serial method:
$ cd /tmp/efs; time sudo find -L . -maxdepth 1 -type f -exec rsync -aR '{}' /efs/ \;
Parallel method:
$ cd /tmp/efs; time find -L . -maxdepth 1 -type f | sudo parallel rsync -aR {} /efs/
3) To remove:
Serial method:
$ cd /mnt/efs; time sudo find -L . -maxdepth 1 -exec rm -rfv {} \;
Parallel method:
$ cd /mnt/efs; find -L . -maxdepth 1 -type f | sudo parallel rm -rfv {}
References:
[+] How To Quickly Generate A Large File On The Command Line (With Linux)
[+] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html