Backup Device - power save

I would like to have a backup device, to safely save down my data. I also want this solution to consume small amounts of power. Thereby, I will build it with a Raspberry PI. The problem, that I am facing, is that I don't want the hard drives to work 24/7. In order to achieve this, I need a way to power them down from my Raspberry. Options that I looked at:

• Some active circuit to power down the USB ports
• A relay based solution
• Optical controll. Dusk till Dawn adaptor in Hungary, it is called "alkonykapcsoló". Conrad sells almost the same item

This could be useful: raspberry leaf

DHCP issues with Arch on Raspberry PI

I had some issues with my Raspberry. It did not always come up after a power on. As I don't have a monitor attached to this small fruit, I was completely locked out. The applied solution was to switch to dhclient. It could be achieved by installing dhclient, and editing a profile file for that interface:

cat /etc/network.d/interfaces/eth0

DHCLIENT=yes

The small fruit seems to be happy now, and so do I.

Starting the Raspberry PI

I arrived to a point, to start my RaspberryPi, that I got for Christmas from my beautiful fiancee. Let's see: Download the image from the Arch Linux / Raspberry PI page:

wget -q http://raspberry.mythic-beasts.com/raspb
erry/images/archlinuxarm/archlinux-hf-2013-02-11/archlinux-hf-2013-02-11.zip
unzip archlinux-hf-2013-02-11.zip 

sudo dd bs=1M if=archlinux-hf-2013-02-11.img of=/dev/mmcblk0 oflag=dsync

Please note the usage of the dsync flag. Without that, I have some nasty dmesg logs, complaining:

INFO: task blkid:3812 blocked for more than 120 seconds.
...

If you want to know where dd is, send him an USR1 signal:

sudo kill -USR1 pid_of_dd

And now, plug it to the Raspberry, and see what happens! Of course, I did not plug in a keyboard, nor a screen, as I don't have such devices at home. I expect the Arch to come up nicely with an ssh server. Wow, in the DHCP, I suddenly noticed a new device, alarmpi. Okay, let's ssh in to that! The user/password is root/root Update the system:

pacman -Syu

It downloaded a new kernel as well, which is a bit scary for the first sight. I'll reboot, and see if the kernel update happened or not. The update process was really slow. I guess it is something to be done offline / in an emulator, not on the real device. The actual kernel:

Linux alarmpi 3.6.11-6-ARCH+ #1 PREEMPT Mon Feb 11 02:33:03 UTC 2013 armv6l GNU/Linux

Okay, the update finished, so I rebooted the small fruit. After that, I looked at the kernel version:

Linux alarmpi 3.6.11-8-ARCH+ #1 PREEMPT Sat Mar 9 00:38:58 UTC 2013 armv6l GNU/Linux

So it managed to successfully upgrade the kernel. Nice! Now, let's use the raspberry to check a disk for errors:

badblocks -b 1048576 -n -o badblocks.sam120 -s -v /dev/sda

Further reading for me, it might be interesting how to emulate an arm device, so that the initial update could happen on an emulator

• Interesting Post
• Look at this

Discovering rsync - TDD approach

rsyncdiscovery

A test script to find out rsync parameters for backup.

Conclusion

To copy src to tgt:

rsync -r -H -l -g -o -t -D -p --del src tgt

And you will end up with

tgt/src

being a mirror of the original src directory.

Test usage

The shell script's source and this document lives here

To discover the tests:

./test_rsync.sh list_tests

To run the tests:

sudo ./test_rsync.sh

Background

I would like to create a file-level backup of some files. My goal is to use rsync to do the job. As rsync has a lot of command line switches, first, I would need to know, which ones to use. For this, I have some expectations:

1. Backup is recursive
2. Extra files removed
3. Links are preserved
4. Symlinks are preserved
5. Symlinks are not rewritten
6. Preserve permissions
7. Modification times are preserved
8. Special files reserved

And of course, a test-driven approach will be used. Bash does not prevent you from writing tests, so go ahead.

Ending slash: The ending slash is important on the source side, if you wish to say to copy the contents of that directory, not the directory itself.

The archive mode includes:

• r - recursive
• l - copy symlinks as symlinks
• p - preserve permissions
• t - preserve modification times
• g - preserve group
• o - preserve owner
• D - preserve device and special files

And the manual also states, that it does not include:

• H - preserve hard links
• A - preserve ACLs
• X - preserve extended attributes

For me the hardlinks seem to be important.

Make an md array writable

I usually set my arrays to readoly, after assembling them. This way I could avoid unattended rebuilds. But sometimes, you would want to update some data on the disk. For doing this, let's see what we have:

root@plugged:~# mdadm --detail /dev/md1 | grep Update 

    Update Time : Sun Dec  2 12:58:58 2012

And what is the short info?

root@plugged:~# cat /proc/mdstat 

Personalities : [raid1] 
md1 : active (read-only) raid1 sdc2[0] sdd2[1]
      83884984 blocks super 1.2 [2/2] [UU]

Now, let's put it into r/w mode:

mdadm --readwrite /dev/md1

Let's see what happened:

cat /proc/mdstat

md1 : active raid1 sdc2[0] sdd2[1]
      83884984 blocks super 1.2 [2/2] [UU]

Did the superblock date changed?

root@plugged:~# mdadm --detail /dev/md1 | grep Update

    Update Time : Sun Mar 10 18:23:18 2013

So, it changed as soon, as I set it to readwrite mode.

Meet with buildroot - Custom Linux

I was thinking about having a really small linux system for cloud images. I already knew about the concept of JeOS (Just enough OS), but I still can't agree having more than 100MB as a disk. Actually, I would like to be able to have something around 50 MB. I knew about CirrOS, which we use for functional testing OpenStack. CirrOS is small, the qcow image is 10 Megs. So I looked at the launchpad site, and discovered, that cirros is using buildroot to create the root filesystem. That was enough inspiration for me to try to create a very minimal Linux. First build

$ mkdir myl
$ cd myl
$ wget -qO - http://buildroot.uclibc.org/downloads/buildroot-2013.02.tar.gz | tar -xzf -
$ cd buildroot*
$ make qemu_x86_defconfig
$ time make

Before starting make, I looked into the configuration, and it turned out, that qemu_x86_defconfig includes building a kernel, so I expect the build to last at least for an hour. In the meanwhile, it makes sense to read how I will start the machine:

$ cat board/qemu/x86/readme.txt

In the end, it was quicker than I expected:

real 26m5.626s
user 32m15.753s
sys  3m55.447s

And, let's see how it boots!

qemu-system-i386 -M pc \
-kernel output/images/bzImage \
-drive file=output/images/rootfs.ext2,if=ide \
-append root=/dev/sda

Wow! that was fast. To get a feeling on how fast it is, see this video And you might be interested in the size of the whole stuff:

$ du -shc output/images/

3.2M output/images/
3.2M total

TDD framework with inotifywait and tox

So let's say, you want to do TDD. It means, that whenever you change something, you would want to run the tests. Okay, it sounds like a boring task, so let's automate it, to meet the Agile practices. So a user-space utility, called

inotifywait

will be used. This blocks, until some even happens with the monitored files. As my source code is in git, I will use:

git ls-files

to get the files to be watched. Okay, let's put it together:

$ git ls-files | inotifywait -q --fromfile -

This blocks, until something happens to a file that is watched. Let's save a file with vim, that is under version control, and see that inotifywait unblocks, and outputs:

fs/__init__.py MOVE_SELF 

Okay, that works, so let's use it as a framework to run our tests:

$ while true; do git ls-files | inotifywait -q --fromfile -; tox; done

As you add a new file to git, that will result in a change in .gitignore, so it should be automatically picked up by our framework. Tox gives you pretty colours as well!