#!/bin/sh
# Filename: /etc/dhcp/dhclient-exit-hooks.d/hostname
# Purpose: Used by dhclient-script to set the hostname of the system
# to match the DNS information for the host as provided by
# DHCP.
#
if [ "$reason" != BOUND ] && [ "$reason" != RENEW ] \
&& [ "$reason" != REBIND ] && [ "$reason" != REBOOT ]
then
return
fi
echo dhclient-exit-hooks.d/hostname: Dynamic IP address = $new_ip_address
hostname=$(host $new_ip_address | cut -d ' ' -f 5 | cut -d "." -f 1)
echo $hostname > /etc/hostname
hostname $hostname
echo dhclient-exit-hooks.d/hostname: Dynamic Hostname = $hostname
Friday 28 November 2014
Ubuntu - Update hostname from DHCP
This script is basically a mutation of the script found in drwindsor's post :
Git mirror without cloning
What I want to do is to set up a bare mirror git repository without fetching the changes. It should be a fast operation compared to:
git clone --mirror $REPOURLHere are the steps that I did to achieve the same results as a full clone:
# Create and update with the clone command git clone --mirror https://github.com/matelakat/vimide vimide-with-clone cd vimide-with-clone/ git fetch cd ..And let's create the equivalent:
git init --bare vimide-manual cd vimide-manual git remote add --mirror origin https://github.com/matelakat/vimide git fetch git pack-refs --all --prune cd ..You can see that the two directories are the same:
diff -r vimide-with-clone/ vimide-manual/ && echo "ALL the same"
ALL the same
Monday 4 August 2014
Differences between git branches
The problem
Given I have two branches, A and B within a git repository, I would like to come up with two sets of changes: (A-B) and (B-A). So I would like to find all the commits that are in A but not in B, and vice versa.Introducing the same change to two branches
Let's set up a git repository to play with. As a first step, I create an initial commit, with only one file in it, a simple greeting:$ git init $ echo "hello there" > hello.txt $ git add hello.txt $ git commit -m "Initial commit"And now, make the branching, and introduce the same change to each of those branches:
$ git checkout -b A $ echo "blah" > blah.txt $ git add blah.txt $ git commit -m "Added blah"And see the B branch:
$ git checkout master -b B $ echo "blah" > blah.txt $ git add blah.txt $ git commit -m "Added blah"See if git is smart enough to find out that the two changes are the same:
$ git log A..B --oneline | wc -l
1Of course, they are not the same, they are copies of the other. So let's see the other command, called git-cherry:
$ git cherry A BThis shows only one line, with a minus sign. From the documentation of git cherry:
git cherry [-v] [<upstream> [<head> [<limit>]]]
Every commit that doesn’t exist in the <upstream> branch has its id (sha1) reported,
prefixed by a symbol. The ones that have equivalent change already in the <upstream>
branch are prefixed with a minus (-) sign, and those that only exist in the <head>
branch are prefixed with a plus (+) symbol:
Saturday 21 June 2014
SSD - Bytes written
I Have an Intel 520 SSD. I can use smartmontools to ask how much bytes have been written to it:
sudo ./smartctl -q noserial -i -l devstat,0 -l devstat /dev/sda
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.2.0-32-generic] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Intel 520 Series SSDs Device Model: INTEL SSDSC2CW240A3 Firmware Version: 400i User Capacity: 240,057,409,536 bytes [240 GB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 T13/2015-D revision 3 SATA Version is: SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Jun 21 08:42:06 2014 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled Device Statistics (GP Log 0x04) supported pages Page Description 0 List of supported log pages 1 General Statistics 4 General Errors Statistics 6 Transport Statistics 7 Solid State Device Statistics Device Statistics (GP Log 0x04) Page Offset Size Value Description 1 ===== = = == General Statistics (rev 2) == 1 0x008 4 1129 Lifetime Power-On Resets 1 0x010 4 8660 Power-on Hours 1 0x018 6 5401949881 Logical Sectors Written 1 0x028 6 5794864033 Logical Sectors Read 4 ===== = = == General Errors Statistics (rev 1) == 4 0x008 4 0 Number of Reported Uncorrectable Errors 4 0x010 4 9232 Resets Between Cmd Acceptance and Completion 6 ===== = = == Transport Statistics (rev 1) == 6 0x008 4 9232 Number of Hardware Resets 6 0x010 4 4085 Number of ASR Events 6 0x018 4 0 Number of Interface CRC Errors 7 ===== = = == Solid State Device Statistics (rev 1) == 7 0x008 1 255 Percentage Used Endurance Indicator
Monday 16 June 2014
Recording terminal sessions
During the weekend I was experimenting with some tools to record terminal sessions. The result of the hacking:
- ttyrec
- ttygif
Tuesday 27 May 2014
Debootstrap as a single user
fakeroot -s fakechroot.save fakechroot debootstrap --variant=fakechroot --components=main,universe precise precise fakeroot -i fakechroot.save -s fakechroot.save fakechroot chroot precise apt-get update fakeroot -i fakechroot.save -s fakechroot.save fakechroot chroot precise apt-get -qy install git
Sunday 18 May 2014
Ubuntu installer - Unmount partitions that are in use?
I am working on the unattended-iso project, and seeing this message:
for the bots and search engines
The installer has detected that the following disks have mounted partitions: /dev/sda Do you want the installer to try to unmount the partitions on these disks before continuing? If you leave them mounted, you will not be able to create, delete, or resize partitions on these disks, but you may be able to install to existing partitions there. Unmount partitions that are in use?
It happens whenever you already have a filesystem on say sda1. Workaround:
d-i preseed/early_command string umount /media || true
Friday 16 May 2014
Sherlocking Mock based builds
Jenkins is frozen, waiting for something. One useful link is to look at
${JENKINS_URL}/threadDump. That reveals which thread is doing what.
Mine is stuck like this:
waiting for hudson.remoting.Channel@f7f725
Okay, I logged onto the other machine, and yes, it still has an open ssh connection to the guest. So log on to the guest, and look at the open files:
sudo lsof -i 4
Okay, I now know some PIDs: 775 and 790
Let's see the file descriptors:
sudo ls -l /proc/790/fd/
lrwx------ 1 root root 64 May 16 19:51 0 -> /dev/null
lrwx------ 1 root root 64 May 16 19:51 1 -> /dev/null
lr-x------ 1 root root 64 May 16 19:51 10 -> pipe:[7800]
lrwx------ 1 root root 64 May 16 19:51 2 -> /dev/null
lrwx------ 1 root root 64 May 16 19:51 3 -> socket:[7744]
lrwx------ 1 root root 64 May 16 19:51 4 -> socket:[7781]
lrwx------ 1 root root 64 May 16 19:51 5 -> socket:[7791]
lr-x------ 1 root root 64 May 16 19:51 6 -> pipe:[7798]
l-wx------ 1 root root 64 May 16 19:51 7 -> pipe:[7798]
Nice, I guess that pipe is the bad guy, but how do I know, what is that?
sudo lsof | grep 7800
sshd 790 ubuntu 10r FIFO 0,8 0t0 7800 pipe
udevd 30043 root 4w FIFO 0,8 0t0 7800 pipe
All right, that means that I know a new PID...
sudo ls -l /proc/30043/fd/
Okay, that's nice, but what was the command line?
sudo cat /proc/30043/cmdline
udevd --daemon
Nice, now I need to look at mock and udevd.
Sunday 11 May 2014
Handwriting to SVG: potrace and image magick
#!/bin/bash
set -eux
INPUT="$1"
OUTPUT="$2"
WORKDIR="$(mktemp -d)"
convert \
-format bmp -colorspace gray -normalize \
-threshold 90% "$INPUT" "$WORKDIR/bitmap.bmp"
potrace -b svg -r 300 -t 5 -o "$OUTPUT" "$WORKDIR/bitmap.bmp"
Thursday 8 May 2014
Machine Callback Design
The Problem
Being able to remotely execute commands on stateless executors. The problem is that the controller does not know how to connect to the executor. Such a situation could be when a VM is started, and the hypervisor is unable to detect its IP information, or when the VM is behind a firewall, so the controller cannot access the executor directly.
To make things a bit more easier, let me define some things for the rest of the document:
| executor: | The virtual machine that will execute the code |
| controller: | The machine that orchestrates the whole process |
There is a 1:1 relationship between the controller instance and the executor instance
Step 1 - Create Configuration
On the controller, some files and settings need to exist:
- generate CONTROLLER_KEY
- allocate two ports on the system: CONTROLLER_PORT and CALLBACK_PORT. CONTROLLER_PORT has to be free on localhost, CALLBACK_PORT needs to be available on an externally accessible IP.
Step 2 - Create AFTER_INSTALL Script for the Specific Distribution
This script's responsibilities are:
- Configure the new system, so that eth0 will work with DHCP
- Configure a CALLBACK service on the new system, that will be executed on each boot, and will:
- Establish an SSH connection to the controller's IP address, with a specific CALLBACK_USER on a specific CALLBACK_PORT with a specific CALLBACK_KEY and forward it's ssh to a CONTROLLER_PORT
- Authenticate CONTROLLER_KEY for CONTROLLER_USER
- Shut down the machine
Step 3 - Create an Install Media
To install an operating system on a VM using some virtualisation platform. I think the Greatest Common Divisior of virtualisation platforms that I know so far is an ISO file. The project unattended-ubuntu-iso creates an ISO that will install Ubuntu, and shut down afterwards. It is also possible to inject a custom AFTER_INSTALL script. The installer should also create CONTROLLER_USER.
Step 4 - Install Operating System
A VM has to be created with a virtual hard drive, and a virtual cdrom - with the Install Media inserted. At this point the VM does not need network connection. It only uses the CDROM. The created VM has to be started, and have to wait until it's halted. The AFTER_INSTALL script will perform the specified steps, and shut the VM down.
Step 5 - Start a Controller
On the controller, an ssh server has to be started, allowing CALLBACK_USER to connect with CALLBACK_KEY.
Step 6 - Save the Image
The installer VM that was used to run the install process can be destroyed, only the hard disk needs to be kept - this is the image.
Step 7 - Start a new Instance
With the image created in the previous step, the executor could be created. This time it needs network connection as well. The virtualisation platform has to start the executor. As the AFTER_INSTALL script has already configured the networking, the executor will be configured by DHCP. As the executor started, it will start the CALLBACK script, which will establish a secure channel on CONTROLLER_PORT.
Step 8 - Execute commands
The controller can now access the executor's ssh server on the CONTROLLER_PORT. It will use CONTROLLER_USER and CONTROLLER_KEY to execute commands on the remote machine.
Tuesday 18 March 2014
Git review
Just a quick note. If you are reviewing the changes in a git repository, the following command might be useful:
git log -p --colorThis gives you a nice, colored diff.
Sunday 2 March 2014
Weekend
This weekend started with friday. I took a day off to relax my mind. This relaxation ended up sitting in front of my laptop, experimenting wit vsphere's opensource java api, while trying to build an embedded vm that can run java - just with buildroot. On saturday, I managed to find out, what osgi is, and I think it will be a useful ground for running java bundles on my emvms (embedded vms). The same day I learned, that a proper libc is needed, so I experimented with the linux kernel - crosstool-ng - buildroot trinity. On sunday we went to cinema to watch the rather old movie: gravity, re done and posted yesterday's build efforts with some extra deployment script, and closed the day with some control theory, learning about zenos, and how to deal with type 1 zenos. Good night.
First post from phone
This is my first post created with my phone. The picture shows a thermapen, we used it during the weekend.
VM From Scratch
I have spent a lot of time installing various Linux distributions, automating all the complex things, trying out things like puppet, writing my own management scripts. Well, in the end of the day, the goal was to run a very simple application. Usually these applications run inside a hypervisor, and we don't really want to spend time with the operating system's management. All I want is a Virtual Machine that has my application and I don't want to know more about it. I think I'm even not interested in sshing into that box. I want to build VMs quickly.
There are several tools to build a VM, and I believe all do their job perfectly. But I think it's time for me to try to build a VM from scratch. The components, that I will use are:
- linux
- crosstool-ng
- buildroot
Getting the kernel
At the time of writing the linux version that I picked, is 3.10.32 - I believe, it has a long term support. So let's dive in, and get the source code
cd /data/matelakat/kernel wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.10.32.tar.xz tar -xJf linux-3.10.32.tar.xz
Now that we have the sources, we can create a cross compiler.
crosstool-ng
This tool will help me to build a cross compiler. At the initial phase, my target architecture is 686 32bit. I picked this one, as I would like to test the VM with nested virtualization (ESXi inside VirtualBox), and you can't run 64 bit guests there. So let's get started:
first, download the tools, and check out a specific revision. This makes it easier for anyone to reproduce.
mkdir /data/matelakat/xtool cd /data/matelakat/xtool hg clone http://crosstool-ng.org/hg/crosstool-ng cd crosstool-ng/ hg checkout 9321d9d7af9b
Now comes the crosstool-ng specific part. First, we need to actually install crosstool-ng:
./bootstrap ./configure --prefix=$(cd .. && pwd)/ctng make make install export PATH=$PATH:$(cd .. && pwd)/ctng/bin
The next step is to generate a configuration for the crosstool. I will start by getting a sample configuration file, and overwriting it with my config:
cd /data/matelakat/xtool/ mkdir configuration cd configuration/ ct-ng x86_64-unknown-linux-gnu cp /data/matelakat/vmfromscratch/crosstool-ng/.config .config ct-ng menuconfig
If you are happy with the config, it's time to build the cross compiler. My config will put the downloaded sources and the built tools to ../src and ../x-tools, so I need to create those directories in advance. Also, as I have a laptop with 4 cores, I will use 6 jobs simultaneously:
mkdir ../src ../x-tools time ct-ng build.6
I can definitely tell: my CPU cores are busy - some hot air flows out of my good old Lenovo R500. After around an hour, the crosscompiler was ready:
du -shc ../x-tools/
364M../x-tools/ 364Mtotal
Build the Kernel
Now that we have a crosscompiler, we can get back to the kernel directory, configure the tool paths, and compile a new kernel:
cd /data/matelakat/kernel/linux-3.10.32/ cp /data/matelakat/vmfromscratch/linux/.config ./ make oldconfig make menuconfig make -j 4
Unfortunately, I didn't measure the time it took, but it was quite fast
Build the Root Filesystem
Now that I have a nice kernel I also need a root filesystem. Buildroot will help me to build that:
mkdir /data/matelakat/br cd /data/matelakat/br wget -qO - http://buildroot.uclibc.org/downloads/buildroot-2014.02.tar.gz | tar -xzf - cd buildroot-2014.02/ cp /data/matelakat/vmfromscratch/buildroot/config .config make menuconfig make
Putting it all together
Now that I have everything, I only need to create an iso cd with my new system
mkdir isoroot cp ../br/buildroot-2014.02/output/images/rootfs.cpio.gz isoroot/initrd.img cp ../kernel/linux-3.10.32/arch/i386/boot/bzImage isoroot/ cp ../br/buildroot-2014.02/output/images/isolinux.bin isoroot/isolinux/ cp isolinux.cfg isoroot/isolinux/ mkisofs -J -o minsys.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table isoroot/ qemu-system-i386 -M pc -cdrom minsys.iso -net nic,model=e1000 -net user
Next steps
My next step will be to automate all these things.
Sunday 9 February 2014
PDF - Two pages per page to one pages per page
So we had a PDF, it's size was 297 x 215 mm. Each page of this PDF contained two pages, like this:
http://superuser.com/questions/54054/convert-pdf-2-sides-per-page-to-1-side-per-page
Page 1 of the input file Page 1 of output Page 2 of output
+------------------------------+ +---------------+ +---------------+
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| Page 1 | Page 2 | | Page 1 | | Page 2 |
| of the book | of the book | | of the book| | of the book|
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
+------------------------------+ +---------------+ +---------------+
+------->
Page 2 of the input file +-------> Page 3 of output Page 4 of output
+------------------------------+ +---------------+ +---------------+
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| Page 3 | Page 4 | | Page 3 | | Page 4 |
| of the book | of the book | | of the book| | of the book|
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
+------------------------------+ +---------------+ +---------------+
So to do this, I extracted the left sides as page-0.pdf, and the right sides as page-1.pdf to the folder out, and the n joined these together:
#!/bin/bash
set -eux
# Default resolution is 720 dpi
# For points, use 72 dpi
# The actual pdf was 297 x 215 mm --(720 dpi)----> 8418 x 6094 pixels
# I only want to extract half of it -------------> 4209 x 6094
# For the right page, I need an offset -(72 dpi)-> -421.0 0
INPUT_FILE="$1"
rm -rf ./out
mkdir out
ghostscript -o "out/%d-0.pdf" -sDEVICE=pdfwrite -g4209x6094 \
-c "<</PageOffset [0 0]>> setpagedevice" -f "$INPUT_FILE"
ghostscript -o "out/%d-1.pdf" -sDEVICE=pdfwrite -g4209x6094 \
-c "<</PageOffset [-421.0 0]>> setpagedevice" -f "$INPUT_FILE"
ghostscript -o "singlepage-$INPUT_FILE" -sDEVICE=pdfwrite -dPDFSETTINGS=/ebook \
$(ls out/* | sort -V)
rm -rf ./out
References:http://superuser.com/questions/54054/convert-pdf-2-sides-per-page-to-1-side-per-page
Sunday 2 February 2014
Back to school - Coursera - Crypto, Control Theory, Android
After KR told me that he signed up for a Cryptography course on Coursera, I decided to do the same. I learned crypto at Uni, but that was a long time ago, so I thought, it's time to refresh my memories. The story does not end here, as this weekend, I signed up to two more courses: an Android course, and a Robot Control course. I'm a bit behind at the moment for each of the courses. I find this Coursera amazing, and I hope my wife won't kill me before I finish these courses. Should you be interested, here is a list of the courses that I started:
- https://class.coursera.org/crypto-009
- https://class.coursera.org/conrob-002
- https://class.coursera.org/android-001
I just started to watch the video lectures for the Android class, and watching this video atm as recommended by the professor:
Subscribe to:
Posts (Atom)