Machine Callback Design

The Problem

Being able to remotely execute commands on stateless executors. The problem is that the controller does not know how to connect to the executor. Such a situation could be when a VM is started, and the hypervisor is unable to detect its IP information, or when the VM is behind a firewall, so the controller cannot access the executor directly.

To make things a bit more easier, let me define some things for the rest of the document:

executor:	The virtual machine that will execute the code
controller:	The machine that orchestrates the whole process

There is a 1:1 relationship between the controller instance and the executor instance

Step 1 - Create Configuration

On the controller, some files and settings need to exist:

• generate CONTROLLER_KEY
• allocate two ports on the system: CONTROLLER_PORT and CALLBACK_PORT. CONTROLLER_PORT has to be free on localhost, CALLBACK_PORT needs to be available on an externally accessible IP.

Step 2 - Create AFTER_INSTALL Script for the Specific Distribution

This script's responsibilities are:

• Configure the new system, so that eth0 will work with DHCP
• Configure a CALLBACK service on the new system, that will be executed on each boot, and will:
• Establish an SSH connection to the controller's IP address, with a specific CALLBACK_USER on a specific CALLBACK_PORT with a specific CALLBACK_KEY and forward it's ssh to a CONTROLLER_PORT
• Authenticate CONTROLLER_KEY for CONTROLLER_USER
• Shut down the machine

Step 3 - Create an Install Media

To install an operating system on a VM using some virtualisation platform. I think the Greatest Common Divisior of virtualisation platforms that I know so far is an ISO file. The project unattended-ubuntu-iso creates an ISO that will install Ubuntu, and shut down afterwards. It is also possible to inject a custom AFTER_INSTALL script. The installer should also create CONTROLLER_USER.

Step 4 - Install Operating System

A VM has to be created with a virtual hard drive, and a virtual cdrom - with the Install Media inserted. At this point the VM does not need network connection. It only uses the CDROM. The created VM has to be started, and have to wait until it's halted. The AFTER_INSTALL script will perform the specified steps, and shut the VM down.

Step 5 - Start a Controller

On the controller, an ssh server has to be started, allowing CALLBACK_USER to connect with CALLBACK_KEY.

Step 6 - Save the Image

The installer VM that was used to run the install process can be destroyed, only the hard disk needs to be kept - this is the image.

Step 7 - Start a new Instance

With the image created in the previous step, the executor could be created. This time it needs network connection as well. The virtualisation platform has to start the executor. As the AFTER_INSTALL script has already configured the networking, the executor will be configured by DHCP. As the executor started, it will start the CALLBACK script, which will establish a secure channel on CONTROLLER_PORT.

Step 8 - Execute commands

The controller can now access the executor's ssh server on the CONTROLLER_PORT. It will use CONTROLLER_USER and CONTROLLER_KEY to execute commands on the remote machine.