How to create OS template in Ganeti?

Submitted by admin on Sun, 01/25/2015 - 10:02
ganeti
Debian 7.8 is out, and I wanted to create a new OS template in my Ganeti cluster, based on this Debian version.

I did it in these 5 steps:

  • download the ISO image
  • setup OS templates
  • install and config debian 7.8
  • create OS filesystem dump for instances
  • create new debian 7.8 based instances

Download the ISO image

On the Ganeti cluster I store the ISO templates at /var/local/iso. So navigate there, and download the last stable ISO image of Debian:

cd /var/local/iso
wget http://cdimage.debian.org/debian-cd/7.8.0/amd64/iso-cd/debian-7.8.0-amd64-CD-1.iso

Do not forget to deploy the ISO file to all the nodes!

gnt-cluster copyfile /var/local/iso/debian-7.8.0-amd64-CD-1.iso

Setup OS templates

A new variant has to be added, with its network setup files, and I'll check some other setup as well. Finally, all the changes has to be deployed to all the nodes of the cluster.

1. Add a new variant

We can configure Ganeti various instance templates, called variants. Settings can be customized via global settings file, as well as by particular variant settings files.

All the variants has to be listed in the /etc/ganeti/instance-image/variants.list file. See what is already there:

cat /etc/ganeti/instance-image/variants.list

Now this will be a new instance OS template, a Debian server with SSH. Add it to variants list:

echo debian-7.8 >> /etc/ganeti/instance-image/variants.list

The new Debian 7.8 variant will use dumps for instance creation, therefore image type is set to "dump". The rest should be self explanatory. One could also add install packages and other overlays, but it’s not covered here.

cat >> /etc/ganeti/instance-image/variants/debian-7.8.conf <<-EOF
 SWAP=no
 FILESYSTEM="ext4"
 IMAGE_NAME="debian-7.8"
 IMAGE_TYPE="dump"
 IMAGE_DIR="/var/cache/ganeti-instance-image"
 ARCH="x86_64"
 CUSTOMIZE_DIR="/etc/ganeti/instance-image/hooks"
 IMAGE_DEBUG=0
EOF

The default variant configuration file must not be empty, image name and architecture will be used in dump generation. Set global settings like mines:

cat >> /etc/ganeti/instance-image/variants/default.conf <<-EOF
##############################################################
# Default instance template
SWAP=no
IMAGE_NAME="default"
ARCH="x86_64"
EOF

2. Network setup

Now we need to customize the OS installation scripts for the instances, starting with network. It’s a standard network setup for instances, with IP address.

The LAN config for all the instances, except IP address, goes to a subnet file. It’s basically a simple ripe text file with the definition of the network for an interface (vmbr0):

cat > /etc/ganeti/instance-image/networks/subnets/vmbr0  <-EOF
   GATEWAY=192.168.1.1
   NETMASK=255.255.255.0
   NETWORK=192.168.1.0
   BROADCAST=192.168.1.255
EOF

Then for each instance, create 1 file with the FQDN name of the instance:

cat > /etc/ganeti/instance-image/networks/instances/os.dxhost.hu   <-EOF
ADDRESS=192.168.1.101
SUBNET=vmbr0
EOF

Check and update also the DNS config for the instance with its FQDN name in /etc/hosts.

cat >> /etc/hosts <<-EOF
192.168.1.101   os.dxhost.hu     os
EOF

3. Hooks

As the instance has its own kernel, some hooks need to be run at OS installation.

This setup has to be done only once on the cluster. If the execution flags are already set, there is nothing to do.

ls -la /etc/ganeti/instance-image

To set the execution flags, use these commands. No change in content needed.

chmod +x /etc/ganeti/instance-image/hooks/grub
chmod +x /etc/ganeti/instance-image/hooks/interfaces
chmod +x /etc/ganeti/instance-image/hooks/ssh

4. Synchronize to all nodes

Synchronize all the files created or updated over all nodes of the cluster. Run it from node master!

# variants
gnt-cluster copyfile /etc/ganeti/instance-image/variants.list
gnt-cluster copyfile /etc/ganeti/instance-image/variants/debian-7.8.conf
# network
gnt-cluster copyfile /etc/ganeti/instance-image/networks/subnets/vmbr0
gnt-cluster copyfile /etc/ganeti/instance-image/networks/instances/os.dxhost.hu
# hooks
gnt-cluster copyfile /etc/ganeti/instance-image/hooks/grub
gnt-cluster copyfile /etc/ganeti/instance-image/hooks/interfaces
gnt-cluster copyfile /etc/ganeti/instance-image/hooks/ssh
# /etc/hosts
gnt-cluster copyfile /etc/hosts

Install and config Debian OS

Create the first instance, which will be only used as a template for filesystem dump of final instances.

gnt-instance add -n node5 -t plain -s 5g -o image+debian-7.8 --no-start --no-install os.dxhost.hu

A plain disk layout is enough here, used image template is the new one defined above, and -n defines the master node.

Note: create this instance on the master node, it will be important at creation of filesystem dumps later!

It neither starts nor installs the instance, just creating its disk in Ganeti, with this outcome:

Sun Jan 25 12:41:27 2015 * disk 0, size 5.0G
Sun Jan 25 12:41:27 2015 * creating instance disks...
Sun Jan 25 12:41:29 2015 adding instance os.dxhost.hu to cluster config
Sun Jan 25 12:41:31 2015  - INFO: Waiting for instance os.dxhost.hu to sync disks
Sun Jan 25 12:41:31 2015  - INFO: Instance os.dxhost.hu's disks are in sync

Check the instance list on master node, and get its primary node (pnode), as well as the VNC connection parameters (the last one, called network_port). You'll need them for the VNC connection.

gnt-instance list -o name,os,pnode,status,disk_template,network_port

Now one can modify the instance configuration, if other then the defaults of the cluster are required.

Add second NIC:

gnt-instance modify --net add:link=vmbr2 os

Change memory:

gnt-instance modify -B memory=2048M os

If you are done, connect via VNC, and start the instance by booting now from the debian ISO image:

gnt-instance start -H boot_order=cdrom,cdrom_image_path=/var/local/iso/debian-7.8.0-amd64-CD-1.iso os.dxhost.hu

It's the usual Debian server process, only with few points to note:

  • setup manual network config, same IP as above
  • manual partitioning, with no swap in partition layout, see why here.
    • 0.4 GB /boot ext4 primary with boot flag
    • 5.0 GB / ext4 primary
      • install only the SSH server and the standard system utilities

Reboot when ready, and check the instance list on the master node:

gnt-instance list -o name,os,pnode,snodes,status,disk_template,network_port | grep running

Login again to the instance, and you will need to run update-grub and reboot the VM to get the new settings.

ssh os.dxhost.hu
update-grub
grub-install /dev/vda

Test, update, configure, install what you want. For example:

  • run basic debian config script like this:
wget http://gitlab.dxhost.hu/doka/ganeti-setup/raw/master/deb77_setup.sh
chmod 755 deb77_setup.sh
./deb77_setup.sh

  • setup SSH and copy public key from master node:

    ssh root@os "mkdir .ssh;chmod 700 .ssh;" cat ~/.ssh/id*sa.pub | ssh root@os "cat >> .ssh/authorizedkeys" ssh root@os "chmod 600 .ssh/authorized_keys;"```

  • Set root password

    passwd

Finally: logout and restart the instance on master node.

gnt-instance reboot os

Create OS filesystem dump for instances

Using the Debian 7.8 instance, created above, I'll create new instances, based on its filesystem dump. Of course, one can take any existing instance, and use it this way to create new instances.

First, in order to generate the filesystem dump, shutdown the instance:

gnt-instance shutdown os

Run the make-dump script.

/usr/share/ganeti/os/image/tools/make-dump os

Note: it only works if the primary node for the instance is the master node of the cluster!

It generates some output lines, like these:

Creating dump from os for boot at /var/cache/ganeti-instance-image/debian-7.8-x86_64-boot.dump
  DUMP: Date of this level 0 dump: Sun Jan 25 13:32:55 2015
  DUMP: Dumping /dev/mapper/ganeti-29bcc64a--3f8b--4530--ac73--48ebc1612dca.disk0-1 (an unlisted file system) to /var/cache/ganeti-instance-image/debian-7.8-x86_64-boot.dump
  DUMP: Label: none
  DUMP: Writing 10 Kilobyte records
  DUMP: Compressing output at compression level 9 (zlib)
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 17401 blocks.
  DUMP: Volume 1 started with block 1 at: Sun Jan 25 13:32:55 2015
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: Closing /var/cache/ganeti-instance-image/debian-7.8-x86_64-boot.dump
  DUMP: Volume 1 completed at: Sun Jan 25 13:32:56 2015
  DUMP: Volume 1 took 0:00:01
  DUMP: Volume 1 transfer rate: 14221 kB/s
  DUMP: Volume 1 17380kB uncompressed, 14221kB compressed, 1.223:1
  DUMP: 17380 blocks (16.97MB) on 1 volume(s)
  DUMP: finished in 1 seconds, throughput 17380 kBytes/sec
  DUMP: Date of this level 0 dump: Sun Jan 25 13:32:55 2015
  DUMP: Date this dump completed:  Sun Jan 25 13:32:56 2015
  DUMP: Average transfer rate: 14221 kB/s
  DUMP: Wrote 17380kB uncompressed, 14221kB compressed, 1.223:1
  DUMP: DUMP IS DONE
Creating dump from os for root at /var/cache/ganeti-instance-image/debian-7.8-x86_64-root.dump
  DUMP: Date of this level 0 dump: Sun Jan 25 13:32:56 2015
  DUMP: Dumping /dev/mapper/ganeti-29bcc64a--3f8b--4530--ac73--48ebc1612dca.disk0-2 (an unlisted file system) to /var/cache/ganeti-instance-image/debian-7.8-x86_64-root.dump
  DUMP: Label: none
  DUMP: Writing 10 Kilobyte records
  DUMP: Compressing output at compression level 9 (zlib)
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 764636 blocks.
  DUMP: Volume 1 started with block 1 at: Sun Jan 25 13:32:57 2015
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: Closing /var/cache/ganeti-instance-image/debian-7.8-x86_64-root.dump
  DUMP: Volume 1 completed at: Sun Jan 25 13:33:43 2015
  DUMP: Volume 1 took 0:00:46
  DUMP: Volume 1 transfer rate: 5852 kB/s
  DUMP: Volume 1 764410kB uncompressed, 269218kB compressed, 2.840:1
  DUMP: 764410 blocks (746.49MB) on 1 volume(s)
  DUMP: finished in 46 seconds, throughput 16617 kBytes/sec
  DUMP: Date of this level 0 dump: Sun Jan 25 13:32:56 2015
  DUMP: Date this dump completed:  Sun Jan 25 13:33:43 2015
  DUMP: Average transfer rate: 5852 kB/s
  DUMP: Wrote 764410kB uncompressed, 269218kB compressed, 2.840:1
  DUMP: DUMP IS DONE

See the dumps:

ls -l /var/cache/ganeti-instance-image

I have these dumps, last two lines are for the fresh Debian 7.8 image.

-rw-r--r-- 1 root root  14M Aug 30 14:47 debian-7.6-x86_64-boot.dump
-rw-r--r-- 1 root root 269M Aug 30 14:47 debian-7.6-x86_64-root.dump
-rw-r--r-- 1 root root  14M Oct 30 19:17 debian-7.7-x86_64-boot.dump
-rw-r--r-- 1 root root 263M Oct 30 19:17 debian-7.7-x86_64-root.dump
-rw-r--r-- 1 root root  14M Jan 25 13:32 debian-7.8-x86_64-boot.dump
-rw-r--r-- 1 root root 263M Jan 25 13:33 debian-7.8-x86_64-root.dump

After dumps are done, do not forget to sync it on all the cluster nodes. Again, Ganeti has the tool to do it:

gnt-cluster copyfile /var/cache/ganeti-instance-image/debian-7.8-x86_64-boot.dump
gnt-cluster copyfile /var/cache/ganeti-instance-image/debian-7.8-x86_64-root.dump

Create new Debian 7.8 based instances

Now, I create a new instance (www.dxhost.hu) based on Debian 7.8 OS image.

First, customize the network (IP address) for the particular instance. For each instance, add one file with the FQDN name to the instances:

cat >> /etc/ganeti/instance-image/networks/instances/www.dxhost.hu   <<-EOF
ADDRESS=192.168.1.102
SUBNET=vmbr0
EOF

Check and update also the DNS config for the instance with its FQDN name in /etc/hosts.

cat >> /etc/hosts <<-EOF
192.168.1.102   www.dxhost.hu    www
EOF

And copy these files to all nodes!

gnt-cluster copyfile /etc/ganeti/instance-image/networks/instances/www.dxhost.hu
gnt-cluster copyfile /etc/hosts

Now I can create instance. Set -o according to the OS template to be used: image+debian-7.8. Disk layout can be drbd, one can add more memory, and so on.

gnt-instance add -t drbd -o image+debian-7.8 -s 5g -B memory=2048M www

Some output:

Sun Jan 25 13:53:16 2015  - INFO: Resolved given name 'www' to 'www.dxhost.hu'
Sun Jan 25 13:53:21 2015  - INFO: Selected nodes for instance gl77.dxhost.hu via iallocator hail: node6.dxhost.hu, node4.dxhost.hu
Sun Jan 25 13:53:22 2015 * creating instance disks...
Sun Jan 25 13:53:49 2015 adding instance www.dxhost.hu to cluster config
Sun Jan 25 13:53:52 2015  - INFO: Waiting for instance www.dxhost.hu to sync disks
Sun Jan 25 13:53:56 2015  - INFO: - device disk/0: 18.50% done, 1m 11s remaining (estimated)
Sun Jan 25 13:55:01 2015  - INFO: - device disk/0: 93.50% done, 5s remaining (estimated)
Sun Jan 25 13:55:09 2015  - INFO: Instance www.dxhost.hu's disks are in sync
Sun Jan 25 13:55:09 2015 * running the instance OS create scripts...
Sun Jan 25 13:55:53 2015 * starting instance...

Check in the instance list, and get its primary node:

gnt-instance list -o name,os,pnode,status,disk_template,network_port | grep www

It shows up as a running instance:

www.dxhost.hu     image+debian-7.8 node6.dxhost.hu running    drbd          11003

Finally, the new, Debian 7.8 based instance is up&running, and I can start to play with it:

ssh root@www

Note: login works from the primary node of the instance in this setup!