XENBEgridClient

From Begrid Wiki
Jump to navigationJump to search

XEN managed BEgrid client

The idea is to use XEN to save on needed hardware resources at BEgrid sites, by combining on the same hardware the NAT box (needed for WN private subnet) and the BEgrid client. This allows to upgrade/reboot the BEgrid client without affecting the NAT functionalities.

Requirements

One machine with 2 network interfaces (needed for NAT) and at least 1GB of memory. The memory will be mainly used by the BEgrid client (e.g. for templates compilation) as the NAT box requires less resources. Provided that the machine has enough memory and diskspace, other DomU's such as a monitoring box can be added.

Quick XEN HOWTO

Have a look at this short introduction to XEN for Centos5 which provides useful and necessary details that are also needed for this XEN setup. From the howto:

  dom0 is the privileged administrative domain of which only one can run. 
  domU is an unprivileged domain, of which many can run at the same time. 
  Although it is an incorrect analogy, it often helps to think of dom0 as the host system, and domU as a guest system.

Installation

The installation consists of setting up the dom0 and then installing the BEgrid client in a domU.

I. Installing dom0 including NAT and DNS

I.1. dom0 server installation

Install a basic SL50 i386 machine (e.g. following the BEgrid client base installation (only the base and post installation are needed).) The only extra requirement is that xen and a xen-enabled kernel are installed. There exists 2 ways to get this:

  • During the installation procedure(recommended!):
    • select Virtualisation during the package selection step. This will install the needed rpms and configure the correct kernel to boot with.
  • After the OS installation:
    • install the xen rpms:
  yum install xen kernel-xen
    • make sure the correct kernel is used during reboot. Edit the /etc/grub.conf file and set the default to a xen enabled kernel. (The first title is 0, the second is 1, etc.)
    • reboot and check if the correct kernel is used

If all went well, you should now have a xen-enabled kernel and the xm tool. Check it with

 xm info

The output should look like

[root@localhost ~]# xm info
host                   : localhost.localdomain
release                : 2.6.18-8.1.3.el5xen
version                : #1 SMP Mon Apr 30 14:45:24 EDT 2007
machine                : i686
nr_cpus                : 1
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 2660
hw_caps                : bfebfbff:00000000:00000000:00000080:00004400
total_memory           : 1006
free_memory            : 0
xen_major              : 3
xen_minor              : 0
xen_extra              : .3-rc5-8.1.3.el
xen_caps               : xen-3.0-x86_32p
xen_pagesize           : 4096
platform_params        : virt_start=0xf5800000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.1 20070105 (Red Hat 4.1.1-52)
cc_compile_by          : brewbuilder
cc_compile_domain      : (none)
cc_compile_date        : Mon Apr 30 14:13:15 EDT 2007
xend_config_format     : 2

I.2. NAT on dom0

  • configure both interfaces (one with public IP and the other for the NAT with private addresses)
  • add iptables rules (the --to-source address has to be the public interface of the dom0)
iptables -t nat -A POSTROUTING -s 192.168.0.0/255.255.0.0 -d ! 192.168.0.0/255.255.0.0 -j SNAT --to-source 193.190.246.188
/sbin/iptables-save > /etc/sysconfig/iptables
  • enable ip forwarding and set it permanently by running these lines:
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 'FORWARD_IPV4=yes' >> /etc/sysconfig/network
  • With the new version of Red Hat Linux 6.2 the FORWARD_IPV4= parameter is now specified in the /etc/sysctl.conf file instead of the /etc/sysconfig/network file. So change this in /etc/sysctl.conf:
net.ipv4.ip_forward = 1
  • restart iptables
service iptables restart
  • test that you get the correct rules set
iptables -L -t nat

I.3. DNS on dom0 (optional/recommended)

  • you need a DNS for the WNs
yum install bind-utils bind caching-nameserver
  • enable the service
chkconfig --add named
chkconfig --level 345 named on
  • configuration files
    • modify with /path/to/your/configfiles
    • /etc/named.conf (example for iihe.ac.be)
// generated by named-bootconf.pl

options {
        directory "/var/named";
        /*
*****If there is a firewall between you and nameservers you want
*****to talk to, you might need to uncomment the query-source
*****directive below.  Previous versions of BIND always asked
*****questions using port 53, but BIND 8.1 uses an unprivileged
*****port by default.
*****/
        // query-source address * port 53;
};

//
// a caching only nameserver config
//
controls {
        inet 127.0.0.1 allow { localhost; } keys { rndckey; };
};
zone "." IN {
        type hint;
        file "named.ca";
};

zone "localhost" IN {
        type master;
        file "localhost.zone";
        allow-update { none; };
};

zone "0.0.127.in-addr.arpa" IN {
        type master;
        file "named.local";
        allow-update { none; };
};

include "/etc/rndc.key";

zone "wn.iihe.ac.be" IN {
  type master;
  file "wn.zone";
  allow-update { none; };
};

zone "168.192.in-addr.arpa" IN {
  type master;
  file "wn.rev";
  allow-update { none; };
};


  • /var/named/wn.zone
    • q is the name of the dom0: replace it with yours
    • q3 is the name of the domU: replace it with yours
$TTL 3600
@                   IN   SOA   q.wn.iihe.ac.be. admin.wn.iihe.ac.be. (
                               2007071200       ; Serial
                               3600             ; Refresh every hour
                               900              ; Retry every 15 minutes
                               3600000          ; Expire 1000 hours
                               3600 )           ; Minimum 1 hour

wn.iihe.ac.be.  IN      NS      q.wn.iihe.ac.be.

q               IN      A       192.168.10.100
q3              IN      A       192.168.10.11

node11-1        IN      A       192.168.11.1
node11-2        IN      A       192.168.11.2
node11-3        IN      A       192.168.11.3
node11-4        IN      A       192.168.11.4
node11-5        IN      A       192.168.11.5
node11-6        IN      A       192.168.11.6
node11-7        IN      A       192.168.11.7
node11-8        IN      A       192.168.11.8

  • /var/named/wn.rev
    • q is the name of the dom0: replace it with yours
    • q3 is the name of the domU: replace it with yours
$TTL 3600
@       IN      SOA     q.wn.iihe.ac.be. admin.wn.iihe.ac.be. (
                                1       ; Serial
                                3600    ; Refresh every hour
                                900     ; Retry every 15 minutes
                                3600000 ; Expire 1000 hours
                                3600 )  ; Minimum 1 hour

          IN      NS      q.wn.iihe.ac.be.

100.10     IN      PTR     q.wn.iihe.ac.be.
11.10     IN      PTR     q3.wn.iihe.ac.be.


#11    IN      PTR     node11-1.wn.iihe.ac.be.
2.11    IN      PTR     node11-2.wn.iihe.ac.be.
3.11    IN      PTR     node11-3.wn.iihe.ac.be.
4.11    IN      PTR     node11-4.wn.iihe.ac.be.
5.11    IN      PTR     node11-5.wn.iihe.ac.be.
6.11    IN      PTR     node11-6.wn.iihe.ac.be.

  • maybe also /var/named/named.local
$TTL    86400
@       IN      SOA     localhost. root.localhost.  (
                                      1997022700 ; Serial
                                      28800      ; Refresh
                                      14400      ; Retry
                                      3600000    ; Expire
                                      86400 )    ; Minimum
              IN      NS      localhost.

1       IN      PTR     localhost.
    • /var/named/localhost.zone
$TTL    86400
@               IN SOA  @       root (
                                        42              ; serial (d. adams)
                                        3H              ; refresh
                                        15M             ; retry
                                        1W              ; expiry
                                        1D )            ; minimum

                IN NS           @
                IN A            127.0.0.1
                IN AAAA         ::1


    • /var/named/named.ca
    • generate it with
dig @e.root-servers.net . ns > /var/named/named.ca
    • check the contents, should contain stuff like
;       This file holds the information on root name servers needed to
;       initialize cache of Internet domain name servers
;       (e.g. reference this file in the "cache  .  <file>"
;       configuration file of BIND domain name servers).
;
;       This file is made available by InterNIC
;       under anonymous FTP as
;           file                /domain/named.cache
;           on server           FTP.INTERNIC.NET
;       -OR-                    RS.INTERNIC.NET
;
;       last update:    Jan 29, 2004
;       related version of root zone:   2004012900
;
;
; formerly NS.INTERNIC.NET
;
.                        3600000  IN  NS    A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET.      3600000      A     198.41.0.4
;
; formerly NS1.ISI.EDU
;
.                        3600000      NS    B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET.      3600000      A     192.228.79.201
;
; formerly C.PSI.NET
;
.                        3600000      NS    C.ROOT-SERVERS.NET.
C.ROOT-SERVERS.NET.      3600000      A     192.33.4.12
;
; formerly TERP.UMD.EDU
;
.                        3600000      NS    D.ROOT-SERVERS.NET.
D.ROOT-SERVERS.NET.      3600000      A     128.8.10.90
;
; formerly NS.NASA.GOV
;
.                        3600000      NS    E.ROOT-SERVERS.NET.
E.ROOT-SERVERS.NET.      3600000      A     192.203.230.10
;
; formerly NS.ISC.ORG
;
.                        3600000      NS    F.ROOT-SERVERS.NET.
F.ROOT-SERVERS.NET.      3600000      A     192.5.5.241
;
; formerly NS.NIC.DDN.MIL
;
.                        3600000      NS    G.ROOT-SERVERS.NET.
G.ROOT-SERVERS.NET.      3600000      A     192.112.36.4
;
; formerly AOS.ARL.ARMY.MIL
;
.                        3600000      NS    H.ROOT-SERVERS.NET.
H.ROOT-SERVERS.NET.      3600000      A     128.63.2.53
;
; formerly NIC.NORDU.NET
;
.                        3600000      NS    I.ROOT-SERVERS.NET.
I.ROOT-SERVERS.NET.      3600000      A     192.36.148.17
;
; operated by VeriSign, Inc.
;
.                        3600000      NS    J.ROOT-SERVERS.NET.
J.ROOT-SERVERS.NET.      3600000      A     192.58.128.30
;
; operated by RIPE NCC
;
.                        3600000      NS    K.ROOT-SERVERS.NET.
K.ROOT-SERVERS.NET.      3600000      A     193.0.14.129
;
; operated by ICANN
;
.                        3600000      NS    L.ROOT-SERVERS.NET.
L.ROOT-SERVERS.NET.      3600000      A     198.32.64.12
;
; operated by WIDE
;
.                        3600000      NS    M.ROOT-SERVERS.NET.
M.ROOT-SERVERS.NET.      3600000      A     202.12.27.33
; End of File

  • Start the service
/etc/init.d/named start
  • small script to generate entries in DNS for the workernodes
#!/bin/bash

for i in <tt>seq 11 20</tt>
do
  for j in <tt>seq 1 254</tt>
  do
    echo "node$i-$j     IN      A       192.168.$i.$j" >> /var/named/wn.zone
echo "$j.$i     IN      PTR     node$i-$j.wn.iihe.ac.be." >> /var/named/wn.rev
  done
done

II. BEgrid client on domU

This section describes the additional steps related to the BEgrid client setup using Xen. All further details are identical to the standard installation of a BEgrid client.

Resources

A BEgrid client basically does 2 things that are important to consider when using XEN:

  • profile compilation: this requires memor; at least 650MB of memory should dedicated to the domU for its proper functioning.
  • caching: this requires some disk space; at least 15GB should be foreseen for catching puposes, in addition to the base OS installation disk space requirements(about 10GB).

The default configuration provides the disk space through file based images that are converted in virtual disks. 2 files will be created, one for the OS and one for the web cache.

The dom0 should have set explicitly the amount of memory reserved for it

  • add in /etc/grub.conf to the default boot kernel dom0_mem=350M
    • the default boot kernel is listed as default=<integer>
    • counting starts from 0!!

II.1. Preparation on dom0

  • download xen enabled SL5 i386 install images:
wget -O /boot/vmlinuz-xen-install-i386 http://linuxsoft.cern.ch/scientific/50/i386/images/xen/vmlinuz
wget -O /boot/initrd-xen-install-i386.img http://linuxsoft.cern.ch/scientific/50/i386/images/xen/initrd.img
  • check if the SElinux context is properly set in /var/lib/xen/images/. The important part is xen_image_t
  [root@localhost ~]# ls -Zd /var/lib/xen/images/
  drwxr-xr-x  root root system_u:object_r:xen_image_t    /var/lib/xen/images/
  • create 2 empty files:
  dd if=/dev/zero of=/var/lib/xen/images/quattor-client-os.img oflag=direct bs=1M count=10000
  dd if=/dev/zero of=/var/lib/xen/images/quattor-client-cache.img oflag=direct bs=1M count=15000
  • check the security context of the 2 files if they have the xen_image_t context
 ls -Z /var/lib/xen/images/quattor-client*
  • add a new virtual network interface (first check if device ethX is unused and available with ifconfig <network device>):
/etc/xen/scripts/network-bridge start vifnum=<number of virtual interface> bridge=<xen bridge name> netdev=<network device>
    • e.g. if you setup the dom0 with interface eth0, you probably already have a device called xenbr0. Check it with ifconfig xenbr0.
    To add a new xen virtual device using eth1, configure eth1 first and then run
  /etc/xen/scripts/network-bridge start vifnum=1 bridge=xenbr1 netdev=eth1
      • if you run into troubles during the configuration of the xen-bridge(s), you can remove the configuration and start over with
  /etc/xen/scripts/network-bridge stop vifnum=1 bridge=xenbr1 netdev=eth1
    • To make it permanent copy the /etc/xen/scripts/network-bridge to /etc/xen/scripts/network-bridge.xen.
      • Edit /etc/xen/xend-config.sxp and add a line to your new network bridge script (this example uses "network-xen-multi-bridge").
      • In the xend-config.sxp file, the new line should reflect your new script (keep the parentheses):
(network-script network-xen-multi-bridge)
      • Make sure to comment the line that states:
(network-script network-bridge)
      • If you want to create multiple Xen bridges, you must create a custom script. We call it /etc/xen/scripts/network-xen-multi-bridge. This example below creates two Xen bridges (called xenbr0 and xenbr1) and attaches them to eth0 and eth1, respectively:
#!/bin/sh
# network-xen-multi-bridge
# Exit if anything goes wrong
set -e
# First arg is operation.
OP=$1
shift
script=/etc/xen/scripts/network-bridge.xen
case "${OP}" in
        start)
                $script start vifnum=1 bridge=xenbr1 netdev=eth1
                $script start vifnum=0 bridge=xenbr0 netdev=eth0
                ;;

        stop)
                $script stop vifnum=1 bridge=xenbr1 netdev=eth1
                $script stop vifnum=0 bridge=xenbr0 netdev=eth0
                ;;

        status)
                $script status vifnum=1 bridge=xenbr1 netdev=eth1
                $script status vifnum=0 bridge=xenbr0 netdev=eth0
                ;;
*****)
                echo 'Unknown command: ' ${OP}
                echo 'Valid commands are: start, stop, status'
                exit 1
esac

domU configuration files

2 XEN configuration files are needed.

  • the first one is solely for the domU installation. It contains references to kernel installation and initrd image.
  • the second configuration file is needed for everything else. It uses pygrub to select the installation kernel.

II.2 domU installation

  • create a file /etc/xen/BEgrid_client_install containing
kernel = "/boot/vmlinuz-xen-install-i386"
ramdisk = "/boot/initrd-xen-install-i386.img"
extra = "text"
name = "BEgrid_client_install"
memory = "700"
disk = [ 'tap:aio:/var/lib/xen/images/quattor-client-os.img,xvda,w',]
vif = [ 'bridge=xenbr0','bridge=xenbr1', ]
vcpus=1
on_reboot = 'destroy'
on_crash = 'destroy'
  • start the domU with xm create BEgrid_client_install.
  • watch and configure the installation of the domU, use xm console BEgrid_client_install.
    • you will have a text terminal with the familiar installation process.
    • to stop it forcefully use xm destroy BEgrid_client_install
  • follow the steps in the normal BEgrid client base install
    • the disk xvda is the quattor-client-os.img created before.
    • when configuring the network interfaces, make sure to assign a public IP address to the device with the same name as on the dom0. (if eth0 is the public interface on dom0, make it also the public interface on the domU)
    • package selection: you can't choose Virtualisation here. It's ok, the installer knows what it's doing.
  • after the installation you will get a prompt back in dom0.

II.3. domU post-installation

  • to continue, you need to create a file /etc/xen/BEgrid_client containing
name = "BEgrid_client"
memory = "700"
disk = [ 'tap:aio:/var/lib/xen/images/quattor-client-os.img,xvda,w','tap:aio:/var/lib/xen/images/quattor-client-cache.img,xvdb,w', ]
vif = [ 'bridge=xenbr0','bridge=xenbr1', ]
vcpus=1
bootloader="/usr/bin/pygrub"
on_reboot = 'restart'
on_crash = 'restart'
  • to start the domain automatically when the (dom0) system is started, copy the domain configuration to the /etc/xen/auto directory.
    • This will also shut down the domain properly when the system is shut down.
ln -s /etc/xen/BEgrid_client /etc/xen/auto/BEgrid_client
  • start this domU with xm create BEgrid_client.
  • you can use xm console BEgrid_client to connect or login using ssh.
    • it's probably best to use ssh
  • you now need to add the device xvdb to the setup (doing it during the installation is difficult).
    • make a partition on it and configure it for mounting as /var/www/cache
#comment
**(note Wim Obbels: parted '/dev/xvdb unit MB print' failed with 'Error: Unable to open /dev/xvdb - unrecognised disk label.')
    I (interactively) created one big partition with fdisk, and deleted it again ...; then following procedure worked)
  
  this was caused by confirmation probblems with parted on a clean device (does not need a 'y'). fixed by making sure the device is empty (and has no label).

  dd if=/dev/zero of=/dev/xvdb bs=1M count=1
  parted /dev/xvdb mklabel msdos
  end_in_MB=<tt>parted /dev/xvdb unit MB print |sed -nr '{ s#Disk[[:space:]]+/dev/[a-z]+:[[:space:]]([0-9]+)MB#\1#p }'</tt>
  parted /dev/xvdb mkpart primary ext3 1 $end_in_MB
  mkfs.ext3 /dev/xvdb1
  mkdir -p /var/www/cache
  echo "/dev/xvdb1 /var/www/cache  ext3    defaults        1 2" >> /etc/fstab
  mount /var/www/cache
  chown apache.apache /var/www/cache
  • make sure that the dom0 is configured as the default DNS server for the domU. Check the content of the file /etc/resolv.conf and see if it contains the dom0 as the first server. Example of a valid file for IIHE cluster (193.190.246.188 is the dom0, 193.190.247.71 is the 'normal' DNS)
search iihe.ac.be
nameserver 193.190.246.188
nameserver 193.190.247.71

Back to BEgrid_And_Quattor page


Template:TracNotice