POWER5 Virtualization: How to set up the SUSE Linux Virtual I/O Server

Document options Document options requiring JavaScript are not displayed Rate this page Help us improve this content

Level: Introductory

Nigel Griffiths (nag@uk.ibm.com), pSeries Technical Support, IBM　

19 May 2005

Reduce your operation costs for complex environments by creating efficient and flexible virtualization capabilities. Nigel Griffiths describes the benefits of the IBM® POWER5™ servers and provides examples on how to set up the environment for pSeries® p5 and eServer™ OpenPower systems.

Introduction

Virtualization is a hot topic, and the computing industry has varying views on recommendations for different technologies and solutions, development, and use. Through examples that apply equally to both pSeries® p5 and eServer™ OpenPower™ systems, this article shows how to set up and use the SUSE Virtual I/O Server (VIO Server). The SUSE Linux™ VIO Server is an alternative to the VIO Server software offered by IBM®. It provides:

• A more familiar Linux command environment.
• The option to run systems management tasks or applications alongside the VIO Server.
• Different device support options.
• Alternative costs and support.

Back to top

Background

The POWER5™-based machines have inherited the "know how" from IBM mainframes to provide opportunities for a significant reduction in operating costs for complex environments. Unlike software solutions available from other vendors, the POWER5 implementation uses advanced processor features, firmware (called the POWER Hypervisor™), and hardware features to create efficient and flexible virtualization capabilities. Uniquely, these capabilities are offered from the top to the bottom of the server range -- from a powerful 64-way SMP machine down to a two-way, desk-side system. The key to this virtualization is the VIO Server.

Back to top

What is a VIO Server?

Since October 2001, pSeries servers from IBM allow you to divide a machine into logical partitions (LPARs), with each LPAR running a different operating system image -- effectively a server within a server. You can achieve this by logically splitting up a large machine into smaller units with an independent CPU, memory, and PCI adapter slots allocations.

The new POWER5 machines (pSeries p5 and OpenPower servers) can also run an LPAR with less than one whole CPU -- up to 10 LPARs per CPU. So on a four CPU machine, 20 LPARs can easily be running. With each LPAR needing a minimum of one SCSI adapter for disk I/O and one Ethernet adapter for networking, the 20 LPARs would require the server to have at least 40 PCI adapters. This is where the VIO Server helps.

The VIO Server owns real PCI adapters (Ethernet, SCSI, or SAN), but lets other LPARs share them remotely using the built-in Hypervisor services. These other LPARs are Virtual I/O client partitions (VIO client) and, because they don't need real physical disks or Ethernet adapters to run, you can create them quickly and cheaply.

VIO Server implementations

You can have many different VIO Server implementations:

• The Advanced Power Virtualization (APV) VIO Server from IBM for pSeries p5 and the Advanced Open Power Virtualization (AOPV) VIO Server from IBM for OpenPower machines are possible implementations. Both of these are special-purpose, single-function appliances and are not intended to run general applications.
• The Linux VIO Server for pSeries p5 or OpenPower hardware first became available with the SUSE SLES 9 distribution. Unlike the APV and AOPV, this is a full copy of the Linux operating system. This means it can run other central services such as:
  • NFS
  • Network installation
  • DNS
  • Web site (Apache)
  • Samba services
  You need to make sure that these functions do not interfere with the performance of the VIO Server service. This software is also available on the Debian Linux for POWER distribution.

There are four different VIO client implementations (actually, these are just the regular operating systems, but they include the device drivers for running a VIO client):

• AIX® 5.3 -- only supported by the APV or AOPV VIO Server
• Linux SUSE SLES 9
• Linux Red Hat 3 update 3 onwards and Red Hat 4
• Linux Debian for POWER

This article only covers the SUSE Linux VIO clients. It should also apply to the Debian version, but this assertion has not been explicitly tested.

Virtual SCSI disks

The VIO Server provides a virtual SCSI disk service. See Figure 1 below.

Figure 1 shows a single VIO Server providing virtual SCSI services to multiple VIO client partitions. Each VIO client operates as if it had a dedicated SCSI device but, in fact, each client device is a real disk partition (like a partition created using the Linux fdisk command) on the VIO Server. Alternatively, you could use a loopback device driver to a file in a regular file system on the VIO Server. The VIO Server and VIO client communicate using the internal pSeries Hypervisor firmware (PHYP) feature, which efficiently allows disk I/O requests to be transferred between the LPARs using a message-passing protocol. The VIO Server has four disks which could be SCSI or Fiber Channel SAN disks. Data protection can be provided by hardware or software (Linux), mirroring, or RAID5. The VIO clients use the VIO client device driver just as they would a regular local device disk to communicate with the matching server VIO device driver. Then the VIO Server actually does the disk transfers on behalf of the VIO client. Note that there is a strict client/server relationship between the VIO client and the VIO Server.

Virtual Ethernet

The LPARs in the machine can use the Virtual Ethernet switch service (in the Hypervisor) in a number of different ways.

• Case one
  Internal only networks: You can use the Virtual Ethernet to allow TPC/IP (Transmission Control Protocol/Internet Protocol) to communicate between the LPARs -- see Figure 2 below. This provides high-speed data transfer without any hardware adapters starting at roughly one GB per second, but can be much higher -- especially using larger block sizes. You'll also notice that there is no client/server relationship between the LPARs -- all are equally using the Virtual Ethernet. There can be many Virtual Ethernets in one machine, where groups of LPARs can communicate only within the Virtual Ethernet they connect to. This allows fast communication and complete security without purchasing additional Ethernet adapters, cables, hubs, or routers.
  
  Figure 2. Virtual Ethernet -- Private/internal only networks
  
• Case two
  Routing to a physical LAN: One LPAR on the Virtual Ethernet can also communicate externally to other machines using a real physical network on behalf of all the LPARs. In this case, this special LPAR is being used to route Ethernet packets between the internal Virtual Ethernet and the external physical Ethernet network. Standard Linux features can be used to do this. This works well, but involves setting up TCP/IP routes between the two networks (internal and external). The setup can take extra time, too. Figure 3 below shows one LPAR with a real physical Ethernet adapter that provides standard network routing between the two Ethernets. Note: This is not using any VIO Server features.

Figure 3. Internal Virtual Ethernet with route to external LAN

• Case three
  Bridging to a physical LAN: Here, the VIO Server is being used to bridge Ethernet packets between the internal Virtual Ethernet and the external physical Ethernet network so that all the LPARs appear as regular machines on the physical network. This is simple to set up and is the option used in the example in this article. In Figure 4 below, you"ll notice that the Virtual IO Server is being used to join the two networks, using the SUSE Bridging Utilities package. Strictly speaking, this is a completely separate function to the Virtual IO Server, but it's common practice to run the virtual SCSI Server and TCP/IP bridging on the same LPAR.

Figure 4. Internal Virtual Ethernet with bridge to external LAN

• Case four
  Bridging with VLANs (virtual LANs): This is the same as Case three, except there is a number of VLANs within the machine (using Virtual Ethernet). These are connected to VLANs on the external network using a bridging LPAR and a network router that supports VLAN. This complex scenario is beyond the scope of this article, but some hints are included and it's supported.

Back to top

Why use the SUSE VIO Server?

You can use a VIO Server in any number of scenarios. Here are five typical examples that would make good use of a VIO Server. You might find your environment needs are similar to one of these:

1. Small machine with limited PCI slots

   The OpenPower 710 or 720 or the p5 models 510, 520, or 550 have one to four CPUs, but limited adapter slots (three or five), in addition to two built-in Ethernets and one built-in SCSI adapter. These machines run out of PCI slots as you add LPARs.

   • You have one set of internal SCSI disks or you can split the SCSI disks in two four packs on the OpenPower 720 or p5-550. This gives you two LPARs (at most) using the internal disks. So, you might run a VIO Server to support the other LPARs.
   • You can use a VIO Server (0.5 of a CPU) plus four to six clients (0.1 to one CPU). Typically, clients might be small, for example, four to 16 GB virtual SCSI disks and one Virtual Ethernet for the whole machine.
   • Figure 5 shows multiple LPARs running on a single disk pack.

     Figure 5. Multiple LPARs
     

   • Mid-range machines with extra small workloads
     
     This might be an eight or 16 CPU machine with large partitions for production use. But many system administrators also want a small number of extra LPARs rather than purchase an extra machine. A VIO Server can easily host a half dozen smaller LPARs.
   • For example, larger production LPARs might have one to four larger dedicated CPUs, disk I/O, and networks each.
   • VIO Server is used for "bits and bobs" LPAR such as test, development, training, upgrade practice, new application trials, and so forth. Typically, VIO clients might have four GB to eight GB virtual SCSI disks and one or two Virtual Ethernets.
   • Figure 6 below shows three large production LPARs running (they would have dedicated disks and Ethernet) with a few extra small VIO clients and one VIO Server on the machine using "spare" capacity. This spare capacity could be demanded by the production LPARs during peaks in their workload.

     Figure 6. Three large production LPARs
     

2. Ranch or server farm style

   Lots of small server consolidation workloads from smaller or older machines or many small servers are required, but they are unlikely to peak at the same time.

   • The machine is to run lots of LPARs (for example, 10 to 20 clients on a four-way machine or many times that on larger machines). Each LPAR is for small applications but not high demand (0.2, or 0.5 CPU up to 2 CPU).
   • This could be server consolidation or, for example, a collection of small Web servers where isolation of the data is important.
   • VIO Server has one or two CPUs and probably RAID 5 SCSI disks or SAN disks.
   • Typically, clients have one or more four GB virtual SCSI disks each and might have different groups of LPARs around a different Virtual Ethernet.

     Figure 7. Different groups of LPARs
     

   • Figure 7 shows dozens of VIO clients with a medium size VIO Server supporting them on what might be several disk packs.

3. Serious I/O setup only once (to reduce setup and management)
   • For example, the VIO Server has SAN disks connected using two to four Fibre Channel adapters and two Ethernet adapters to run Ether-Channel for redundancy and additional bandwidth.
   • The VIO Server has load balancing and failover, but VIO clients have much simpler disk and Ethernet setup.
   • Typically, the VIO Server could have one to three CPUs, but the VIO clients are larger, too. For example, the VIO client might have one to eight CPUs to run large applications. They can have over 100 GB of virtual SCSI disks and many Virtual Ethernets.
   • This complex setup is not covered in this article, as it's an advanced topic.
   • Figure 8 shows one regular LPAR (it would have dedicated disks and Ethernet) and a large configured VIO Server with multiple paths to disks and Ethernet. This is supporting some large VIO client LPARs.

     Figure 8. Regular LPAR
     

4. Serious with high availability backup

   Same as above, but use a second VIO Server for availability or throughput.

   • For very high availability, there are arguments that you should spread your access to virtual SCSI and Virtual Ethernet across two VIO servers. Why? Because if you loose one VIO Server, you can carry on running.
   • The counter argument is that the VIO Server is only running a few device drivers. Devices drivers are extremely reliable. Also, anything that would crash one VIO server could crash the second one, too! Figure 9 shows that instead of using a local physical device driver, the VIO client uses the virtual resource device drivers to communicate with the VIO Server, which does the real I/O. Except for the virtual VIO Server device drivers and the physical resource device drivers, there is very little code running on the VIO Server. This means there is little to go wrong on the VIO Server side.
     
     Figure 9. VIO Server
     

This article does not cover duplicated VIO Servers. Further details can be found in the Advanced POWER Virtualization on IBM p5 Servers redbook.

Back to top

Prerequisites before you start

Software

Your first question might be:

• Where do I get the SUSE VIO Server and how do you know if SUSE VIO Server software is installed?

Answer:

• It's always installed with SUSE itself, as it's an integral function of the SUSE SLES 9.

Notes:

• It's not available with SUSE SLES 8, as Linux 2.6 kernel features are used.
• It's not available with Red Hat EL3 or Red Hat EL4.
• Install all the available SUSE Service Packs.
• The VIO Server is also available with the Debian distribution of Linux on POWER.

Hardware

You need:

• An OpenPower pSeries or p5 machine with spare resources:
  • Some CPU resources -- can be less than one CPU.
  • Memory -- 512 GB per LPAR (can be just 256 MB).
  • Real Ethernet adapter.
  • Some time with the CD drive -- unless you prefer network installation.
  • SCSI adapter and a SCSI disk -- (A SAN disk could equally be used, but the details are not included here.).
• The hardware virtualization feature, which is needed for LPAR and VIO Server features but optional on some POWER5 machines.
• SUSE SLES 9 for POWER set of CDs.
• There is also a bridge utilities package you need to install, but these are on the standard SUSE SLES9 CDs.

Skills

This document does't take a "boil-the-ocean" approach and will not show you a screen-by-screen level of detail of each input field.

You should already understand the following:

• Basic Linux systems administration such as installing an RPM (rpm -Uvh <package>.rpm), configuring an Ethernet network adapter (ifconfig eth0 <ipaddress> mask 255.255.255.0), and managing a filesystem (mount /dev/sda5 /mnt). As these tasks are identical to working on the Intel platform, there are many books, training courses, and Internet material covering these regular system administration commands and tasks.
• How to install SUSE Linux in either text mode (on a dumb/ASCII screen) or using a VNC session. Once you have installed Linux a couple of times, this becomes a simple, "follow your nose" task. For the VNC install, the boot prompt extra command is "vnc=1 password=abc123". Note six characters for the password. The system also prompts you for the other details.
• The HMC:
  • How to install the HMC hardware and software.
  • How to set it up (it's assumed this has already been done).
  • How to use the HMC to create and start a simple LPAR and its profiles.
• The pSeries p5 and OpenPower range of machines internals like the names of the adapter positions. For example, the Tn names for internal adapters and Cn for real adapters in a PCI slot. You are expected to create the VIO Server LPAR with the right SCSI disks and Ethernet resources on the HMC, with and without the CD. Note: Above "n" is a number of the slot and details can be found in the hardware manuals, IBM Redbooks, or on the large sticker on the outside of the machine covers.

Network

The SUSE VIO Server must be able to communicate directly with the HMC for advanced functions and error reporting, and this is easily forgotten. The following network network is recommended:

Figure 10. The network

Many sites also have other dedicated purpose networks in addition to the above. For example, a network for remote backup or a network dedicated for systems administration. These would be in addition to the networks in the diagram.

Back to top

Getting started

Here is the process to get you started with VIO Server (covered below) in detail.

1. Step 1. Logical diagram of the example
2. Step 2. Planning your setup
3. Step 3. Create the SUSE SLES 9 VIO Server LPAR
4. Step 4. Install SUSE SLES 9 VIO Server
5. Step 5. Install bridge utilities
6. Step 6. HMC defining the VIO Server -- Virtual Ethernet
7. Step 7. HMC defining the VIO Server -- virtual SCSI
8. Step 8. HMC creating the VIO client LPARs
9. Step 9. Clean up the HMC
10. Step 10. Preparing VIO Server for clients
    1. Step 10a. Virtual Ethernet
    2. Step 10b. Virtual SCSI using fdisk type real disk partition for Client A
    3. Step 10c. Virtual SCSI using loopback and filesystem file for Client B
11. Step 11. VIO client LPAR installations

And three common tasks that are recommended or useful:

1. Step 12. Backing up a VIO Server and VIO client
2. Step 13. Cloning a client
3. Step 14. Dynamic LPARs (DLPARs) and RAS

I discuss these topics in detail below, but please note it takes longer to describe than actually implement!

Step 1. Logical diagram of the example setup

Figure 11 below is the diagram of the SUSE VIO Server in an LPAR and two Virtual I/O clients LPARs that are going to be set up for this article.

Ethernet
For simplicity, the VIO Client LPARs are given Ethernet IP addresses within the address range of the regular physical Ethernet network in this computer room. The VIO Server bridges between physical and virtual networks. This means that the client LPARs appears like any other computer to users. This is the most likely option to be implemented and hides the Virtual Ethernet network completely from users, so it's simple to access the client LPARs.

Disks
For the disks, you are going to use the internal SCSI adapter in the VIO Server and one disk.

• The first client's (called Client A) virtual disk connects to a disk partition created with fdisk on the VIO Server.
• The second client's (called Client B) virtual disk is supported using a four GB file in a filesystem and makes use of a loopback driver on the VIO Server.

This shows all of the common types of setup such as bridging networks, disk partitions, and loopback. In practice, most people use disk partitions or loopback, and not both.

Step 2. Planning your setup

The first task is to do some planning of the VIO Server and client LPARs. Experience has shown that creating LPARs without some planning causes problems and can waste a lot of time.

Below is the planning I've done for this example, which is an OpenPower 720. Except for the references to PIC slots like C3, T6, and T14, which are machine dependant, this could be any pSeries p5 or OpenPower machine.

Step 3. Create the SUSE SLES 9 Server LPAR

Next, you need to create the SUSE VIO Server LPAR. You do this on the HMC and create a special VIO Server LPAR, but initially with no extra virtualization features. You will add the virtual features later on. There is just one feature that is different for a regular Linux LPAR -- the LPAR partition environment on the first panel of the Create LPAR wizard. You must not select the AIX or Linux option but must select the VIO Server option, as shown in Figure 12 below.

Create the LPAR and the first profile with the details in Table 1 above.

This document assumes you are familiar with the HMC and creating LPARs -- if not, take a look at the Resources section at the end of this article for other documents that describe how to create LPARs on in the InfoCenter for the manuals. You can call the LPAR profile that is normally used with a name or Normal.

Hints:

• A VIO Server LPAR can use dedicated CPUs -- This is a good idea if you have plenty of CPUs or are expecting to do lots of I/O for many VIO client LPARs and avoids any delay in starting the I/O on the real adapters. Dedicated CPUs are running the VIO Server all the time.
• A VIO Server LPAR can use shared CPUs -- This is a good idea if you don't have whole CPUs that can be assigned. This also means unused CPU cycles are given back to the shared pool for other LPARs to use. If the machine becomes heavily loaded, this can introduce tiny delays in starting the I/O on the real adapters. Shared CPU partitions are time sliced on to the CPU, along with other LPARs. Setting the VIO Server partition uncapped and with a high weight is generally a good idea.
• If you want a simple CPU rule of thumb for those CPUs which are going to be used for the VIO Server and client partitions, assign at least 10 percent of those CPUs to the VIO Server. For example, for five CPUs in the shared pool being used for both VIO Server and VIO clients, allocate 0.5 of a CPU to the VIO Server.
• If you want a simple memory rule of thumb, use 512 MB of memory.
• It's recommended to also have an LPAR profile with the adapter connected to the CD drive included to make installing SUSE SLES 9 from CD straightforward. Copy the "Normal" profile and rename it "Normal with CD", then change the new profile properties to include the CD SCSI adapter. This will be used to initially boot the LPAR with a DVD/CD drive for installing SUSE SLES 9.
• If this is a new machine and you are the only user, it's worth noting that installations go much faster if you assign the LPAR a whole CPU or more. If the LPAR is going to be assigned less than this in production, it can always be reduced later but this simple "trick" might save you 10 minutes per LPAR installation.

Step 4. Install SUSE SLES 9 VIO Server

Next, you need to install SUSE SLES 9 into this partition as a normal SUSE Linux install.

Note that there is no special features for the SUSE-based VIO Server -- it's just a regular SUSE SLES 9 operating system, but with the bridging utilities installed (see the Install the bridge utilities section).

Warning: If you want to use disk partitions for your VIO client LPARs, then you will need some disk space that is not allocated to the basic SUSE partitions.

• If you have just one disk when installing SUSE SLES 9, make sure that the default disk partitions do not use the whole disk or all the disks -- you might have to reduce the root file system disk partition size to do this. It's recommended that you have at least four GB per VIO client, and don't forget you can run out of primary disk partitions. An extended disk partition can be used to allow more disk partitions to be created and used to VIO clients.
• If you have more than one disk, you can install on the first disk and use the second disk for hosting the VIO client disk space.

Fortunately, if you end up with no spare raw disk space, you can always use the loopback scheme to a file in the file system approach (assuming you have the space there).

If you have the SUSE SLES 9 Service Pack, boot from the first CD of the Service Pack and follow the instructions. But note that it will also require the original CDs to install and it,s easy to put the wrong CD in the drive when requested (as there are two CD1s and so forth).

Don't install fancy features like firewalls, enhanced security, printers, and so forth to keep the install simple. If required, these can be added later.

Also, it's recommended that you don't get your VIO Server LPAR working on the network at this point -- this will be done using the bridging utilities later.

Assuming that you now have SUSE SLES 9 up and running, add and set up the VIO Server virtualization features.

Step 5. Install bridge utilities

The only feature that you need, in addition to the standard SUSE SLES 9 installation for Ethernet bridging, is a package on the standard CDs called the bridging utilities.

You will find the commands and tools you need on CD3 as follows:

/SUSE/ppc64/bridge-utils-0.9.6-121.1.ppc64.rpm

Assuming that you have an entry in /etc/fstab like:

/dev/dvd /media/dvd subfs fs=cdfss,ro,procuid,nosuid,nodev,exec,iocharset=utf8 0 0

Then:

# mount /media/dvd
# cd /media/dvd

Install this RPM with:

# rpm -ivh   SUSE/ppc64/bridge-utils-0.9.6-121.1.ppc64.rpm

Check to see they are installed with:

# rpm -qa | grep bridge

Hints: If you have the DLPAR change software installed and working, it's possible to dynamically add Virtual Ethernet and virtual SCSI. But in practice, I recommend you shutdown down your VIO Server and VIO client LPARs during this initial setup to make sure it works fine for the first time. If you set up DLPAR later on, you can then experiment, but remember DLPAR changes also have to be implemented identically in your LPAR profile (if you want the same configuration next time, you need to reboot your LPAR).

In this article, I take the simple and safe approach of shutting down the VIO Server, making changes to the VIO Server profile and restarting it to avoid any confusion and complications.

If you make changes to the LPAR profile, you need to shutdown the LPAR and restart from the HMC in order to pick up those changes. If you use:

shutdown -fr now

-r means reboot in the LPAR, then you will have only the same resources that were available when the LPAR was previously started from the HMC.

Step 6. HMC defining the VIO Server -- Virtual Ethernet

On the HMC, you can now define the Virtual Ethernet. First, shutdown the SUSE VIO Server LPAR (as root: shutdown -fh now). Change your "normal" profile properties on the HCM (right-click the Profile and select Properties) and select the Virtual I/O tab. Then select Ethernet at the bottom and click on Create. By default, this will be allocated to slot number two and port VLAN ID of one. Any LPAR with the same port VLAN ID will be able to communicate with each other. As this is going to be set up as a bridging VIO Server, set the two options as below:

• Check the trunk adapter.
• Leave the IEEE 802.1Q compatible adapter unchecked -- this is only needed if you are using VLANs internally.

Note: If you want different Virtual Ethernet LANs so that different groups of LPARS can communicate with each other, all you need to do is have different port VLAN ID numbers. These more complex configurations are not covered in this article.

In Figure 13 below, you should see the VIO Server in the lower half and the VIO client in the top half. It shows that if the port VLAN IDs are the same, then the LPARs can communicate. It also shows the additional settings for the VIO Server (trunk is selected and IEEE 802.1Q not selected). These additional settings are really for the bridging feature, as Virtual Ethernet does not really have a client/server relationship -- all LPARs are equal on the network.

Figure 13 below shows the Virtual Ethernet settings. At the bottom is the VIO Server (or any LPAR that will be doing the bridging to the real Ethernet adapter) and at the top is the VIO client (any LPAR that only uses the Virtual Ethernet).

On the other non-bridging Virtual Ethernet LPARs, you can use the ifconfig command to set up your network just as you would any network (if using SUSE, use the YaST tool), and it will find the Virtual Ethernet adapter just like any other, as shown in Figure 14 below.

Step 7. HMC defining the VIO Server -- virtual SCSI

On the HMC, you can now define two different virtual SCSI devices. These two types of virtual disk (fdisk-type disk partitions and the loopback to a file) will appear identical on the HMC. It's only on the actual VIO Server LPAR that they are set up any differently.

If not done already, shutdown the SUSE VIO Server LPAR (as root: shutdown -fh now). On the HMC, select the VIO Server and change your "normal" profile properties (right-click the Profile and select Properties) and select the Virtual I/O tab.

• Then select SCSI at the bottom and click on Create.
• This is the VIO Server, so select Adapter Type: Server.
• Then select any Remote Partition and Slot can connect. Ideally, this should name the specific LPAR and slot to eliminate the risk of the wrong connection between server and client, but at this point, you have not created the client partition. So, you can't name it yet. This is fixed up later on (see Clean up the HMC section).
• Then select OK.
• Do this a second time for the second virtual SCSI.

The client LPARs are going to use the SUSE VIO Server slots three and four. Any more SCSI adapters are optional in this example. In practice, the writer typically sets up a handful of extra virtual devices so they can be used in the future, without stopping the VIO Server or having to do dynamic changes. Unused virtual adapters cost very little, so it's not a waste.

Figure 15 illustrates the eventual configuration, showing how the VIO Server (shown at the bottom) and client (shown at the top) both explicitly refer to each other to eliminate errors.

Step 8. Create the VIO client LPARs

Now you can create the two VIO client LPARs for the two different types of virtual SCSI being used in the example. It's assumed that you already know the procedure for creating a regular LPAR. Listed below are the additional things that you need to consider.

• This is a bit obvious, but you don't need real adapters for your disks or Ethernet connection because you are going to use virtual resources for these.
• It's recommended that you install the client LPAR using a CD because it's very simple and straightforward, so you will want to have the CD SCSI adapter within your LPAR. Once installed, you can remove it from the LPAR profile.
• Create two identical LPAR profiles, one with and one without the CD. Once installed, the writer uses NFS to remotely mount a filesystem containing the Linux CDs, so you don't need the CD drive from then on.
• Add the Virtual Ethernet adapter on the VIO screen with the same port, VLAN ID, which is one in this example. Do not select the Trunk or IEEE 802.1Q compatible adapter options -- These are just for the VIO Server partition.
• Add the virtual SCSI adapter.
  • Set the adapter type to Client.
  • Explicitly name the Remote Partition -- For example, the LPAR in which you have the SUSE VIO Server.
  • Explicitly name the Remote Partition Virtual Slot Number -- This is slots three for the first client LPAR (Client A) and four for the second client LPAR (Client B).

Don't forget: You have two client LPARs to create with the two different SCSI remote partition virtual slot numbers, but the same Virtual Ethernet Port VLAN ID.

Step 9. Clean up the HMC

Now that you have created the client LPARs, you can go back to the SUSE VIO Server LPAR and connect up the virtual SCSI adapters explicitly to their virtual client LPARs and slots.

This ensures that only the right client LPAR connects to the right virtual SCSI disk. This is a safety precaution but still worth doing.

See Figure 14 and 15 to check they are right.

So on the HMC highlight -- the VIO Server LPAR profile -- bring up its properties. In the Virtual I/O tab, select each Server SCSI resource, select the Properties Button, and set the:

• Only selected remote partition and slot connect option.
• Correct remote partition name.
• Correct remote partition virtual slot number.

Remember in this example, you have two virtual SCSI adapters on the VIO Server to "clean up".This can be confusing and very easy to get wrong. So, you should plan in advance (see Table 1).

Step 10. Preparing VIO Server for clients

You now have all the connections for the VIO Server and virtual clients, but still have to connect to the:

• Virtual SCSI disk to a piece of real disk space.
• Virtual and real Ethernets using the bridge utilities.

This is done on the VIO Server only as follows:

Step 10.1. Virtual Ethernet

Once the VIO Server is running and no network is set up (if it's, then stop the network before continuing further with: ifconfig eth0 0.0.0.0 ), then do the following:

1. Give your machine a hostname: hostname <mymachine>.
2. Install the IBM virtual device driver: modprobe ibmveth.
3. Check that it's running: lsmod | grep ibmveth.
4. Install the bridge device driver: modprobe bridge.
5. Check that it's running: lsmod | grep bridge.
6. Create the bridge: brctl addbr br0.
7. Add the real Ethernet interface: brctl addif br0 eth0.
8. Add the Virtual Ethernet interface: brctl addif br0 eth1.
9. Let the bridge settle down: sleep 1.

Note: If your real Ethernet adapter has, for example two ports, it can be confusing. In this case, the Virtual Ethernet name might be eth2, as eth1 can be the second port on the real adapter. The bridge software needs little time to logically connect the networks. If you are typing the commands, this is not a problem. If it's a script, the sleep command gives it the time before continuing.

You now have a network called br0. The configuration of this network is just like any other network. It sets up the real and virtual adapters all in one go.

ifconfig br0 9.137.62.170 netmask 255.255.255.0
sleep 1

The bridge software needs little time to settle down and explore the network topology. If you are typing the commands, this is not a problem. If it's a script, the sleep command gives the bridge software some time before continuing.

The following might not be required, but I found that in some cases the networks were not actually started, and it does no harm.

ifconfig br0 up
ifconfig eth0 up
ifconfig eth1 up

You will probably need a gateway to:

route add default gw 9.137.62.1

Explaining the device driver directory
To configure the partition, you have to get to the right directory and change three files (type, device, and active). The directory is: /sys/devices/vio/XXXXXXXX/bus0/target0.

Where XXXXXXXX is the virtual SCSI adapter slot number plus 30000000.

• Slot three is 30000003.
• Slot four is 30000004.
• Slot 42 is 30000042.

This directory contains the following files:

• Active -- write one for active or zero for not active (do this last).
• Type -- write b for binary the only option currently.
• Device -- write the name of the disk device that you wish to associate with the virtual SCSI adapter. As in /dev/sdc or /dev/sdc2 or /dev/loop4.

Step 10.2. Virtual SCSI using fdisk- type real disk partition for Client A

Once the VIO Server is running and before the virtual clients are started:

Initial setup

Actually, create the disk partition that the client uses. Linux has many ways to create a partition on the disk. All you need to do is:

• Create the partition.
• Know its name.
• Make it big enough.

A four GB as a minimum is recommended -- This allows Linux, the boot partition, some swap space, and at least one GB of spare disk space to install an application and data. If you have more disk space available, you can allocate a much larger size. You can also allocate more than one virtual SCSI disk in a single client LPAR.

Use the fdisk tool to create a partition on a spare disk, but you might want to user the YaST or DiskDruid tools to create mirrored or software RAID partitions. I'm assuming you know how to do this, but please be aware it's a high-risk operation, as getting this wrong can destroy disk contents or worse.

The fdisk other partition management commands and tools can cause damage to your running system, operating system, filesystems, and files and data. These commands are not covered here. You need to check the manual pages, other documentation, and how to files. Using them strictly is at your own risk.

To list your partitions and double check the name and size, you can use:

fdisk -l

.

Note: The fdisk option is lowercase L.

In this example, the device created is /dev/sdb6.

An alternative is that you allocate a whole unused disk to your LPAR. To do this, simply use the whole disk name instead of the partitions name. For example, /dev/sdc for the third whole disk (this is what the c means).

Initial setup and each time the VIO Server is rebooted

Install the IBM virtual SCSI server device driver: modprobe ibmvscsis.

Check that it's running: lsmod | grep ibmvscsis.

The VIO Server profile (on the HMC) includes the virtual SCSI adapter for Client B, and its slot number was three. This "three" is vital to get right. It's the Server slot number -- not the slot number in the client. For example, the file name below uses 300000003.

As the root user:

• Switch off the device in case it's active.

  # cd /sys/devices/vio/30000003/bus0/target0 # echo 0 >active

  
  
• Program it with new values:

  # cd /sys/devices/vio/30000003/bus0/target0 # echo /dev/sdb6 >$VIO/device # echo b >type # echo 1 >active

  
  

It would be best to script these actions so they are performed correctly every time. There is an example script in Listing 1 below.

Step 10.3 Virtual SCSI using loopback and filesystem file for client B

Once the VIO Server is running and before the virtual clients are started:

Initial setup

Create the file that the client uses.

You can create the "disk" file, assuming you want to do this in the directory or filesystem /clients and you have 4 GB free space in the filesystem.

As root:

Listing 1. Example script

# cd /clients # Use the command bc to work out one million # bc 1024*1024 1048576 Control-D # dd if=/dev/zero of=/clients/B bs=4096 count=1048576 # ls -l /clients/B

Next, check the length of the file with ls -l. It should be 4194304 bytes.

Initial setup and each time the VIO Server is rebooted

Install the IBM virtual SCSI server device driver: modprobe ibmvscsis

Check that it's running: lsmod | grep ibmvscsis

The VIO Server profile (on the HMC) includes the virtual SCSI adapter for Client B, and its slot number was four. This "four" is vital to get right. It's the server slot number -- not the slot number in the client. For example, the file name below uses 300000004.

As the root user:

• Switch off the device (just in case it's active).

  # cd /sys/devices/vio/30000004/bus0/target0 # echo 0 >active

  
  
• Now program it with new values.

  # cd /sys/devices/vio/30000004/bus0/target0 # now program it with new values losetup /dev/loop0 /clients/B echo /dev/loop0 >device echo b >type echo 1 >active

  
  

You can check the details of the loopback command (losetup) in the manual pages.

Note that you have to use a different loopback resource name (such as loop0, loop1, loop2, and so forth) for each virtual SCSI adapter.

It would be best to script these actions so they are performed correctly every time. There is an example script in Listing 2 below.

Step 11. VIO client LPAR installations

Now you can start up your virtual I/O clients and install them. This can be SUSES SLES 9, Red Hat 3 update 3 onwards, or Debian. They should find both the:

• Virtual Ethernet -- It will be named a Virtual Ethernet much like a real Ethernet is given a name, such as the "Intel 100 pro" adapter.
• Virtual SCSI disk -- It's presented just like a SCSI disk ,but it will only be the size of the underlying disk partition or file.

These should install just like a regular real Ethernet and SCSI disk.

Once running, the Virtual Ethernet looks and behaves like a very fast one GB real adapter.


Listing 2. Example script

                    
clienta:~ # ifconfig 
eth0      Link encap:Ethernet  HWaddr AE:38:00:00:D0:02
          inet addr:9.137.62.178  Bcast:9.137.62.255  Mask:255.255.255.0
          inet6 addr: fe80::ac38:ff:fe00:d002/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1075 errors:0 dropped:0 overruns:0 frame:0
          TX packets:350 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:113566 (110.9 Kb)  TX bytes:40940 (39.9 Kb)
          Interrupt:184

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:10 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:641 (641.0 b)  TX bytes:641 (641.0 b)

Once running, you can see how the virtual SCSI disk is being treated just like a regular disk.

clienta:~ # fdisk -l

Disk /dev/sda: 4194 MB, 4194304000 bytes
130 heads, 62 sectors/track, 1016 cylinders
Units = cylinders of 8060 * 512 = 4126720 bytes
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1           1        3999   41  PPC PReP Boot
/dev/sda2               6         132      511810   82  Linux swap
/dev/sda3             133        1016     3562520   83  Linux

Network installation might be trickier, as you might have to activate the device drivers for the installation tools to find the virtual adapters. Note that network installations are not covered in this article.

Depending on which release you use, the installer might give you a series of menus and options to install the IBM virtual SCSI client and Virtual Ethernet drivers before you begin installation. Later releases fully understand and install the device drivers for these virtual resources without manual intervention.

Step 12. Backing up a VIO Server and VIO client

Once you create your client LPAR and set it up the way you like, you should consider backing up the operating system images. Backups are a large subject and many books have been written in this area. There are many backup solutions, both commercial applications and freely available tools in the Linux world. One of the popular freely available tools is AMANDA (Advanced Maryland Automatic Network Disk Archiver). This tool provides remote backup with disk caching and tape library management, and it's worth a look. Also, there is a "Linux Backup and Recovery How To" on the Internet for more information.

This article only covers the special considerations for VIO Servers and clients. Backups are important for the following reasons:

• Recovery of files that are accidentally removed.
• Disk failure -- Assuming your disks are not already protected with a mirror or RAID 5, or you are very unlucky and loose more than one disk.
• Recreating the entire system for a disaster (total machine loss) from backups held off-site.

• Definitions of the LPAR physical resources such as CPU, memory, and PCI slots.
• Definitions of the LPAR virtual resources such as the connections between VIO Server and clients.

If the HMC fails, the data is still held in the service processor and can be read by a replacement or recovered HMC. It's vita that the configuration details are available in case of a disaster. HMC backups are documented in the manuals, the InfoCenter help files and IBM Redbooks. it's also recommended that details of the LPARs are documented on paper. For example, something similar to the planning sheet used to create the LPARs in this article.

1. Back up the details of the client physical disk partitions or the loopback files and filesystem layout details. The details include number partitions/files, their size, and disk layout.
2. Back up the initialisation scripts, which includes the disk links and Ethernet details.
3. Back up the contents of the client physical disk partitions or loopback files.

To recover the VIO Server, you can simply re-install SUSE SLES 9 and recover using the above.

If you are using the VIO Server for other purposes in addition to VIO, you need to also back up the VIO Server system itself, just as you would any Linux system. This might mean a back up to a local or remote tape drive, over NFS to another server, or using one of the automated backup services in a client server mode.

There are different approaches to backing up the VIO client images from the VIO Server. First, you have the option of doing hot or cold backups:

• Hot backup is while the VIO client is running. This is not recommended.
• Cold backup is the only sensible way to back up from the VIO Server. This is simply a matter of shutting down the VIO client -- as root on the client: shutdown -fh now.
  • For the disk partitions method, you have to use the dd command to copy the disk partition images to a file.
  • For the loopback adapter method, you have to create a copy of the file.
    Note: the cp command is not a good idea, as it copies a file using small blocks. This is very inefficient and slow. A better command is dd , with a large block size. For example, 64 KB blocks use the bs=64k option.
  • Loopback example: dd bs=64k if=/clients/B of=/backup/B
  • Disk partition example: dd bs=64k if=/dev/sdb6 of=/backup/B
    Alternatively, you can back up straight to a tape drive using a command like tar or backup. Some machines support a writeable DVD device that can also be used as a backup medium.
    As SUSE SLES 9 can perform DLPAR changes of PCI slots, a single tape drive and its associated SCSI adapter can be moved to the VIO Server for the backup period and then removed (so it can be used in other LPARs).
    Recovery of a VIO client involves getting the disk image back in the right place and starting the VIO client LPAR again.

It's unlikely that VIO client LPARs will have a tape, as the purpose of a VIO client, to share physical resources to reduce hardware requirements. As with the VIO Server, you can use DLPAR changes of PCI slots to temporarily introduce a tape driver to the client and back up its own data. Automating this process can be hard to coordinate between multiple client LPARs, but it can be done using scripts on a central machine. Some machines support a writeable DVD device that can also be used as a backup medium.

A second option is for the client to use another LPAR (possibly the VIO Server) or another machine to save the data using either a:

• Remote tape drive -- You can find lots of information on how to do this and make it secure on the Internet. You might need to check the speed of this mechanism and use a Linux tool called "buffer" to ensure the tape drive streams data onto the tape drive efficiently.
• NFS server -- You can use this server to temporarily store the backup data before it's backed up to tape.
• Remote backup application -- See opening notes at the top of this section, which uses a local client application to transfer data to the server machine to provide a backup service.

In all three cases, the high speed of the Virtual Ethernet can boost backup performance. Recovery using these methods can be harder work. You might have to re-install from the original CD-ROMs and then overwrite the running Linux with the backup.

Step 13. Cloning a client

Another option is to use the client SCSI disk backup as the virtual SCSI disk for a new VIO client LPAR. For the two types of client disk, perform one of the following:

• For the loopback adapter SCSI disk, just use the dd command to create a new copy of the file and use the second copy plus a different loopback resource name (such as loop1, loop2, and so forth). For the disk partitions SCSI disk, you need to create an identical disk partition size and then make a copy from the original to the new partition using the dd command.

You need to make sure the cloned LPAR is not using the same Ethernet IP addresses as the original. Alternatively, you can clone the original LPAR before putting it on the network.

Step 14. DLPARs and RAS

This is not really part of the VIO Server, but it's important for reliability, availability, and serviceability (RAS). After installing SUSE SLES 9, it's strongly recommended that you install the IBM packages for both DLPARs (this works for physical and virtual resources) and the daemons and tools to increase RAS. This ensures that you get the expected reliability from your pSeries p5 or OpenPower machine. These RPMs put the LPAR in touch with the HMC for dynamic changes, reporting problems, and tools to use on the LPAR.

You can download these tools from the following Web site: http://techsupport.services.ibm.com/server/lopdiags.

On this Web site, take the "SUSE Linux" tab and then the "HMC-managed" option to find the list. Here's the list of RPMs:

• src-1.2.2.1-0.ppc.rpm
• rsct.core.utils-2.3.4.2-0.ppc.rpm
• rsct.core-2.3.4.2-0.ppc.rpm
• csm.core-1.4.0.3-79.ppc.rpm
• csm.client-1.4.0.3-79.ppc.rpm
• devices.chrp.base.ServiceRM-2.2.0.0-1.ppc.rpm
• DynamicRM-1.1-2.ppc.rpm
• rpa-dlpar-1.0-11.ppc.rpm
• rpa-pci-hotplug-1.0-8.ppc.rpm
• librtas-1.1-12.ppc64.rpm

Note: The pre-requisite is the rdist command that is on the Linux SUSE SLES 9 CD3 and file name rdist-6.1.5-792.1.ppc.rpm.

Download these RPMs and install them. You should be able to do DLPAR changes and report problems to the HMC with Linux, which is used to forward on problems to IBM (if set up) and used by hardware maintenance staff for diagnosis and correction.

Back to top

My simple VIO script

Listing 3 below is the script that you can use on an Open Power machine. Automatically run the LPAR at start time using the /etc/rc scripts. I like to run it manually each time because I experiment a lot, and like to start the LPAR with nothing running. The addition of new LPARs becomes just a matter of adding lines at the bottom of the script.

                

set_loop()
{
	echo Setup Virtual SCSI $1 to Loopback $2 on file $3
	export VIO=/sys/devices/vio/$1/bus0/target0
	# first switch if off in case it's active
	echo 0  >$VIO/active
	losetup $2 $3
	echo $2 >$VIO/device
	echo b  >$VIO/type
	echo 1  >$VIO/active
}

set_scsi()
{
	echo Setup Virtual SCSI $1 to disks $2
	export VIO=/sys/devices/vio/$1/bus0/target0
	# first switch if off in case it's active
	echo 0  >$VIO/active
	echo $2 >$VIO/device
	echo b  >$VIO/type
	echo 1  >$VIO/active
}

IF1=eth0
IF2=eth2

stop_bridge()
{
echo Stop Bridge on $IF1 and $IF2
ifconfig br0 down
ifconfig $IF1 down
ifconfig $IF2 down
brctl delif br0 $IF1
brctl delif br0 $IF2
brctl delbr br0
}

set_bridge()
{
echo Create bridged network
brctl addbr br0
brctl addif br0 $IF1
brctl addif br0 $IF2
sleep 1
echo Configure bridged network
hostname 
ifconfig br0 9.137.62.170 netmask 255.255.255.0
sleep 1
ifconfig br0 up
ifconfig $IF1 up
ifconfig $IF2 up
echo Configure default gateway
route add default gw 9.137.62.1
}
echo Add Kernel modules ibmveth and bridge and SCSI Server
modprobe ibmveth
modprobe bridge
modprobe ibmvscsis
echo Check the drives are running
lsmod | grep ibmveth
lsmod | grep bridge
lsmod | grep ibmvscsis

set_bridge
set_loop 30000003 /dev/loop0 /client/A
set_loop 30000004 /dev/loop1 /client/B
set_loop 30000005 /dev/loop2 /client/C
set_scsi 30000007 /dev/sdb6 
set_scsi 30000008 /dev/sdb7 

Resources

• The article "Setting up your IBM eServer OpenPower server" (developerWorks, January 2005) provides a fastpath approach for setting up an OpenPower server.
  
  
• The article "Linux for pSeries installation and administration (SLES 9)" (developerWorks, November 2004) covers the installation of Linux on an IBM pSeries system.
  
  
• See the IBM Linux on POWER Facts and Features for supported hardware details.
  
  
• The pSeries p5, OpenPower and JS20 Facts and Features provides a quick reference for IBM UNIX systems.
  
  
• The IBM eServer pSeries Linux on pSeries Overview whitepaper provides an introduce to Linux on pSeries.
  
  
• Linux on POWER applications provides a full list of Linux on Power application.
  
  
• For Linux developers looking to get started on Power Architecture, you'll find a collection of articles and resources on the Linux on Power Architecture Developer's corner.
  
  
• The IBM Redbook Advanced POWER Virtualisation on IBM eServer p5 Servers: Introduction and Basic Configuration provides an overview for system Administrators.
  
  
• IBM eServer p5 570 Server Consolidation Using POWER5 Virtualisation white paper describes how Micro-Partitioning can be employed on IBM p5 570 systems for server consolidation.
  
  
• The IBM eServer p5 570 Workload Balancing Using POWER5 Virtualisation white paper describes how Micro-Partitioning can be employed on the IBM eServer p5 570 with AIX 5L V5.3 to set up flexible partitioned configurations for dynamic workload balancing of available CPU resources.
  
  
• See Using the VIO Server infocenter for conceptual, procedural, and reference information.
  
  
• Browse the VIO Server and Partition Load Manager Commands Reference guide.
  
  
• Eclipse In Action: A Guide for Java Developers (Independent Pub Group, 2003) is a must-read for Java developers using Eclipse.
  
  
• Visit the Virtual Innovation Center for Hardware for AIX development support. This is the primary source for all pSeries AIX development.
  
  
• Browse for books on these and other technical topics.
  
  
• Want more? The developerWorks eServer zone hosts hundreds of informative articles and introductory, intermediate, and advanced tutorials on the eServer brand.
  
  
• Get involved in the developerWorks community by participating in developerWorks blogs.
  
  
• The IBM developerWorks team hosts hundreds of technical briefings around the world which you can attend at no charge.
  

About the author

		Nigel Griffiths works in the IBM eServer pSeries Technical Support Advanced Technology Group. He specialises in performance, sizing, tools, benchmarks, and Oracle RDBMS. The nmon tool was developed to support benchmarks and performance tuning for internal use, but by popular demand is given away to deserving friends. You can contact Nigel at nag@uk.ibm.com

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top