This blog is for the open exchange of ideas relating to storage and storage networking hardware, software and services. Tony Pearson is a Senior Storage Consultant for the IBM System Storage product line, has been working in IBM storage for over 20 years, and is author of the blook Inside System Storage: Volume I.

Off to Japan and India

For those looking for something to do to enjoy the "4th of July" US Independence day holiday tomorrow, there is the [Team America: Sing-a-long at Tucson's Loft Cinema at 6pm, you can still see the fireworks after the show is over. I did this last year and it was a lot of fun.

Also, you can check out the IBM Wimbledon build on Second Life. Here's the SLURL: [http://slurl.com/secondlife/IBM%207/133/180/23]. Several IBMers will be "in world" at this virtual location on 4th of July. For all of my readers looking to check out Second Life, see what IBM can do, or talk to people who are familiar with this technology, here's your chance.

As for me, I'll be spending my "long weekend" in an airplane. Here's my travel schedule.

• July 7-11: Tokyo, Japan - business meetings with IBM sales reps
• July 13-18: Mumbai, India - business meetings with IBM business partners
• July 24-27: San Diego, California - [Comic-Con] and [Octagon Global Recruiting] -- my thanks to ABC, the producers of the TV show "LOST", and the [Dharma Initiative] for the invitation!

(I was hoping that while I was in Asia, I could stop over and visit the schools I helped in Nepal and my friends at the Open Learning Exchange [OLE Nepal] as part of the One Laptop Per Child [OLPC Nepal] program, but I did not get all my ducks lined up for this with the appropriate travel approvals, visas and logistics. My apologies to Bryan, Sulochan and the rest of the team. Perhaps next year!)

Enjoy your weekend!

technorati tags: IBM, Tokyo, Japan, Mumbai, India, Wimbledon, SecondLife, OLE Nepal, OLPC, ABC, LOST, Dharma Initiative, Octagon Global Recruiting

Sending my baby off to school

(Earlier this year, I build a test XS School Server that was used to help and support [OLPC Nepal] by working with their local NGO team [OLE Nepal]. I wrote about this back in February in my post [Understanding the LAMP platform for Web 2.0 workloads].)

Based on this success, and perhaps because I am also fluent in Spanish, I was asked to help with Proyecto Ceibal, the team for OLPC Uruguay. Normally the XS school server resides at the school location itself, so that even if the internet connection is disrupted or limited, the school kids can continue to access each other and the web cache content until internet connection is resumed. However, with a diverse development team with people in United States, Uruguay, and India, we first looked to Linux hosting providers that would agree to provide free or low-cost monthly access. We spent (make that "wasted") the month of May investigating. Most that I talked to were not interested in having a customized Linux kernel on non-standard hardware on their shop floor, and wanted instead to offer their own standard Linux build on existing standard servers, managed by their own system administrators, or were not interested in providing it for free. Since the XS-163 kernel is customized for the x86 architecture, it is one of those exceptions where we could not host it on an IBM POWER or mainframe as a virtual guest.

This got picked up as an [idea] for the Google's [Summer of Code] and we are mentoring Tarun, a 19-year-old student to act as lead software developer. However, summer was fast approaching, and we wanted this ready for the next semester. In June, our project leader, Greg, came up with a new plan. Build a machine and have it connected at an internet service provider that would cover the cost of bandwidth, and be willing to accept this with remote administration. We found a volunteer organization to cover this -- Thank you Glen and Vicki!

We found a location, so the request to me sounded simple enough: put together a PC from commodity parts that meet the requirements of the customized Linux kernel, the latest release being called [XS-163]. The server would have two disk drives, three Ethernet ports, and 2GB of memory; and be installed with the customized XS-163 software, SSHD for remote administration, Apache web server, PostgreSQL database and PHP programming language. Of course, the team wanted this for as little cost as possible, and for me to document the process, so that it could be repeated elsewhere. Some stretch goals included having a dual-boot with Debian 4.0 Etch Linux for development/test purposes, an alternative database such as MySQL for testing, a backup procedure, and a Recover-DVD in case something goes wrong.

Some interesting things happened:

1. The XS-163 is shipped as an ISO file representing a LiveCD bootable Linux that will wipe your system clean and lay down the exact customized software for a one-drive, three-Ethernet-port server. Since it is based on Red Hat's Fedora 7 Linux base, I found it helpful to install that instead, and experiment moving sections of code over. This is similar to geneticists extracting the DNA from the cell of a pit bull and putting it into the cell for a poodle. I would not recommend this for anyone not familiar with Linux.

   I also experimented with modifying the pre-built XS-163 CD image by cracking open the squashfs, hacking the contents, and then putting it back together and burning a new CD. This provided some interesting insight, but in the end was able to do it all from the standard XS-163 image.

2. Once I figured out the appropriate "scaffolding" required, I managed to proceed quickly, with running versions of XS-163, plain vanilla Fedora 7, and Debian 4, in a multi-boot configuration.

3. The BIOS "raid" capability was really more like BIOS-assisted RAID for Windows operating system drivers. This "fake raid" wasn't supported by Linux, so I used Linux's built-in "software raid" instead, which allowed some partitions to be raid-mirrored, and other partitions to be un-mirrored. Why not mirror everything? With two 160GB SATA drives, you have three choices:
   • No RAID, for a total space of 320GB
   • RAID everything, for a total space of 160GB
   • Tiered information infrastructure, use RAID for some partitions, but not all.

   The last approach made sense, as a lot of of the data is cache web page images, and is easily retrievable from the internet. This also allowed to have some "scratch space" for downloading large files and so on. For example, 90GB mirrored that contained the OS images, settings and critical applications, and 70GB on each drive for scratch and web cache, results in a total of 230GB of disk space, which is 43 percent improvement over an all-RAID solution.

4. While [Linux LVM2] provides software-based "storage virtualization" similar to the hardware-based IBM System Storage SAN Volume Controller (SVC), it was a bad idea putting different "root" directories of my many OS images on there. With Linux, as with most operating systems, it expects things to be in the same place where it last shutdown, but in a multi-boot environment, you might boot the first OS, move things around, and then when you try to boot second OS, it doesn't work anymore, or corrupts what it does find, or hangs with a "kernel panic". In the end, I decided to use RAID non-LVM partitions for the root directories, and only use LVM2 for data that is not needed at boot time.

5. While they are both Linux, Debian and Fedora were different enough to cause me headaches. Settings were different, parameters were different, file directories were different. Not quite as religious as MacOS-versus-Windows, but you get the picture.

6. During this time, the facility was out getting a domain name, IP address, subnet mask and so on, so I tested with my internal 192.168.x.y and figured I would change this to whatever it should be the day I shipped the unit. (I'll find out next week if that was the right approach!)

7. Afraid that something might go wrong while I am in Tokyo, Japan next week (July 7-11), or Mumbai, India the following week (July 14-18), I added a Secure Shell [SSH] daemon that runs automatically at boot time. This involves putting the public key on the server, and each remote admin has their own private key on their own client machine. I know all about public/private key pairs, as IBM is a leader in encryption technology, and was the first to deliver built-in encryption with the IBM System Storage TS1120 tape drive.

8. To have users have access to all their files from any OS image required that I either (a) have identical copies everywhere, or (b) have a shared partition. The latter turned out to be the best choice, with an LVM2 logical volume for "/home" directory that is shared among all of the OS images. As we develop the application, we might find other directories that make sense to share as well.

9. For developing across platforms, I wanted the Ethernet devices (eth0, eth1, and so on) match the actual ports they are supposed to be connected to in a static IP configuration. Most people use DHCP so it doesn't matter, but the XS software requires this, so it did. For example, "eth0" as the 1 Gbps port to the WAN, and "eth1/eth2" as the two 10/100 Mbps PCI NIC cards to other servers. Naming the internet interfaces to specific hardware ports was different on Fedora and Debian, but I got it working.

10. While it was a stretch goal to develop a backup method, one that could perform Bare Machine Recovery from media burned by the DVD, it turned out I needed to do this anyways just to prevent me from losing my work in case things went wrong. I used an external USB drive to develop the process, and got everything to fit onto a single 4GB DVD. Using IBM Tivoli Storage Manager (TSM) for this seemed overkill, and [Mondo Rescue] didn't handle LVM2+RAID as well as I wanted, so I chose [partimage] instead, which backs up each primary partition, mirrored partition, or LVM2 logical volume, keeping all the time stamps, ownerships, and symbolic links in tact. It has the ability to chop up the output into fixed sized pieces, which is helpful if you are going to burn them on 700MB CDs or 4.7GB DVDs. In my case, my FAT32-formatted external USB disk drive can't handle files bigger than 2GB, so this feature was helpful for that as well. I standardized to 660 GiB [about 692GB] per piece, since that met all criteria.

    (The mainframe equivalent is DFSMShsm or DFSMSdss DUMP, which by the way can be used with Linux for System z DASD CKD partitions. See this helpful [ HOWTO back up your Linux partitions and volumes through z/OS] guide.)

11. The folks at [SysRescCD] saved the day. The standard "SysRescueCD" assigned eth0, eth1, and eth2 differently than the three base OS images, but the nice folks in France that write SysRescCD created a customized [kernel parameter that allowed the assignments to be fixed per MAC address ] in support of this project. With this in place, I was able to make a live Boot-CD that brings up SSH, with all the users, passwords, and Ethernet devices to match the hardware. Install this LiveCD as the "Rescue Image" on the hard disk itself, and also made a Recovery-DVD that boots up just like the Boot-CD, but contains the 4GB of backup files.

    For testing, I used Linux's built-in Kernel-based Virtual Machine [KVM] which works like VMware, but is open source and included into the 2.6.20 kernels that I am using. IBM is the leading reseller of Vmware and has been doing server virtualization for the past 40 years, so I am comfortable with the technology. The XS-163 platform with Apache and PostgreSQL servers as a platform for [Moodle], an open source class management system, and the combination is memory-intensive enough that I did not want to incur the overheads running production this manner, but it was great for testing!

With all this in place, it is designed to not need a Linux system admin or XS-163/Moodle expert at the facility. Instead, all we need is someone to insert the Boot-CD or Recover-DVD and reboot the system if needed.

Just before packing up the unit for shipment, I changed the IP addresses to the values they need at the destination facility, updated the [GRUB boot loader] default, and made a final backup which burned the Recover-DVD. Hopefully, it works by just turning on the unit, [headless], without any keyboard, monitor or configuration required. Fingers crossed!

So, thanks to the rest of my team: Greg, Glen, Vicki, Tarun, Marcel, Pablo and Said. I am very excited to be part of this, and look forward to seeing this become something remarkable!

technorati tags: XS School Server, Proyecto, Ceibal, OLPC, Uruguay, OLE, Nepal, LAMP, Web2.0, Google, Summer of Code, SSH, sshd, Apache, PostgreSQL, PHP, Red Hat, Fedora, Debian, Linux, Ethernet, BIOS, RAID, fakeraid, LiveCD, Boot-CD, Recover-DVD, DFSMS, DFSMShsm, DFSMSdss, DUMP, mainframe, LVM2, SVC, TSM, GRUB, Mondo Rescue, partimage, SysRescCD, KVM, GRUB

Summer Reading for z10 EC server virtualization

My colleagues in IBM Poughkeepsie recommended these books to provide more insight and in-depth understanding. Looks like some interesting summer reading. I put in quotes the sections I excerpted from the synopsis I found for each.

[In Search of Clusters] by Gregory F. Pfister "From Microsoft to IBM, Compaq to Sun to DEC, virtually every large computer company now uses clustering as a key strategy for high-availability, high-performance computing. This book tells you why-and how. It cuts through the marketing hype and techno-religious wars surrounding parallel processing, delivering the practical information you need to purchase, market, plan or design servers and other high-performance computing systems. Microsoft Cluster Services ("Wolfpack") IBM Parallel Sysplex and SP systems DEC OpenVMS Cluster and Memory Channel Tandem ServerNet and Himalaya Intel Virtual Interface Architecture Symmetric Multiprocessors (SMPs) and NUMA systems"

[Guerilla Capacity Planning] by Neil J. Gunther "In these days of shortened fiscal horizons and contracted time-to-market schedules, traditional approaches to capacity planning are often seen by management as tending to inflate their production schedules. Rather than giving up in the face of this kind of relentless pressure to get things done faster, Guerrilla Capacity Planning facilitates rapid forecasting of capacity requirements based on the opportunistic use of whatever performance data and tools are available in such a way that management insight is expanded but their schedules are not."

Neil Gunther points out that vendor claims of near linear scaling are not to be trusted and shows a method to “derate” scaling claims. His suggested scaling values for data base servers is closer IBM's LSPR-like scaling model, than TPC-C or SPEC scaling. I had mentioned that "While a 1-way z10 EC can handle 920 MIPS, the 64-way can only handle 30,657 MIPS." in my post, but still people felt I was using "linear scaling". Linear scaling would mean that if a 1Ghz single-core AMD Opteron can do four(4) MIPS, and an one-way z10 EC can do 920 MIPS, than one might assume that 1GHz dual-core AMD could do eight(8) MIPS, and the largest 64-way z10 EC can do theoretically 64 x 920 = 58,880 MIPS. The reality is closer to 6.866 and 30,657 MIPS, respectively.

This was never an IBM-vs-Sun debate. One could easily make the same argument that a large Sun or HP system could replace a bunch of small 2-way x86 servers from Dell. Both types of servers have their place and purpose, and IBM sells both to meet the different needs of our clients. The savings are in total cost of ownership, reducing power and cooling costs, floorspace, software licenses, administration costs, and outages.

I hope we covered enough information so that Jeff can go back about talking about Sun products, and I can go back to talk about IBM storage products.

technorati tags: IBM, z10, EC, Jon Toigo, DrunkenData, Jeff Savit, Sun, Gregory Pfister, Austin, Neil Gunther, MIPS, LSPR, SPEC, TPC-C, mainframe, scalability, UNIX, Linux, z/OS, HP, Dell

Virtualization, Carpools and Marathons

One car, one driver

If you wonder why so many companies subscribe to the notion that you should only run a single application per server, blame Sun, who I think helped promote this idea. Not to be out-done, Microsoft, HP and Dell think that it is a great idea too. Imagine the convenience for operators to be able to switch off a single machine and impact only a single application. Imagine how much this simplifies new application development, knowing that you are the only workload on a set of dedicated resources.

This is analogous to a single car, single driver, where the car helps get the person from "point A" to "point B" and the single driver represents the driver and sole passenger of the vehicle. If this were a single driver on a energy-efficient motorcycle or scooter, than would be reasonable, but people often drive alone much bigger vehicles, what Jeff Savit would call "over-provisioning". Chips have increased in processing power much faster than individual applications have increased their requirements, so as a result, you have over-provisioning.

Carpooling - one bus, one driver, and many other passengers riding along

This is how z/OS operates. Yes, you could have up to 60 LPARs that you could individually turn on and off, but where z/OS gets most of its advantages is that you can run many applications in a single OS instance, through the use of "Address Spaces" which act as application containers. Of course, it is more difficult to write for this environment, because you have to be a good "z/OS citizen", share resources nicely, and be WLM-compliant to allow your application to be swapped out for others.

While you get efficiencies with this approach, when you bring the OS down, all the apps on that OS image have to stop with it. For those who have "Parallel Sysplex" that is not an issue. For example, let's say you have three mainframes, each running several LPARs of z/OS, and your various z/OS images all are able to process incoming transactions for a common shared DB2 database. Thanks to DB2 sharing technology, you could take down an individual LPAR or z/OS image, and not disrupt transaction processing, because the IP spreader just sends them to the remaining LPARs. A "Coupling Facility" allows for smooth operations if any of the OS images are lost from an unexpected disaster or disruption.

Needless to say, IBM does not give each z/OS developer his or her own mainframe. Instead, we get to run z/OS guest images under z/VM. It was even possible to emulate the next generation S/390 chipset to allow us to test software on hardware that hasn't been created yet. With HiperSockets, we can have virtual TCP/IP LAN connections between images, have virtual coupling facilities, have virtual disk and virtual tape, and so on. It made development and test that much more efficient, which is why z/OS is recognized as one of the most rock-solid bullet-proof operating systems in existence.

The negatives of carpooling or taking the bus applies here as well. I have been on buses that have stopped working, and 50 people are stranded. And you don't need more than two people to make the logistics of most carpools complicated. This feeds the fear that people want to have separate manageable units one-car-one-driver than putting all of their eggs into one basket, having to schedule outages together, and so on.

(Disclaimer: From 1986 to 2001 I helped the development of z/OS and Linux on System z. Most of my 17 patents are from that time of my career!)

Bicycle races and Marathons

The third computing model is the Supercomputer. Here we take a lot of one-way and two-way machines, and lash them together to form an incredible machine able to perform mathematical computations faster than any mainframe. The supercomputer that IBM built for Los Alamos National Laboratory just clocked in at 1,000,000,000,000,000 floating point operations per second. This is not a single operating system, but rather each machine runs its own OS, is given its primary objective, and tries to get it done. NetworkWorld has a nice article on this titled: [IBM, Los Alamos smash petaflop barrier, triple supercomputer speed record]. If every person in the world was armed with a handheld calculator and performed one calculation per second, it would take us 46 years collectively to do everything this supercomputer can do in one day.

I originally thought of bicycle races as an analogy for this, but having listened to Lance Armstrong at the [IBM Pulse 2008] conference, I learned that biking is a team sport, and I wanted something that had the "every-man-for-himself" approach to computing. So, I changed this to marathons.

The marathon was named after a fabled greek soldier was sent as messenger from the [Battle of Marathon to the City of Athens], a distance that is now standardized to 26 miles and 385 yards, or 42.195 kilometers for my readers outside the United States.

If you were given the task to get thousands of people from "point A" to "point B" 26 plus miles away, would you chose thousands of cars, each with a lone driver? Conferences with a lot of people in a few hotels use shuttle buses instead. A few drivers, a few buses, and you can get thousands of people from a few places to a few places. But the workloads that are sent to supercomputers have a single end point, so a dispatcher node gives a message to each "greek soldier" compute node, and has them run it on their own. Some make it, some don't, but for a supercomputer that is OK. When the message is delivered, the calculation for that little piece is done, and the compute node gives it another message to process. All of the computations are assembled to come up with the final result. Applications must be coded very specially to be able to handle this approach, but for the ones that are, amazing things happen.

So, how does "server virtualization" come into play?

Logical Partitions

IBM has had Logical Partitions for quite some time. A logical partition, or LPAR, can run its own OS image, and can be turned on and off without impacting other LPARs. LPARs can have dedicated resources, or shared resources with other LPARs. The IBM z10 EC can have up to 60 LPARs. System p and System i, now merged into the new "POWER Systems" product line, also support LPARs in this manner. Depending on the size of your LPAR, this could be for a single OS and application, or a single OS with lots of applications.

Address Spaces/Application Containers

This is the bus approach. You have a single OS, and that is shared by a set of application containers. z/OS does this with address spaces, all running under a single z/OS image, and for x86 there are products like [Parallels Virtuozzo Containers] that can run hundred of Windows instances under a single Windows OS image, or a hundred Linux images under a single Linux OS image. However, you cannot mix and match Windows with Linux, just as all the address spaces on z/OS all have to be coded for the same z/OS level on the LPAR they run in.

Virtual Guests

The term "guests" were chosen to model this after the way hotels are organized. Each guest has a room with its own lockable entrance and privacy, but shared lobby, and in some countries, shared bathrooms on every hall. This approach is used by z/VM, VMware and others. The z/VM operating system can handle any S/390-chip operating system guest, so you could have a mix of z/OS, TPF, z/VSE, Linux and OpenSolaris, and even other z/VM levels running as guests. Many z/VM developers run in this "second level" mode to develop new versions of the z/VM operating system!

As part of the One Laptop Per Child [OLPC] development team (yes, I am a member of their open source community, and now have developer keys to provide contributions), I have been experimenting with Linux KVM. This was [folded into the base Linux 2.6.20 kernel and available to run Linux and Windows guest images. This is a nice write-up on [Wikipedia].

The key advantage of this approach is that you are back to one-car-one-driver simplistic mode of thinking. Each guest can be turned on and off without impacting other applications. Each guest has its own OS image, so you can mix different OS on the same server hardware. You can have your own customized kernel modules, levels of Java, etc. Externally, it looks like you are running dozens of applications on a single server, but internally, each application thinks it is the only one running on its own OS. This gives you simpler coding model to base your test and development with.

Jeff is correct that running less than 10 percent utilization average across your servers is a crying shame, and that it could be managed in a manner that raises the utilization of the servers so that fewer are neeeded. Just as people could carpool, or could take the bus to work, it just doesn't happen, and data centers are full of single-application servers.

VMware has an architectural limit of 128 guests per machine, and IBM is able to reach this with its beefiest System x3850 M2 servers, but most of the x86 machines from HP, Dell and Sun are less powerful, and only run a dozen or so guests. In all cases, fewer servers means it is simpler to manage, so more applications per server is always the goal in mind.

VMware can soak up 30 to 40 percent of the cycles, meaning the most you can get from a VMware-based solution is 60 to 70 percent CPU utilization (which is still much better than the typical 5 to 10 percent average utilization we see today!) z/VM has been finely tuned to incur as little as 7 percent overhead, so IBM can achieve up to 93 percent utilization.

Jeff argues that since many of the z/OS technologies that allow customers to get over 90 percent utilization don't apply to Linux guests under z/VM, then all of the numbers are wrong. My point is that there are two ways to achieve 90 percent utilization on the mainframe, one is through z/OS running many applications on a single LPAR (the application container approach), and the other through z/VM supporting many Linux OS images, each with one (or a few) applications (the virtual guest approach).

I am still gathering more research on this topic, so I will try to have it ready later this week.

technorati tags: IBM, z10, EC, mainframe, Jeff Savit, Sun, Microsoft, HP, Dell, car, driver, bus, carpool, marathon, z/OS, LPAR, z/VM, guest, address space, application containers, ibmpulse, pulse2008, VMware, x3850

Seven Words

Jon: So, where is the 1500 number coming from?
Tony: I’ll investigate and get back to you.

My post tried to explain how IBM estimated that number. However, my fellow blogger from Sun, Jeff Savit, posted on his blog [No, there isn't a Santa Claus] in response. (If Sun's shareholders are expecting anything other than a [lump of coal] under the tree this year, they should probably read Sun's press release about their last [financial results].) A few others contacted me about this also, from a bunch of rather different angles, from reverse-engineering emulation of other company's chipsets to my use of internal codenames. (There are now MORE than seven words I can't type in this blog!) Jon is just trying to gather information, but his [head hurts] from all of this debate.

This week I will try to clarify some of the confusion.

technorati tags: George Carlin, Jon Toigo, DrunkenData, IBM, z10, Jeff Savit, Sun, quarterly loss, lump of coal

Seven Tiers of Storage at ABN Amro

Back in December 2005, Henk and his colleagues had come to visit the IBM Tucson Executive Briefing Center (EBC) to hear about IBM products and services. At the time, I was part of our "STG Lab Services" team that performed ILM assessments at client locations. I explained to ABN Amro that the ILM methodology does not require an all-IBM solution, and that ILM could even provide benefits with their current mix of storage, software and service providers. The ABN Amro team liked what I had to say, and my team was commissioned to perform ILM assessments at three of their data centers:

• Amsterdam (Netherlands)
• Sao Paulo (Brazil)
• Chicago, IL (USA)

Each data center had its own management, its own decision making, and its own set of issues, so we structured each ILM assessment independently. When we presented our results, we showed what each data center could do better with their existing mixed bag of storage, software and service providers, and also showed how much better their life would be with IBM storage, software and services. They agreed to give IBM a chance to prove it, and so a new "Global Storage Study" was launched to take the recommendations from our three ILM studies, and flesh out the details to make a globally-integrated enterprise work for them. Once completed, it was renamed the "Global Storage Solution" (GSS).

Henk summarized the above with "I am glad to see Tony Pearson in the audience, who was instrumental to making this all happen." As with many client testimonials, he presented a few charts on who ABN Amro is today, the 12th largest bank worldwide, 8th largest in Europe. They operate in 53 countries and manage over a trillion euros in assets.

They have over 20 data centers, with about 7 PB of disk, and over 20 PB of tape, both growing at 50 to 70 percent CAGR. About 2/3 of their operations are now outsourced to IBM Global Services, the remaining 1/3 is non-IBM equipment managed by a different service provider.

ABN Amro deployed IBM TotalStorage Productivity Center, various IBM System Storage DS family disk systems, SAN Volume Controller (SVC), Tivoli Storage Manager (TSM), Tivoli Provisioning Manager (TPM), and several other products. Armed with these products, they performed the following:

• Clean Up. IBM uses the term "rationalization" to relate to the assignment of business value, to avoid confusion with the term "classification" which many in IT relate to identifying ownership, read and write authorization levels. Often, in the initial phases of an ILM deployment, a portion of the data is determined to be eligible for clean up, either to move to a lower-cost tier or deleted immediately. ABN Amro and IBM set a goal to identify at least 20 percent of their data for clean up.

• New tiers. Rather than traditional "storage tiers" which are often just Tier 1 for Fibre Channel disk and Tier 2 for SATA disk, ABN Amro and IBM came up with seven "information infrastructure tiers" that incorporate service levels, availability and protection status. They are:
  1. High-performance, Highly-available disk with Remote replication.
  2. High-performance, Highly-available disk (no remote replication)
  3. Mid-performance, high-capacity disk with Remote replication
  4. Mid-performance, high-capacity disk (no remote replication)
  5. Non-erasable, Non-rewriteable (NENR) storage employing a blended disk and tape solution.
  6. Enterprise Virtual Tape Library with remote replication and back-end physical tape
  7. Mid-performance physical tape

  These tiers are applied equally across their mainframe and distributed platforms. All of the tiers are priced per "primary GB", so any additional capacity required for replication or point-in-time copies, either local or remote, are all folded into the base price. ABN Amro felt a mission-critical application on Windows or UNIX deserves the same Tier 1 service level as a mission-critical mainframe application. Exactly!

• Deployed storage virtualization for disk and tape. This involved the SAN Volume Controller and IBM TS7000 series library.
• Implemented workflow automation. The key product here is IBM Tivoli Provisioning Manager
• Started an investigation for HSM on distributed. This would be policy-based space management to migrate less frequently accessed data to the TSM pool for Windows or UNIX data.

While the deployment is not yet complete, ABN Amro feels they have already recognized business value:

• Reduced cost by identifying data that should be stored on lower tiers
• Simplified management, consolidated across all operating systems (mainframe, UNIX, Windows)
• Increased utilization of existing storage resources
• Reduced manual effort through policy-based automation, which can lead to fewer human errors and faster adaptability to new business opportunities
• Standardized backup and other operational procedures

Henk and the rest of ABN Amro are quite pleased with the progress so far, although recent developments in terms of the takeover of ABN AMRO by a consortium of banks means that the model is only implemented so far in Europe. Further rollout depends on the storage strategy of the new owners. Nonetheless, I am glad that I was able to work with Henk, Jason, Barbara, Steve, Tom, Dennis, Craig and others to be part of this from the beginning and be able to see it rollout successfully over the years.

For more about what was presented at Pulse 2008 conference, see the videos of the keynote speakers at [IBM Pulse - YouTube channel]!

technorati tags: IBM, ABN Amro, Henk de Ruiter, merge, mainframe, Tucson, Executive Briefing Center, EBC, STG, Lab Services, ILM, Amsterdam, Netherlands, Sao Paulo, Brazil, Chicago, Global Storage, study, solution, Productivity Center, DS8000, DS4000, SVC, storage tiers, rationalization, NENR, FC, SATA, Windows, UNIX, TS7000, HSM

IBM WebCast - Storage for SAP environments

IBM is hosting a webcast about storage for SAP Environments. Learn how integrated IBM infrastructure solutions, specifically, customized for your SAP environments, can help lower your business costs, increases productivity in SAP development and test tasks, and improve resource utilization. This will include discussion of archive solutions with WebDAV, ArchiveLink and DR550; IBM Business Intelligence (BI) Accelerator; IBM support for SAP [Adaptive Computing]; and performance benchmark results. The session is intended for SAP and storage administrators, IT directors and managers.

Here are the details:

• Date: Wednesday, June 18, 2008
• Time: 11:00am EDT (8:00am for those of us in Arizona or California)
• Registration: Register at the On24 Web site: [http://event.on24.com/clients/ibm/110218/storage]

technorati tags: IBM, SAP, storage, virtualization, webDAV, ArchiveLink, DR550, Adaptive Computing, BI Accelerator, benchmark, performance

Yes, Jon, there is a mainframe that can help replace 1500 x86 servers

( I cannot take credit for coining the new term "bleg". I saw this term first used over on the [Freakonomics Blog]. If you have not yet read the book "Freakonomics", I highly recommend it! The authors' blog is excellent as well.)

For this comparison, it is important to figure out how much workload a mainframe can support, how much an x86 can support, and then divide one from the other. Sounds simple enough, right? And what workload should you choose? IBM chose a business-oriented "data-intensive" workload using Oracle database. (If you wanted instead a scientific "compute-intensive" workload, consider an [IBM supercomputer] instead, the most recent of which clocked in over 1 quadrillion floating point operations per second, or PetaFLOP.) IBM compares the following two systems:

Sun Fire X2100 M2, model 1220 server (2-way)

IBM did not pick a wimpy machine to compare against. The model 1220 is the fastest in the series, with a 2.8Ghz x86-64 dual-core AMD Opteron processor, capable of running various levels of Solaris, Linux or Windows. In our case, we will use Oracle workloads running on Red Hat Enterprise Linux. All of the technical specifications are available at the [Sun Microsystems Sun Fire X1200] Web site. I am sure that there are comparable models from HP, Dell or even IBM that could have been used for this comparison.

IBM z10 Enterprise Class mainframe model E64 (64-way)

This machine can run a variety of operating systems also, including Red Hat Enterprise Linux (RHEL). The E64 has four "multiple processor modules" called "processor books" for a total of 77 processing units: 64 central processors, 11 system assist processors (SAP) and 2 spares. That's right, spare processors, in case any others go bad, IBM has got your back. You can designate a central processor in a variety of flavors. For running z/VM and Linux operating systems, the central processors can be put into "Integrated Facility for Linux" (IFL) mode. On IT Jungle, Timothy Patrick Morgan explains the z10 EC in his article [IBM Launches 64-Way z10 Enterprise Class Mainframe Behemoth]. For more information on the z10 EC, see the 110-page [Technical Introduction], or read the specifications on the [IBM z10 EC] Web site.

Moving Oracle workloads from x86 over to mainframe is quite common since [IBM and Cisco joined forces to meet Linux On Mainframe demand]. For more information on consolidating x86 servers running Oracle over to a mainframe, read the [Quick Reference], IBM Redbook titled ["Using Oracle Solutions on Linux for System z"], or this [presentation by Jim Elliott, IBM System z Specialist].

In a shop full of x86 servers, there are production servers, test and development servers, quality assurance servers, standby idle servers for high availability, and so on. On average, these are only 10 percent utilized. For example, consider the following mix of servers:

• 125 Production machines running 70 percent busy
• 125 Backup machines running idle ready for active failover in case a production machine fails
• 1250 machines for test, development and quality assurance, running at 5 percent average utilization

While [some might question, dispute or challenge this ten percent] estimate, it matches the logic used to justify VMware, XEN, Virtual Iron or other virtualization technologies. Running 10 to 20 "virtual servers" on a single physical x86 machine assumes a similar 5-10 percent utilization rate.

Note: The following paragraphs have been revised per comments received.

Now the math. Jon, I want to make it clear I was not involved in writing the press release nor assisted with these math calculations. Please, don't shoot the messenger! Remember this cartoon where two scientists in white lab coats are writing math calculations on a chalkboard, and in the middle there is "and then a miracle happens..." to continue the rest of the calculations? In this case, the miracle is the number that compares one server hardware platform to another. I am not going to bore people with details like the number of concurrent processor threads or the differences between L1 and L3 cache. IBM used sophisticated tools and third party involvement that I am not allowed to talk about, and I have discussed this post with lawyers representing four (now five) different organizations already, so for the purposes of illustration and explanation only, I have reverse-engineered a new z10-to-Opteron conversion factor as 6.866 z10 EC MIPS per GHz of dual-core AMD Opteron for I/O-intensive workloads running only 10 percent average CPU utilization. Business applications that perform a lot of I/O don't use their CPU as much as other workloads. For compute-intensive or memory-intensive workloads, the conversion factor may be quite different, like 200 MIPS per GHz, as Jeff Savit from Sun Microsystems points out in the comments below.

Keep in mind that each processor is different, and we now have Intel, AMD, SPARC, PA-RISC and POWER (and others); 32-bit versus 64-bit; dual-core and quad-core; and different co-processor chip sets to worry about. AMD Opteron processors come in different speeds, but we are comparing against the 2.8GHz, so 1500 times 6.866 times 2.8 is 28,337. Since these would be running as Linux guests under z/VM, we add an additional 7 percent overhead or 2,019 MIPS. We then subtract 15 percent for "smoothing", which is what happens when you consolidate workloads that have different peaks and valleys in workload, or 4,326 MIPS. The end is that we need a machine to do 26,530 MIPS. Thanks to advances in "Hypervisor" technological synergy between the z/VM operating system and the underlying z10 EC hardware, the mainframe can easily run 90 percent utilized when aggregating multiple workloads, so a 29,477 MIPS machine running at 90 percent utilization can handle these 26,530 MIPS.

N-way machines, from a little 2-way Sun Fire X2100 to the might 64-way z10 EC mainframe, are called "Symmetric Multiprocessors". All of the processors or cores are in play, but sometimes they have to take turns, wait for exclusive access on a shared resource, such as cache or the bus. When your car is stopped at a red light, you are waiting for your turn to use the shared "intersection". As a result, you don't get linear improvement, but rather you get diminishing returns. This is known generically as the "SMP effect", and in IBM documents this as [Large System Performance Reference]. While a 1-way z10 EC can handle 920 MIPS, the 64-way can only handle 30,657 MIPS. The 29,477 MIPS needed for the Sun x2100 workload can be handled by a 61-way, giving you three extra processors to handle unexpected peaks in workload.

But are 1500 Linux guest images architecturally possible? A long time ago, David Boyes of [Sine Nomine Associates] ran 41,400 Linux guest images on a single mainframe using his [Test Plan Charlie], and IBM internally was able to get 98,000 images, and in both cases these were on machines less powerful than the z10 EC. Neither of these were tests ran I/O intensive workloads, but extreme limits are always worth testing. The 1500-to-1 reduction in IBM's press release is edge-of-the-envelope as well, so in production environments, several hundred guest images are probably more realistic, and still offer significant TCO savings.

The z10 EC can handle up to 60 LPARs, and each LPAR can run z/VM which acts much like VMware in allowing multiple Linux guests per z/VM instance. For 1500 Linux guests, you could have 25 guests each on 60 z/VM LPARs, or 250 guests on each of six z/VM LPARs, or 750 guests on two LPARs. with z/VM 5.3, each LPAR can support up to 256GB of memory and 32 processors, so you need at least two LPAR to use all 64 engines. Also, there are good reasons to have different guests under different z/VM LPARs, such as separating development/test from production workloads. If you had to re-IPL a specific z/VM LPAR, it could be done without impacting the workloads on other LPARs.

To access storage, IBM offers N-port ID Virtualization (NPIV). Without NPIV, two Linux guest images could not access the same LUN through the same FCP port because this would confuse the Host Bus Adapter (HBA), which IBM calls "FICON Express" cards. For example, Linux guest 1 asks to read LUN 587 block 32 and this is sent out a specific port, to a switch, to a disk system. Meanwhile, Linux guest 2 asks to read LUN 587 block 49. The data comes back to the z10 EC with the data, gives it to the correct z/VM LPAR, but then what? How does z/VM know which of the many Linux guests to give the data to? Both touched the same LUN, so it is unclear which made the request. To solve this, NPIV assigns a virtual "World Wide Port Name" (WWPN), up to 256 of them per physical port, so you can have up to 256 Linux guests sharing the same physical HBA port to access the same LUN. If you had 250 guests on each of six z/VM LPARs, and each LPAR had its own set of HBA ports, then all 1500 guests could access the same LUN.

Yes, the z10 EC machines support Sysplex. The concept is confusing, but "Sysplex" in IBM terminology just means that you can have LPARs either on the same machine or on separate mainframes, all sharing the same time source, whether this be a "Sysplex Timer" or by using the "Server Time Protocol" (STP). The z10 EC can have STP over 6 Gbps Infiniband over distance. If you wanted to have all 1500 Linux guests time stamp data identically, all six z/VM LPARs need access to the shared time source. This can help in a re-do or roll-back situation for Oracle databases to complete or back-out "Units of Work" transactions. This time stamp is also used to form consistency groups in "z/OS Global Mirror", formerly called "XRC" for Extended Remote Distance Copy. Currently, the "timestamp" on I/O applies only to z/OS and Linux and not other operating systems. (The time stamp is done through the CDK driver on Linux, and contributed back to the open source community so that it is available from both Novell SUSE and Red Hat distributions.) To have XRC have consistency between z/OS and Linux, the Linux guests would need to access native CKD volumes, rather than VM Minidisks or FCP-oriented LUNs.

Note: this is different than "Parallel Sysplex" which refers to having up to 32 z/OS images sharing a common "Coupling Facility" which acts as shared memory for applications. z/VM and Linux do not participate in "Parallel Sysplex".

As for the price, mainframes list for as little as "six figures" to as much as several million dollars, but I have no idea how much this particular model would cost. And, of course, this is just the hardware cost. I could not find the math for the $667 per server replacement you mentioned, so don't have details on that. You would need to purchase z/VM licenses, and possibly support contracts for Linux on System z to be fully comparable to all of the software license and support costs of the VMware, Solaris, Linux and/or Windows licenses you run on the x86 machines.

This is where a lot of the savings come from, as a lot of software is licensed "per processor" or "per core", and so software on 64 mainframe processors can be substantially less expensive than 1500 processors or 3000 cores. IBM does "eat its own cooking" in this case. IBM is consolidating 3900 one-application-each rack-mounted servers onto 30 mainframes, for a ratio of 130-to-1 and getting amazingly reduced TCO. The savings are in the following areas:

• Hardware infrastructure. It's not just servers, but racks, PDUs, etc. It turns out to be less expensive to incrementally add more CPU and storage to an existing mainframe than to add or replace older rack-em-and-stack-em with newer models of the same.
• Cables. Virtual servers can talk to each other in the same machine virtually, such as HiperSockets, eliminating many cables. NPIV allows many guests to share expensive cables to external devices.
• Networking ports. Both LAN and SAN networking gear can be greatly reduced because fewer ports are needed.
• Administration. We have Universities that can offer a guest image for every student without having a major impact to the sys-admins, as the students can do much of their administration remotely, without having physical access to the machinery. Companies uses mainframe to host hundreds of virtual guests find reductions too!
• Connectivity. Consolidating distributed servers in many locations to a mainframe in one location allows you to reduce connections to the outside world. Instead of sixteen OC3 lines for sixteen different data centers, you could have one big OC48 line instead to a single data center.
• Software licenses. Licenses based on servers, cores or CPUs are reduced when you consolidate to the mainframe.
• Floorspace. Generally, floorspace is not in short supply in the USA, but in other areas it can be an issue.
• Power and Cooling. IBM has experienced significant reduction in power consumption and cooling requirements in its own consolidation efforts.

All of the components of DFSMS (including DFP, DFHSM, DFDSS and DFRMM) were merged into a single product "DFSMS for z/OS" and is now an included element in the base z/OS operating system. As a result of these, customers typically have 80 to 90 percent utilization on their mainframe disk. For the 1500 Linux guests, however, most of the DFSMS features of z/OS do not apply. These functions were not "ported over" to z/VM nor Linux on any platform.

Note: DFSMS can backup or dump Linux on System z partitions or volumes. See this [Appendix C. HOWTO backup Linux data through z/OS] for details.

Instead, the DFSMS concepts have been re-implemented into a new product called "Scale-Out File Services" (SOFS) which would provide NAS interfaces to a blended disk-and-tape environment. The SOFS disk can be kept at 90 percent utilization because policies can place data, move data and even expire files, just like DFSMS does for z/OS data sets. SOFS supports standard NAS protocols such as CIFS, NFS, FTP and HTTP, and these could be access from the 1500 Linux guests over an Ethernet Network Interface Card (NIC), which IBM calls "OSA Express" cards.

Lastly, IBM z10 EC is not emulating x86 or x86-64 interfaces for any of these workloads. No doubt IBM and AMD could collaborate together to come up with an AMD Opteron emulator for the S/390 chipset, and load Windows 2003 right on top of it, but that would just result in all kinds of emulation overhead. Instead, Linux on System z guests can run comparable workloads. There are many Linux applications that are functionally equivalent or the same as their Windows counterparts. If you run Oracle on Windows, you could run Oracle on Linux. If you run MS Exchange on Windows, you could run Bynari on Linux and let all of your Outlook Express users not even know their Exchange server had been moved! Linux guest images can be application servers, web servers, database servers, network infrastructure servers, file servers, firewall, DNS, and so on. For nearly any business workload you can assign to an x86 server in a datacenter, there is likely an option for Linux on System z.

Hope this answers all of your questions, Jon. These were estimates based on basic assumptions. This is not to imply that IBM z10 EC and VMware are the only technologies that help in this area, you can certainly find virtualization on other systems and through other software. I have asked IBM to make public the "TCO framework" that sheds more light on this. As they say, "Your mileage may vary."

For more on this series, check out the following posts:

• [Seven Words]
• [Virtualization, Carpools and Marathons]
• [Summer Reading for the z10 EC]

If in your travels, Jon, you run into someone interested to see how IBM could help consolidate rack-mounted servers over to a z10 EC mainframe, have them ask IBM for a "Scorpion study". That is the name of the assessment that evaluates a specific client situation, and can then recommend a more accurate estimate configuration.

technorati tags: Jon Toigo, DrunkenData, bleg, IBM, z10, EC, E64, mainframe, x86, AMD, Opteron, Sun, Fire, X2100, petaFLOP, Freakonomics, Red Hat, RHEL, IFL, VMware, Jim Elliott, Xen, Virtual Iron, Solaris, Linux, Windows, Project Big Green, Infiniband, STP, Sysplex, Scorpion study, MS Exchange, Bynari, Oracle

Summer Jobs and the Singularity

This is June. Many students are graduating from high school or college and looking for jobs. Some of these might be jobs just for the summer to make some spending money, and others mights be jobs like internships to explore different career paths. I found both programming and marketing are rewarding and interesting work, but each person is different.

There are a variety of ways to find out what your personality traits are, and then focus on those jobs or career paths that are best for those strengths. Here is an online [Typology Test] based on the work of psychologists Carl Jung and Isabel Myers-Briggs. The result is a four-letter score that represents 16 possible personalities. For example, mine is "ENTP", which stands for "Extroverted, Intuitive, Thinking, Perceiving". You can find out other famous people that match your personality type. For ENTP, I am lumped together with fellow master inventor Thomas Edison, fellow author Lewis Carrol (Alice in Wonderland), Cooking great Julia Child, Comedians George Carlin and Rodney Dangerfield (I get no respect!), movie director Alfred Hitchcock, and actor Tom Hanks.

USA Today had an article ["CEOs value lessons from teen jobs"] which offers some career advice from successful business people. Of course, what worked for them may not work for you, all based on different personality types. Here is an excerpt of the advice I thought the most useful:

• "If you are committed, you will be successful." (unfortunately, the reverse is also true: if you are successful, you will be asked to move to a different job)
• "Tackle offbeat jobs. Challenge conventional wisdom within reason. Come into contact with people from all walks of life."
• "Show an interest, demonstrate you want to be on the job."
• "Never limit yourself. Look beyond to what needs to be done, or should be done. Then do it. Stretch. Go beyond what others expect."
• "Find a job that forces you to work effectively with people. No matter what you end up doing, dealing with others will be critical."
• "Bring your best to the table every day. Learn professional responsibility and how to handle difficult situations."
• "Listen carefully to what customers want."

Before IBM, I ran my own business. If you are thinking, "Maybe I will start my own business instead?" you might want to see this advice from Venture Capitalist [Guy Kawasaki on Innovation]. While running your own business has advantages, like avoiding issues "working for the man", it has some disadvantages as well. It is certainly not as easy as some people make it seem to be.

Of course, things are a lot different nowadays than they were when these CEOs were teenagers. And the pace of change does not seem to be slowing down any either. Here is a presentation on [SlideShare.net] that helps bring to focus the realities of globalization:

| View | Upload your own

But in the future, you will not just be competing with other smart people around the globe. Which brings me to "the Singularity". The Institute of Electrical and Electronics Engineers, Inc. (IEEE) has [The Singularity - A Special Report], a whole issue of Spectrum Online devoted to this. Nick Carr offers his thoughts on this on his Rough Type blog in his post [What we talk about when we talk about singularity].

Whether you are a student looking for a job, or a seasoned professional willing to share some career advice, enter a comment below.

technorati tags: career advice, Carl Jung, Myers-Briggs, ENTP, USA Today, Thomas Edison, Guy Kawasaki, venture capitalist, Shift Happens, IEEE, Spectrum, Nick Carr, Rough Type, engineers

Software Programmers as Bees

"The environment that nurtures creative programmers kills management and marketing types - and vice versa."

By this, he means "kills" in the UNIX sense, I imagine, and not the "Grand Theft Auto IV" sense. Different people solve problems differently. Some programmers have the luxury that they can often focus on a single platform, single chipset, single OS, and so on, but Marketing types are trying to come up with messaging that appeals to a broad audience, from people with business backgrounds to others with more technical backgrounds, and that can be more challenging. For programmers, "creative" is an adjective; for marketers, it's a noun.

"Programming is the Great Game. It consumes you, body and soul. When you're caught up in it, nothing else matters."

True. As a storage consultant, I find myself writing code a lot, from small programs, scripts, and even HTML code for this blog. When you are in your zone, working on something, one can easily lose track of time.

"Here's the secret that every successful software company is based on: You can domesticate programmers the way beekeepers tame bees. You can't exactly communicate with them, but you can get them to swarm in one place and when they're not looking, you can carry off the honey. You keep these bees from stinging by paying them money. More money than they know what to do with. But that's less than you might think."

I have never tamed bees, but many of my friends who are still programmers are motivated by factors other than maximizing their income, such as: friendly co-workers, job security, casual attire, and interesting challenges. A few make more than they know what to do with, the rest have girlfriends "significant others" who solve that problem for them.

"One way or another, marketers get control. But...control of what? Instead of finding assembly lines of productive workers, they quickly discover that their product is produced by utterly unpredictable, uncooperative, disobedient, and worst of all, unattractive people who resist all attempts at management."

False. Either marketing had control in the first place (ala Apple, Inc.) or they never had. "Control of what?" is the key phrase here.

"The shock is greater for the coder, though. He suddenly finds that alien creatures control his life. Meetings, Schedules, Reports. And now someone demands that he PLAN all his programming and then stick to the plan, never improving, never tweaking, and never, never touching some other team's code."

True. But if you don't like surprises, perhaps software engineering is not the right career path for you.

"The hive has been ruined. The best coders leave. And the marketers, comfortable now because they're surrounded by power neckties and they have things under control, are baffled that each new iteration of their software loses market share as the code bloats and the bugs proliferate. Got to get some better packaging. Yeah, that's it."

This one depends. I've seen teams survive and manage, with junior programmers stepping up to backfill leadership roles, and other times, projects are scrapped, or started anew elsewhere. As for marketers, it doesn't take much to get one baffled, does it?

Thanks for the link, Tom!

technorati tags: coders, programmers, bees, marketers, management, hive, UNIX

Eleven Answers about Deduplication from IBM

1. Please provide the name of your company and the de-dupe product(s) you sell. Please summarize what you think are the key values and differentiators of your wares.

   IBM offers two different forms of deduplication. The first is IBM System Storage N series disk system with Advanced Single Instance Storage (A-SIS), and the second is IBM Diligent ProtecTier software. Larry Freeman from NetApp already explains A-SIS in the [comments on Jon's post], so I will focus on the Diligent offering in this post. The key differentiators for Diligent are:

   • Data agnostic. Diligent does not require content-awareness, format-awareness nor identification of backup software used to send the data. No special client or agent software is required on servers sending data to an IBM Diligent deployment.
   • Inline processing. Diligent does not require temporarily storing data on back-end disk to post-process later.
   • Scalability. Up to 1PB of back-end disk managed with an in-memory dictionary.
   • Data Integrity. All data is diff-compared for full 100 percent integrity. No data is accidentally discarded based on assumptions about the rarity of hash collisions.

2. InfoPro has said that de-dupe is the number one technology that companies are seeking today — well ahead of even server or storage virtualization. Is there any appeal beyond squeezing more undifferentiated data into the storage junk drawer?

   Diligent is focused on backup workloads, which has the best opportunity for deduplication benefits. The two main benefits are:

   • Keeping more backup data available online for fast recovery.
   • Mirroring the backup data to another remote location for added protection. With inline processing, only the deduplicated data is sent to the back-end disk, and this greatly reduces the amount of data sent over the wire to the remote location.

3. Every vendor seems to have its own secret sauce de-dupe algorithm and implementation. One, Diligent Technologies (just acquired by IBM), claims that their’s is best because it collapses two functions — de-dupe then ingest — into one inline function, achieving great throughput in the process. What should be the gating factors in selecting the right de-dupe technology?

   As with any storage offering, the three gating factors are typically:

   • Will this meet my current business requirements?
   • Will this meet my future requirements for the next 3-5 years that I plan to use this solution?
   • What is the Total Cost of Ownership (TCO) for the next 3-5 years?

   Assuming you already have backup software operational in your existing environment, it is possible to determine the necessary ingest rate. How many "Terabytes per Hour" (TB/h) must be received, processed and stored from the backup software during the backup window. IBM intends to document its performance test results of specific software/hardware combinations to provide guidance to clients' purchase and planning decisions.

   For post-process deployments, such as the IBM N series A-SIS feature, the "ingest rate" during the backup only has to receive and store the data, and the rest of the 24-hour period can be spent doing the post-processing to find duplicates. This might be fine now, but as your data grows, you might find your backup window growing, and that leaves less time for post-processing to catch up. IBM Diligent does the processing inline, so is unaffected by an expansion of the backup window.

   IBM Diligent can scale up to 1PB of back-end data, and the ingest rate does not suffer as more data is managed.

   As for TCO, post-process solutions must have additional back-end storage to temporarily hold the data until the duplicates can be found. With IBM Diligent's inline methodology, only deduplicated data is stored, so less disk space is required for the same workloads.

4. Despite the nuances, it seems that all block level de-dupe technology does the same thing: removes bit string patterns and substitutes a stub. Is this technically accurate or does your product do things differently?

   IBM Diligent emulates a tape library, so the incoming data appears as files to be written sequentially to tape. A file is a string of bytes. Unlike block-level algorithms that divide files up into fixed chunks, IBM Diligent performs diff-compares of incoming data with existing data, and identifies ranges of bytes that duplicate what already is stored on the back-end disk. The file is then a sequence of "extents" representing either unique data or existing data. The file is represented as a sequence of pointers to these extents. An extent can vary from 2KB to 16MB in size.

5. De-dupe is changing data. To return data to its original state (pre-de-dupe) seems to require access to the original algorithm plus stubs/pointers to bit patterns that have been removed to deflate data. If I am correct in this assumption, please explain how data recovery is accomplished if there is a disaster. Do I need to backup your wares and store them off site, or do I need another copy of your appliance or software at a recovery center?

   For IBM Diligent, all of the data needed to reconstitute the data is stored on back-end disks. Assuming that all of your back-end disks are available after the disaster, either the original or mirrored copy, then you only need the IBM Diligent software to make sense of the bytes written to reconstitute the data. If the data was written by backup software, you would also need compatible backup software to recover the original data.

6. De-dupe changes data. Is there any possibility that this will get me into trouble with the regulators or legal eagles when I respond to a subpoena or discovery request? Does de-dupe conflict with the non-repudiation requirements of certain laws?

   I am not a lawyer, and certainly there are aspects of [non-repudiation] that may or may not apply to specific cases.

   What I can say is that storage is expected to return back a "bit-perfect" copy of the data that was written. There are laws against changing the format. For exa