An exchange and discussion of Storage Virtualization
Barry Whyte is a 'Master Inventor' working in the Systems & Technology Group based in IBM Hursley, UK. Barry primarly works on the IBM SAN Volume Controller virtualization system. Barry graduated from The University of Glasgow in 1996 with a B.Sc (Hons) in Computing Science. In his 12 years at IBM he has worked on the successful Serial Storage Architecture (SSA) range of products and the follow on Fibre-channel products used in the IBM DS8000 range. Barry joined the SVC development team soon after its inception and has held many positions before taking on his current role as SVC performance architect. Outside of work, Barry enjoys playing golf and all things to do with Rotary Engines.
Tuesday September 23, 2008
Hursley at 50 and some more on Quicksliver
I was a little out of my normal loop the last week or two. This week is full with customer and potential customer visits. Today we had an SVC stalwart customer visit, to not only provide some great feedback on how SVC has over the last 5 years helped them to reclaim their lives (storage admins can have a life in the evenings and weekends, outside work - because you can now do those things during the working day) but they also gave some details of how they have implemented their multi cluster data-centers to get the best that SVC can provide. Its always refreshing to meet some SVC evangelists that really understand not only storage, but virtual storage. Thanks guys, I know you will be reading this once you made the trip back home. Equally refreshing was the constructive feedback - how can we make it even better - what could be great features to add, some obvious features to add - that yes we've known some of them for a while, but it's all about resource and priorities. But all comments, feedback and criticism are appreciated. With my performance hat on, the more feedback I can provide to bang my performance drum the better - hehe...
Meanwhile last week we held a series of events to help continue the celebrations of 50 years of IBM at the Hursley site, Hursley @ 50.
The opening day we had a visit from the UK Member of Parliament for Innovation of Universities who spent the morning talking with Distinguished Engineers, the Lab Director and visiting the podiums on show from the 10+ major products currently being developed and tested in the Hursley Labs.... every time you use a "hole in the wall" cash machine - think of Hursley and the software IBM developed to make it possible, yesterday and today.
The idea of the podiums was to not showcase what has happened over the last 50 years, but where we are heading today, and what up and coming technologies and advances the Lab is producing today... cue Quicksilver
The second day was an invite event for UK based CIO and CTO's - even a few CEO's turned up and over the day we all had a great time explaining what we are doing today and tomorrow and its always great to hear and get acknowledgment of the problems facing todays data centers and how what we are doing with SVC and the other Hursley software products can and are helping address these fundamental business issues.
The final day was for press and academics. I did get interviewed with the Lab Director discussing Quicksilver - but thankfully they 'edited' the piece to not include our conversation.
However the Hursley Museum is well worth a visit - unfortunately only open to IBMers or Guests, but for a quick fly through check out the Sky News technofile piece. Looking carefully you can just make myself and the SVC team out in the background (with a small Quicksilver demo rack) during John's camera piece.
Finally, I see that not everything IBM recently announced was un-interesting to Chris Mellor, as he has an interview worth reading with Rick White of FusionIO discussing how SSD's can be adapted to Enterprise Storage. He makes some conclusions which would seem to make sense, but I guess he missed the point that the form factor isn't the issue we were making more of a statement about the "wrapper" (commodity scale out vs monolithic scale up) - and given the recent history of IBM vs EMC on emerging storage methodology/technology i.e. SVC vs Invista... I know who I'd back...
Last week I was about to post congratulating Chris Mellor on his 'Blocks and Files' site being absorbed, acquired, defunct, by his move to The Register. I've followed Chris since his days over at TechWorld, where he posted a few articles after reading my blog, abstracting some 'jouralistic license' and making 2+2=5. However it was all relatively positive and interesting to see his take.
Blocks and Files became one of my regular daily RSS toolbar views, checking what Chris and the team's perspective on things.
I read his post on The Register regarding this weeks product announcements and was shocked at his tone. Something I'd expect more from a competitor. I considered if I should post a retort, or not give him the link. Other than his obvious un-impressedness, I think he did miss a few point.
IBM has a corporate requirement to RFA (Release for Announce) its products 30-60 days prior to their availability. This has always been the case, and the RFA site now gets picked up by google and so new RFA's easier to find! So it looks like Chris has discovered this and checks for updates so he can 'scoop' new offerings. There is a difference between an RFA (a technical breakdown of what a product contains), and an official press release, where we set the scene for the product and give more general background to its place in the portfolio.
The research press release from Monday was a summary of the ongoing research work being done in storage, so yes we have disclosed most of these in the past, and of course most recently our 'Quicksliver' flash proof of concept project, judging by the response we have had from clients, there is huge interest in this out there and rest assured we have many people working on a whole raft of next generation storage products.
The product releases, XIV, DS5000, DS8000 R4 and Tape products are new, they may have been already scooped by Chris an others, but they are only just now available to customers, and this was the first IBM had openly spoken of them, wanting to save the details and the strategy for the Montpelier launch day. So although Chris reeled off the links to his Block and Files snippets, the press that attended the 3 day event now have a lot more details of the products and how they fit with our Information Infrastructure strategy. These products build on IBM's existing world leading storage products, services, software and Systems.
If Chris had attended the days and the breakout product sessions maybe he would have had more to talk about, indeed maybe that was his frustration, I guess his invite was lost in the mail..
The interop just keeps growing. As you would expect you can today attach the new IBM Storage products behind SVC, running the latest 4.3.0 software.
SVC has actually supported XIV since August, we completed the controller qualification during our last SVT cycle, and the IBM Storage teams in the US completed the qualification of the DS5000 on our behalf.
Why would you want to put XIV behind SVC? Well the same reasons you would put any storage devices behind SVC. You gain all the advanced functions that SVC can provide on the backend storage. The ability to migrate data into XIV in a non-disruptive way as usual, the ability to backup, archive, migrate, replicate data from the XIV to other controllers. The performance enhancements over normal SATA based controllers allows for a much larger capacity pool to be attached to SVC while maintaining the performance of enterprise disks. The simplicity and ease of use of the interface means you can get this storage up and running behind SVC in next to no time.
The has been a lot of FUD over the last few weeks - coming from those that feel they need to spread such tales.
Here are a couple of the more important un-truths/comments that have been spread
1-way cache memory without mirror writes
Not true. By the time the data is in cache its already been mirrored to another data-module, so the cache in each module is 1-way, but there are two copies.
No GlobalMirror
This is true, however, as this post discusses, we have a workable, and industry leading virtualization device that provides Metro and GlobalMirror, so we can provide this functionality, and of course the remote site does not need to be an XIV array, it can be any supported SVC controller - much more flexible.
Lack of non-disruptive upgrades
I understand that the r10 software does provide a means (the same as we do in SVC) to upgrade each module in turn non-disrutively.
The general gist of BarryB's post is to suggest it uses more power than comparable systems. I'm not going to start arguing the point nor his maths, I presume he made sure they didn't involve any 'Hitachi Maths' - but one question you may want to ask back in response is how much overhead in power is DMX adding to those EFD's It maybe all well and good to spout the energy savings that flash can give at the EFD level, but what about the 'wrapper' how energy efficient is a DMX with 32 flash drives? Just a counter point. Also in this post he comments of all data flowing over a 1Gb link and it implies this is a single link - there are many 1Gb ethernet ports in use. He also suggests there is no upgrade path for existing customers - there is a migrate in mode that allows you to suck data into the XIV - and of course if you have SVC above it you can do this everyday with any disruption once its behind SVC, and with minimal downtime as you "image mode" import the volumes.
Double drive failures
These are nasty no matter what the controller. RAID-6 allows for more protection, but the extra P+Q parity rebuild workloads are just as likely to result in a 3rd drive failure occur. I'm no statistician, but I'm sure there are probably thesis out there looking at the likelyhood. The minimal work being spread over a large number of disks and the key thing being the greatly reduced rebuild time (minutes rather than hours) means there is less of a chance of this occuring. Most stroage controllers provide some level of data striping, array pooling, over and above the RAID protection. I'm sure there are many many thousands of users out there using RAID-5 with striped volumes above multiple arrays. What happens if two drives fail on one of the RAID-5 arrays - yup same thing. The most critical case for this to occur is a high performance application where data is spread over many hundreds of spindles to get the performance needed. The chances are you'll have regular backups and even offsite replication to protect such critical workloads.
I'm sure I've missed a few more, I need to read back through various posts as I'm sure I've seen a few 'artistic license' comments out there
It's worth noting that when running XIV behind SVC, we don't support the thin provisioning or copy services functions down at the XIV level - this is for the same reason we don't in general support these functions on other controllers.
Copy services are simple to explain, as only SVC knows the allocation of a vdisk to physical storage, so copy services need to be above the storage pooling laye. Similarily it would require a flush of the SVC cache in conjunction with the controller cache to ensure the data on disk was a consistent image. The alternative is to go for something like Invista's approach and provide volume level virtualization with no cache. This doesn't get the best out of virtualization however, as you can't stripe for performance, you are limited to the performance of the single volume. Nor can you perform sub-volume level migrations, hot-spot management, enhance performance with caching etc.
Thin provisioning behind SVC is an interesting one. Because we split the capacity into extents, and we stripe over these extents across multiple volumes, we expect there to be capacity. If you happen to run out of space on the thin provisioned disks (missing the alerts etc) then you risk taking a lot of vdisks offline. Since SVC provides Space-efficient Vdisks anyway, its not an issue.
Anyway, I know we've had lots of interest from existing SVC customers regarding XIV support, and a number prospective SVC and XIV sales are in the pipeline.
I'll cover some more details of the DS5000 and its future-proofing interface technology, the jump in performance and the increased drive support over the next couple of days.
There's been much discussion on the inter-web thing and the blogosphere regarding the various RFA (Release for Announce) documents IBM provides prior to official press-releasing products. Well with so much to cover the IBM Information Infrastructure event in France this week was chosen as the launch pad for everything we have today officially announced.
IBM XIV Yes, its finally official, the FUD and mud slinging from EMC was just that, the only reason we weren't discussing was to save the official statements until today in Montpelier. Look at the free publicity it produced - even with so much FUD and mud I've seen various independent writers comment - EMC must be worried.
IBM DS5000 A new enterprise class modular (adaptable) storage system for mid-range costs.
IBM DS8000 R4 Including RAID-6, larger FC drives, enhancements for our System Z users, flexible LPARs and more.
N6000 Storage System including new updates for snapshot copies with SnapManager and SnapVault.
Just a quick update to point to the press releases/updates and between myself and Tony we will be covering all this in more detail over the next few days, including the inside scoop from Tony in France.
I was excited about the press release and finally making public news of our work with SVC and NAND flash technology, but I wasn't prepared for the sheer scale of the press coverage we would get!
Just google "IBM quicksilver" for an idea of the coverage - the first day of the release there was almost nothing, now over 800,000 hits...
There's a few independent articles, many re-hashes of the press release, and even my discussions with Mr Burke lead to an article on EETimes. We are doing well at the moment though, with EMC advertising out products for free, and although it maybe a 'science experiment' in Mr Burkes eyes, our customers are seeing the potential - I've had many requests along the lines of 'when can we have it', 'we'd like to work with you on this', and 'can we come and see it'.
One of the key points I wanted to make in this post was to clear up any confusion / attempts at black-listing IBM, when it comes to the SPC fair-use rules. Mr Burke implied in his 'science experiment' post that we were claiming this as a non-standard SPC benchmark. Its not an SPC benchmark at all. As I clearly stated last week its an internal benchmark of 70% read 30% write, all miss, at 4KB transfers. For some time this has been known in the industry as a 'typical' workload for an open systems database workload, where the filesystem uses 4K pages. We fully understood the implications of SPC fair use rules, and my close colleague in Tucson, Bruce McNutt (co-author of many SVC white papers and one of IBM's SPC representatives) was in Hursley in July to help us run various internal benchmarks. At no point did we plan to release SPC like or SPC comparison numbers. We simply run the same 70/30 4K test on an SVC system that was comparable to the same system which we did submit to the SPC for auditing.
Any vendor can state amazing IOPs numbers. Flash vendors do this, storage vendors do this and everyone knows to take them with a pinch of salt. A read cache hit number is interesting, but doesn't really tell you anything more than how well the upper level of a code stack performs. Sometimes even hardware. Hitachi claim over 3 Million IOPs from USP-V - but that's their 'super-cache-hit' where the same data doesn't even leave hardware buffers on the host attached controller ports. Anyone can provide those kind of numbers and we didn't want to risk being tarnished with the same "so what" brush. The 1 Million 4K IOPs were real operations all the way down to the physical storage medium. With 70% of reads, and 30% of writes - not DRAM, no hardware and no controller caching. Just pure physical IOPs. The comparisons made in the press release were therefore against the same benchmark (going down to physical 15K RPM HDD and missing the SVC cache) using an 8-node SVC cluster with the same number of HDD and logical configuration as the system we used to perform our latest SPC-1 benchmark.
The key point here is that we are comparing a non-SPC benchmark on SVC with a non-SPC benchmark on 'Quicksilver' - which allows the comparison to be grounded with a known world-record breaking HDD solution, SVC.
Anyway, its been fun watching and responding to the media storm we created, and its now 6 for 6 out of Mr Burkes last 6 posts that have been discussing IBM products - rather than his own... defensive... it would seem so. Free marketing... for sure... Highly amusing... you bet.
Next week all will become clear. The week after that I will be attending our Hursley 50 years of Innovation events, CIO's, press and academic sessions. For those of you visiting Hursley we will have a 'Quicksilver' system on demo and I'll be helping man the storage podium, so come by and say hi.
1M IOPs from Flash - actions speak louder than words
Its been so difficult to keep all this to myself over the last few months, especially last month when we actually reached our goal. 1 Million real life IOPs at less than 1ms - no caching, just pure raw performance.
You may, or may not have seen the IBM press release, covering the first part of IBM's flash and SSD strategy. Unlike some vendors, we didn't want to just jump on the band wagon, and make some vague statements about following in EMC's footsteps and simply plugging them into an existing box that wasn't designed for them, limits the top end performance potential and almost definitely will not scale to many hundreds of the devices.
There are three plays here, SSD (drive form factor) that plug into a traditional storage controller, which all the major vendors are working on supporting - thats great, you get a much reduced response time (assuming the controller can cope) and a new tier of storage in the box that you can manually manage. The second play is the addition of non standard storage form factors to a host, most likely PCIe adapters, maybe even DIMM form factors or the like, the Systems folks in IBM are actively investigating how this can be used to best benefit our customers, as Sun and Intel are too. The third play is to build a storage controller that is truly optimized for this game changing technology. Thats what we have prototyped and is described in today's press release.
Myself and a few SVC developers in Hursley have been working closely with the storage research team in Almaden (yes, some of the same people that helped us bring you SVC in the first place) to build a highly scalable modular controller system that can not only be optimised for flash based devices, but by using this in addition to SVC can attach traditional HDD based controllers, with of course the ability to migrate data seemlessly in an online manner between the tiers/controllers.
One of the great things about the SVC software base is that it is extremely flexible, both with how we can add new features and functions (as discussed before) but also with the hardware configuration. We needed to build a cluster capable of sustaining 1 Million cache miss IOPs. So we did, and once the hardware was on-site, it took the team less than a week to get it running. This gave us the host attachment we needed, and the ability to attach flash based storage to our high end Power system hosts (which would be needed to generate that much I/O).
Now we have the functionality, features and host attachment, all we need is some flash storage. One thing that EMC won't have told you what can be the achilies heel of flash, mixed workloads. We all know that reads are stunning, but writes are much more complicated. That is why most of your 'laptop' SSDs don't actually give that much of a performance benefit over HDDs (Someone told me recently Windows does 100's of thousands of writes during boot...) Write endurance - I'm not going to go there, its not an issue, any problem can be solved, most enterprise flash vendors have worked around these problems in one way or another, but at what cost? Usually performance. Depending on how good the low level ASIC or FPGA code is, mixing reads and writes can be problematic. The search was on for a suitable flash device.
Robin Harris posted an interesting article last year that sent me off investigating the FusionIO - ioDrive, and after some discussions at a corporate level we got hold of a couple of cards to evaluate. They provided an interesting angle from a storage controller perspective and IBM has been working closely with FusionIO over the last few months to help both companies squeeze every last drop of performance out of the low level flash hardware.
Now we have the flash, we have the virtualizer, we need to connect them together. Internally on our testfloor, we use a huge numbers of different vendors storage controllers, but we also use SVC to test SVC (using SVC software and hardware that to SVC 'looks' like a storage controller). Take this a step further and we used the Fibre-Channel host attachment side of the SVC code to build a prototype modular flash controller. Using a high performance System X server, some ioDrives in the PCIe slots, add our Fibre-Channel HBA and modify the SVC code to read and write from the flash device... et voila - you have a very high performance flash controller. (SVC has already proved the Intel based System X hardware can handle huge numbers of IOPs and MB/s)
Now our critics tell us SVC adds latency... and that its going to cause you problems. So with a device that gives super low latency, if that were true we'd be in trouble. Not so.
The benchmark we ran was a pretty typical Open Systems database style workload. 70% read, 30% write at 4K transfer size. With no cache hits. We actually turned off, or created the vdisks as cache disabled, the SVC cache, guaranteeing the IO was going down to flash storage. It was with this workload we provided over 1 Million IOPs (1.1M at peak) with a response time maintained under 700us (microseconds) for many hours - so much for SVC being a slouch or adding latency. I have seen BarryB's customer pitch regarding EMC flash drives (I love the way they call them EMC flash drives, gives the impression they designed them, when they are actually made by STEC)... its probably not politically correct of me to divulge details of what he discusses, but if you are thinking about SSDs, and are contacted by EMC, ask them for a comparison... how many EFD's would they need to maintain the same rate of IOPs, and can they even match the same response time (with or without cache) more importantly how many DMX quadrants would be needed to reach 1M iops at such a low response time... we did this in just over one and a half EIA racks. About 71U to be precise, this includes the SVC nodes, and the UPS's needed for SVC.
Despite the obvious comparison we could draw, and would be recognised by all - i.e. SPC-1, one of the rules of performing Storage Performance Council benchmarks is that the product must be GA'd. As this is a proof of concept system at present, we cannot publish any SPC results relating to this system. SVC is audited by the SPC and has the world record disk based benchmark1. In order to ground the flash based 70/30 4K test against a known industry recognised product, we ran the same 70/30 4K 100% miss test againt the SVC configuration used to produce the SPC result. This gave us the comparisons used in the press release, the modular flash controller system provided 3.5x the IOPs at 1/20th of the response time.
I'm sure our critics will be along to say thats all very well and good, but is it shipping now... the whole point of this work was to look to the future and to show that a modular and flash optimised controller approach (rather than monolithic) is capable of providing the best that flash can bring. Combining this with storage virtualization enables existing SATA, SAS or FCAL devices to be added to the picture and most importantly you gain the ability to online migrate hot/cold data at will from flash to traditional storage (and vice versa), maybe even autonomically... an investment in a product like SVC is an investment in the future of your infrastructure... the future is bright... the future is virtual... the future is modular...
1 : Details of the SAN Volume Controller SPC-1 Results are available at: http://www.storageperformance.org/results/benchmark_results_spc1#a00052
IBM TV has been been running since 2006, as means to share marketing, educational and other informational videos with our customers, business partners and prospective customers.
This is just one of the "new media" and "social media" methods being used by large multi-national corporations - not to mention blogs such as this one - if you are into blogging from a corporate perspective, and haven't seen "naked conversations by Robert Scoble/Shel Israel" then its well worth reading as a background to why such media is being embraced by companies like IBM, and why its important to let the likes of me on here as a face behind the corporation!
Anyway, here's a new IBM TV program that discusses SVC, in particular the business problems it helps to solve, including some of the new features we introduced with 4.3.0 back in June.
One part of the celebration of 50 years of IBM at the Hursley site involves this great CBR supplement that covers those 50 years across all software, and hardware developments, and essentially innovation in the IT industry that have come out of Hursley. The lab is less than 5 miles from Winchester (originally the capital of England) hence the name of the original MB HDD - the Winchester Drive.
There are many things we all take for granted that have started life in Hursley, not to mention the backbone of the banking infrastructure, CICS, without which we wouldn't have cash points or 'holes in the wall'... and of course from my perspective a workable everyday Storage Virtualization solution.
Next month we are running an event for CIO's and the press, that covers ongoing innovation at Hursley, I've been asked to help organise the arenas to showcase what IBM STG are doing in Hursley and we should have some great things to talk about with respect to where we are going and where we are innovating for the future, more of which I'll cover over the next few weeks.
It seems hard to believe that its been a year since my first post on this blog. As it seems with all things as we get older, in some ways it seems only yesterday, but in other ways its feels like I've always been here...
Anyway, just a quickie post to say thanks to all my readers, comment-ers, and the storage blogging community who have made the last year a fun, amusing, and rewarding one (well most of you ;-P )
Here's to the next year, and I hope my ramblings have been useful to our existing customers, and maybe even helped a few of you considering SVC or other IBM storage products.
I was catching up on a few interesting comments on BarryB's post about the evolution of DMX - notice how he deftly ignored my loaded questions about host side IOPs / MBs - but then we all know EMC, for some reason, just won't publish any kind of yardstick for their products - ho hum.
Anyway, the following comments got me thinking about what we as vendors mean by non-disruptive. We all use it to describe upgrades to our boxes, to varying degrees of accuracy and realism. I have to therefore turn to what we mean by non disruptive from an SVC perspective.
No interruption to service.
No interruption to access.
It is likely this was the true definition as originally defined by the first vendor to coin the term, but in practice users will encounter different perceptions of different products. It is our goal that any supported upgrade should complete without disruption, and we test this extensively during system test as well as our ongoing stress and regression testing. Although our support matrix lists the recommended levels of software that should be used with the new code, it is mainly because we like to keep current with associated software levels to cover all installs and may have discovered problems or issues in previous levels of the associated code. Thats not to say for any reason that a previous level of code may work in your environment with the latest level of SVC. Caveat - don't be surprised to be asked - by any vendor - to upgrade if you do experience a problem, however one of our goals is to not suggest this as a pre-req, only as a result of analysis of issues and known problems.
One key thing that a product like SVC promises is the ability to perform non-disruptive upgrades of underlying storage controllers. Yet, we've now introduced a layer inside your SAN that itself must be upgradable. That's why we test SVC code upgrades to the ultimate degree. During the upgrade each node in turn is upgraded, however our 'split execution' model, of 1-way code and clustered code, means we can run the new 1-way code during the process and then the cluster itself makes a decision if it should roll forward to the new code once all nodes have access to it. Once the nodes agree, it happens. We don't ask that your stop applications or reduce workload during this time, as only one node in the cluster is upgraded at a time and this is done in a controlled manner that should limit any performance impacts. Inevitably most upgrades will happen at weekends or during periods of low utilisation, and this can make sense from a business continuity point of view should something outwith SVC's control happen during this time. Its critical to ensure the health of your SAN, hosts and multipathing layers prior to commencing the upgrade. We provide tools (CCU checker) to help users to health check their cluster.
Now that you have a layer between your hosts and storage, it must be itself upgradable. So software wise, we are covered. Hardware wise, yup we got that too. Recently we had a large (not at liberty to name) but key customer who was reaching the peak of what their original 2003 4F2 SVC nodes could achieve, they physically replaced all eight SVC nodes without issue and saw an improvement in response time from their existing storage, but from an SVC point of view utilisation dropped from >75% to <25%. This was all done non-disruptively (even the WWNN's were migrated so no re-zoning or reconfiguration of the SAN is needed either). I do not know of another Storage Virtualization product that can do the same, without some kind of host migration software, 3 month migration cycle or sometimes its just not possible. Ask HDS or EMC if they can replace the hardware online in a few hours - I know the answer and I'm sure you do to.
Even if you have to take a controller completely offline for some disruptive maintenance our latest 4.3.0 code allows you to use the Vdsisk Mirroring feature to prepare for this event, offline the controller, fix it, and then only sync back the data that has changed since it went offline.
Its difficult to find a single other product that provides such dynamic, truly non-disruptive and time-saving features that allow you to solve common (painful) problems storage administrators face on a daily basis.
Before I forget I wanted to wish SVC all the best in its 6th year in production. I guess it seems a lot longer, but then I have been (like a lot of the team) working on it for over 8 years now.
IBM just celebrated its 50th year at the Hursley site. For those of you that don't know Hursley, its probably one of the nicest places you could ask to work. The main 'house' in its current form was rebuilt in the Victorian era building on Georgian brick building (Sir Thomas Heathcote) and before that his father Sir William built a red brick, Queen Anne style mansion on the site of a Tudor house in Hursley Park, a former hunting lodge, and started a the present day Hursley dynasty.
Last weekend, the weather just held for a great afternoon/evening of entertainment. Our kids loved it, bouncy castles, tea cup rides and all. However my daughter was petrified of the Spitfire aerobatics, worrying that the plane was going to fall out the sky. Even now, that merlin engine sounds great and the maneuverability of the aircraft is stunning. Why a Spitfire - well as I may have mentioned in the past, when the Vicars factory was bombed during WWII the British Ministry of Defense took over 'the house' as their development base. When IBM took over in 1968 there was still hangers and the like on site. The scary thing is I remember all too well the 40th celebrations just after we had moved to Hursley from our exile in Havant (now Xyratex) on the same site IBM used to manufacture HDD's in the 80's and 90's.
Anyway, SVC has been through five hardware ramps in its five years in the field, more are on their way, and this just goes to prove the beauty of the platform agnostic nature of the code base we have developed, not to mention the benefits of piggy-backing on the bandwidth and processing enhancements of the Intel based Xeon platforms. (I'd love to tell you how many different platforms we've ported the code to, but can't - needless to say none of them perform as well as the Xeon platform)
2003 - 2145-4F2 - Based on the x335 - 2way SMP, 400MHz FSB, 2Gbit FC ports
2004 - 2145-4F2 - Based on the x335 - 2way SMP, 533MHz FSB, 2Gbit FC ports
2005 - 2145-8F2 - Based on the x336 - 2way SMP, 800MHz FSB, 2Gbit FC ports
2006 - 2145-8F4 - Based on the x336 - 2way SMP, 800MHz FSB, 4Gbit FC ports
2007 - 2145-8G4 - Based on the x3550 - 2way DualCore SMP, 1333MHz FSB, 4Gbit FC Ports
The multi-cored approaches, keeping up with next gen FC, ever increasing PCI bandwidths, FSB advances (replacements), memory speed and density improvements all bode well for the future of our flexible code stack. Here's to the next 5 years, and providing even faster, more advanced functionality to help our customers achieve their day to day business goals, and help to remove storage function inadequacies from the equation.
I thought after the 4.3.0 release I'd have a bit more spare time to post more often, however I'm working on a couple of interesting projects that have really taken off over the last few months and I'm once again flat out, not to mention being asked to start benchmarking things already waiting to be added to the 4.3.1 code stream...
I do want to take some time to explain why most of what Mr Burke has tried to imply about SVC and its SEV is really just the somewhat unsurprising FUD posting. I wouldn't have expected anything else... I'd prefer to liken the two Barry's more with Morcombe and Wise - with me being the latter obviously :P lol
Meta-data space usage. Less than 1% means just that. Its what we are recommending as a ball-park for customers to think about - mainly to make sure they realise there is an overhead. Mathematically its actually 0.1%.
Performance, yes I have done the tests, yes I do know the numbers, and yes I will make them known, but all in due course.
As for the total and utter FUD statements (I keep thinking we need a FUD2.0 type acronym, as some vendors throw FUD around, and others are the masters of FUD) aimed at trying to dis-credit SVC's ability to virtualize and SE-virtualize storage. Now I am lead to believe that most of DMX's ability lies around its 'secret sauce' caching algorithms, detecting application workloads and switching to modes to make the most that - and thats great if it knows you application. Its not so great when it doesn't and there are a lot of applications out there that don't have a unique signature of I/O patterns etc... in these cases you see DMX for what it can really achieve - for example SPC-1 actively tried to negate such algorithms, which is what I personally believe to be one of the key reasons EMC withdrew from the Storage Performance Council and fail to submit their products for independent audit - lets not go there again just now... Anyway, BarryB is trying to suggest that SVC does weird workloads that storage controllers cannot understand - but to storage, SVC is just a host. Something that is submitting blocks of data as writes, or requesting blocks of data for reads. I don't see the issue? Maybe the real issue is that yet again BarryB is trying to protect the monolith and hang onto the paradigm that the controller is the only place to do stuff. With SVC we've proved thats not true, and our many customers are benefiting every day from not having to buy big monoliths, and can tender for their needs across the vendors they desire.
The final point he makes about dumbing-down the storage, he's made a perfect example of how you wouldn't setup an SEV implementation. If you are doing large sequential reads and writes, the you'd pick a larger 256K grain size for starters. The data for a given vdisk will also come from the same physical array, contiguously, for the size at which you set the group extent size. In some cases up to 2GB before moving on to the next array. So the controllers at the backed will see the sequential stream of 256K requests for at least 2GB, then if they are good at detecting a sequential stream, they will jump to the next array and start pre-fetching for another 2GB... (Why would you be storing mp3's or videos on a DMX or DS8000 anyway? Even if you are a VoD customer, you want lots and lots of cheaper storage)
The is always going to be cases of workload and applications that don't work well with one grain size vs another - or even some cases where you may not want to use SEV at all. I explained all that in my posts and I guess I could promote Mr Burke up to the "Wise" status for spending the time to find a few example of what not to do in order to help his FUD-machine.
Just a quickie folks to let you know that 4.3.0 code is now available for download one day ahead of our planned GA.
This has been a major undertaking by all involved with SVC inside IBM, and those beta customers out there that provided useful feedback and helpful insights into how this new release stacked up in the real world. I know how hard we have all worked in development, test and support to get this release out and hope it can benefit our existing and any potential new customers.
I see that Hu thought he'd better make up some mumbo-jumbo about their 'chubby-provisioning' solution. I did post a comment but, as usual, his moderating means he ignores comments he doesn't like. I'd point you back to my discussion about random 4K writes randomly across a large volume, say one hits every 50 to 100MB apart. Very quickly in HDS's solution you will have fully-provisioned the disk... Can anyone enlighten me on what he's talking about with respect to de-frag? When we allocate a 'grain' of say 32K, its a contiguous 32K chunk on the vritual disk. For example, from LBA 64 to LBA 127 (512byte blocks). If you then allocate something at LBA 256, there is a 64K gap on the virtual disk, but at the managed disk level its two contiguous 32K blocks. No de-frag needed... maybe I'm missing something with HDS's implementation - but his waffle about this getting more complicated with smaller grain sizes - i.e. true thin provisioning - is not at all relevant to our implementation. Meanwhile Chuck has been saying Virtualization has failed to deliver on its promise (curious that he comes out the woodwork to bang this old school stance when we make a big splash), and that if you buy lots of expensive EMC software you can get out of the mess that they have let you get into in the first place.
Anyway, I wanted to talk about Virtual Disk Mirroring (VDM) today. This was one feature that people maybe didn't see or hear was coming, but it is something that many of our enterprise customers have been asking for and actually has some nifty uses that may not at first be aparent.
Virtual Disk Mirroring (VDM)
Mirroring itself isn't anything new or ground breaking, but mirroring across controllers is something that until now only host software has been able to provide. In a host based solution you need to provision two LUNs from your storage, then use something like LVM mirroring to manage. We've taken this capability out of the host and embedded the functionality into SVC. Building on my 'flexible stacked I/O component' post from last week, yet again we slipped the Mirroring component into the stack. This time we inserted the new component below FlashCopy and above space-efficient, for good reasons. For those of you that know, or knew, the SVC component stack here is the updated picture :
Before diving in with why we did this and what it means lets step back and look at what VDM does for you. Basically VDM means you can now have two copies of a virtual disk. The host therefore sees a single vdisk, but behind the scenes its two copies of the vdisk are stored on two sets of managed disks. The primary use case being to copy a virtual disk onto two physical storage systems, usually in two different managed disk groups. This lets you not only protect against single array failures (multiple physical disk failures) but also the loss of an entire disk storage system itself. It also allows you to greatly improve the availability of low-end storage systems - bringing the Mirrored vdisk up to the expected availability of enterprise volume. Virtual Disk Mirroring is designed to protect critical business data against storage system failures. In that sense, it is a high availability function comparable in scope with HACMP. Virtual Disk Mirroring is not a disaster recovery solution due to both copies being accessed by the same node pair and only addressable by a single cluster in one site.
Existing customers can add a new 'copy' to an existing vdisk, which will require an initial synchronisation process. The rate of synchronisation can be controlled. New vdisks can be created as a mirrored vdisk (two copies) which can be created as 'synchornised' or needing synchronisation (That is, once created a the sync operation will begin - the vdisk is online during this operation). One copy has to be marked as the primary copy. This primary designation can be changed at any time without disruption to applications. (Care should be taken with the create sync'd option as it will assume the primary copy is valid and the other copy is already in sync) Once a mirrored vdisk is created, SVC will maintain the two copies in sync. Writes to the vdisk will be sent to both copies. VDM does not load balance on reads, it will always read from the primary copy - unless the primary copy is offline. Of course you can configure your system to load balance on reads. Assume you have two storage controllers that you are using, each one storing a single copy of each vdisk. There is nothing to stop you alternating the primary copy across the two controllers. Thus you can spread the read workload across all the spindles and get the best performance by getting as many spindles going at once.
Looking at availability, the whole idea of VDM is really to protect against the loss of an mdisk that takes out one or more vdisks. If such an event occurs the mirrored vdisk will switch to use the one remaining online copy. A bitmap of changed grains is maintained from this point onwards. The grain size for VDM is fixed at 256K. When you bring the offline copy back online - by fixing the missing mdisk or controller, only the changed grains since the copy went offline are re-synced. Should you need to take a complete controller offline for some disruptive maintenance actions you could add new copies to the vdisks and thus maintain access during this mainteance period. We provide a new command line view to list the 'vdisks dependent on a controller' to aid just such operations. On a different note, if one copy has a medium error, the other copy will be used and the medium error will attempt to be fixed.
A comment on my previous post was asking if VDM essentially looks like RAID-10 when used in conjunction with SVC's striped vdisk option. In a sense yes , you could think of this as a form of RAID-10, however you still need to use some form of RAID below SVC. VDM does not provide any hot-spare ability, nor does it provide automatic data scrubbing - however it does provide repair and validate. These two options allow you to first validate that the two copies are in sync, and if not, perform a repair to re-synchronise the two copies. VDM does not limit any of the existing SVC features or functions, the vdisk itself can be any of the currently supported vdisk types and you can specify if each copy is fully allocated or space-efficient.
VDM therefore provides a way to migrate a vdisk from a space-efficient vdisk to a fully-allocated vdisk. Suppose you have a space-efficient vdisk that has become almost fully-allocated, or you have decided for other reasons to convert it to a fully-allocated vdisk. You can 'add a copy' to the vdisk that is fully-allocated and synchronise the two copies. You now have a fully allocated second copy of the disk. Now switch the primary copy to be the fully allocated copy and 'split' them. Split is a function that allows you to separate the two copies, with the primary copy remaining the online active copy. So you can migrate from space-efficient to fully-allocated without any disruption to the host application. Then you can return the second copy back to the free space on its managed disk group. You could of course make the switch the other way around, however you'd end up with a fully allocated space-efficient disk after the synchronisation. There is nothing to stop you making both copies space-efficient. Again this is another advantage of the stacked I/O component model used in SVC, and the thinking behind inserting VDM above space-efficient.
SVC currently has a restriction within migration, that is, you can only migrate vdisks between managed disk groups that have the same basic extent size. VDM solves this by allowing you to create mirrored copies between managed disk groups with different extent sizes. Simply add a second copy, that belongs to the managed disk group you wish to migrate onto, to the vdisk you wish to migrate. Synchronise the copies, then split them and delete the original copy. Some customers I spoke to during the run up to 4.3.0, coming from a Symmetrix view of the world saw the split function as something they recognise their systems today. This maybe a way to migrate from a Symm view of the world while getting to grips with the much more efficient FlashCopy features - especially incremental space-efficient FlashCopy.
Performance wise, you are now making two copies of the data, and on writes we won't complete the write until both copies have completed. The slowest mdisk in the VDM will dictate the performance. You should try and mirror across similar performing storage systems.
With the cache sitting about VDM, it should hide some of the impact on writing to two copies. Reads are sent to the primary, so to gain best performance on reads as I mentioned, round robin the primary copy across both managed disk groups/storage systems. The synchronisation operation is comparable to a FlashCopy or migration operation - in terms of performance impact.
Additional Features and Support
The other key features added to SVC 4.3.0 is a doubling of the supported vdisks. You can now map 2048 virtual disks from a single node pair, thus 8192 in total for an 8 node cluster. I've already covered the 256 flash copy targets per single source vdisk.
IPv6 support is added to both the SVC cluster - DHCP support, front panel interactions (which have also seen some re-work based on user forums) and the SSPC console also support IPv6 addresses.
The 4.3.0 release also adds to our very extensive support matrix :
Microsoft Windows 2008 Enterprise x64 Edition, SAN boot, 32-bit support, and clustering
Microsoft Windows 2008 Enterprise Edition for Itanium-based systems, including SAN boot
HP-UX 11i V3 for PA-RISC and Itanium-based systems including clustering with HP ServiceGuard
Apple Mac OS X Server 10.5.2 with ATTO Celerity FC-42ES HBA
We are also changing the licensing for FlashCopy with SVC 4.3. In the past, you licensed FlashCopy for the sum of source and target vdisk capacity that you wanted to copy. So if you wanted to make a single copy of 2TB of data, you would license 4TB of FlashCopy (2TB source+2TB target). Note that this is the maximum amount of data being copied at any time. With the ability to make up to 256 copies and the fact that space-efficient copies occupy very little physical space, we figured that charging for the full virtual capacity of the targets didn't make much sense. So, with immediate effect, FlashCopy is now licensed by the source capacity only. You can make as many copies as you like. In the example above, you would need to license only 2TB of FlashCopy capacity. This change is also retroactive for our existing SVC customers. An existing customer licensed for 4TB of FlashCopy can now make as many copies as they like of up to 4TB of virtual capacity; that's at least double the replication ability for no extra charge. If you are using cascaded FlashCopy, note that intermediate copies are both sources and targets. If you copy A->B->C, virtual disks 'A' and 'B' are sources and so count towards the FlashCopy entitlement ('B' and 'C' are targets).
Thats my roundup of the more detailed explanations behind the new SVC 4.3.0 release which will be available for download from the SVC support pages on June 27th and will be pre-installed for new customers on hardware shipped from manufacturing in early July.
As I promised yesterday this post is devoted to the technical details behind the new Space-Efficient Virtual Disk (SEV) and Space-Efficient FlashCopy (SEFC) functions available for no extra charge upon upgrade to the 4.3.0 software release of SVC. This post covers :
SEV Implementation Details
SEV Meta-data Management
Configuration & Application Implications
SEV with FlashCopy - SEFC
SEV Implementation details
In previous posts I have described how SVC is implemented as a stack of I/O components, each of these performs a specific encapsulated function, with the interface between each component being the same. This allows new components to be inserted into the stack in the most appropriate place. The key point for us in development is the lack of changes that need to be made to existing components, thus allowing fast development of new functions and limited chance of regression in existing code. Anyway, the SEV I/O component is inserted in the stack above the Virtualization layer (that performs the mapping between vdisks and mdisks) and below the FlashCopy layer. Thus FlashCopy does not need to know if a source or target vdisk is actually Space-Efficient. Thus SEFC is implemented in SVC with almost no additional development effort (over and above SEV that is!). Of course there are components that do need to be altered to make use of the new SEV nature, like the configuration interfaces, but generally the new function is encapsulated in the one new component.
SVC SEV is implemented as a fine-grain thin provisioning solution. By fine-grained I mean a relatively small allocation size is used. For each vdisk you can specify the grain size. This can be 32K, 64K, 128K or 256K. In general we expect people to use the smallest 32K grain size, as this means a 16K write is only going to allocate 32K on disk, with the free 16K available for the next write. In SVC this means we now have vdisk/mdisk extents, and within an extent we have grains. I'm assuming you have some knowledge of how SVC works, and understand what an extent is. This first write will allocate a new grain at the start of an extent belonging to the vdisk and the write will be inserted into this grain. This may result in only a partial grain being allocated. Each write I/O will result in the directory structure being updated to specify which extent, which grain, and the offset into the grain.
If we step back a bit and look at the SVC I/O flow, I/O requests now flow down through the 'Remote Copy' (MetroMirror and GlobalMirror) layer, into Cache, into FlashCopy and then down to SEV. When SEV receives the I/O request it has to perform a lookup in the directory structure for the relevant vdisk (The directory is implemented as a b-tree). I've described what happens on a new write above, but if this is a write to an existing grain that has already been allocated, then a lookup in the directory is needed to find where to write this I/O. This is still an LBA in terms of the virtual disk.
If the I/O is a read then the directory should contain a tree node that points to the required location (grain) within the virtual disk extent. In both cases (read/write) we now have the correct LBA to reference within the virtual extent and the I/O passed down to the Virtualization layer. This behaves as it always has, i.e. performs the virtual extent to physical extent lookup in the virtualization table and the I/O can be submitted to disk. ( There could be the strange case that a read is coming into a grain that has not yet been allocated. If so, SEV will simply return the I/O with a bunch of zeros - we get read miss performance that is on a par with read hit.)
Still with me I hope! The key to SEV is really the directory.
SEV Meta-data management
We could have gone for a course-grained solution as HDS have (42MB allocation unit) but that wouldn't be very Space-Efficient. That is, we could have stuck with our extent size being the allocation unit, 16MB to 2GB (in powers of 2) but it would mean with some applications that write all over the disk we'd be fully provisioned very quickly!
With the ability to fine-grain provision the SEVdisks we are pretty close to only using the space you use to store your data. If you are doing sequential 4K writes over an SEVdisk, with a grain size of 32K, you can do 8 writes into a single grain before we go ahead and allocate a new grain. Even an application that writes small blocks randomly across the whole disk range will end up only consuming the ( number of writes x grain size ). Compare this with a 42MB allocation unit a-la HDS!
We now have more meta-data to manage and store. As with all fine grained solutions, this makes the directory structure potentially too large to store in memory and we actually cache it, destage / stage / prestage from the same disks that are providing capacity for the SEVdisk. The overhead of additional capacity we need if you end up allocating the entire SEVdisk is less than 1%. ( But if you've fully allocated an SEVdisk you may as well convert it to a normal vdisk (i.e. fully provisioned) as you won't need to do the directory lookup. We even provide the means to do that with the new Virtual Disk Mirroring function - more of that in a later post. ) In general although we need <1% additional capacity to store the meta-data, this is nothing when compared to the potential savings SEV can provide - if for example you provision a 2TB SEVdisk, but only used 100MB of real space...
Storing the directory data with the real data means that we can also easily re-create the SEVdisk directory in the event of a disaster taking out the cluster and its live state information. We checkpoint this data and other meta-data to the quorum disks for use in the event of such a disaster. If you wanted to extract an SEVdisk from SVC, then again a combination of Virtual Disk Mirroring and image mode migrates can turn the SEVdisk back into a directly accessible fully provisioned LUN.
Configuration & Application Implications
With SVC we have provided two main 'types' of SEVdisk. You can chose to let SEVdisks automatically expand when they need more space, or you can do it manually. Doing it manually may suit you at first as you get to know the function. At any time, you can dynamically switch between the two types without any disruption to the application.
When you create an SEVdisk its just like creating a normal vdisk, except you now specify the 'type' and a 'real size'. The real size specifies how much capacity is upfront 'reserved' for the SEVdisk. SVC allocates this space from the managed disk group. As has always been the case, a Vdisk uses space from just one managed disk group. This helps to limit the risks of an application or user going crazy and using lots of space - thus reducing the potential impact. Unlike other vendors implementations (EMC, HDS) you do not need to create a special pool of storage for SEV to use. You can use managed disk groups you have today, you can mix Vdisks and SEVdisks in the same group, or you can keep them separate if you wish. The choice is yours.
If an SEVdisk is set not to expand automatically it will not be able to grow beyond the specified 'real size', it will go offline until you increase the real size or convert it to automatically expand. You can also specify a per vdisk warning level. This will raise an event in the cluster error log, and can send an SNMP trap or email notification to the storage administrator. The vdisk warning level is provided to alert when an SEVdisk is getting close to its defined real size.
If an SEVdisk is set to expand automatically it will be allowed to grow until it reaches its defined virtual size (the size as reported to the host system). You can also specify a contingency value for auto expanding SEVdisks. This specifies the amount that the 'real size' grows ahead of the allocated size. This can be used to prioritise which SEVdisks can grow and by how much. If you have 'over provisioned' and you have created more virtual capacity than you have physical disk capacity in the managed disk group, it is possible that you may run out of available physical space. If this happens, SEVdisks requiring additional capacity go offline. (Its worth noting that in this case only SEVdisks that need more capacity will go offline, any fully allocated vdisks or SEVdisks that do not require additional capacity at this time will remain online) The vdisk warning level can be used with auto expanding SEVdisks to catch 'rogue' disks that are growing fast. A second warning level is also provided at the managed disk group level. This notifies you when the group itself is over the threshold that you have set. It may be that none of the SEVdisks are over their limit, but overall you are consuming space in the managed disk group. Of course even in the non SEV case, the new managed disk group warning level maybe a useful warning when you need to think about buying more storage, or using a different group. Both limits can be configured in terms of capacity or percentage.
When to use SEV is a good question. It really depends on the application, and how it behaves. Many open system applications typically run at no more than 50% capacity utilisation. Thats a lot of spindles and space being wasted. With SEV you can now specify a base size and let the application grow to the size it needs. Generally again most applications only then grow at a slow and steady rate. Traditional techniques are to provision as large a disk as maybe needed next year, or the year after, now you can really use what you need. Again, it comes down to understanding your needs, your application needs and the corresponding performance requirements of the application and IBM can help you with all aspects of planning, implementing, testing and through into production.
Space-Efficient FlashCopy
I mentioned above that one of the great things about the SVC 'I/O stack' design is that we can pick and choose where to place new functionality while encapsulating the impact on other components. For example, FlashCopy itself needs to sit below the cache component so that you help to hide the 'copy on write' nature of FlashCopy, in this case the write simply completes to cache and later we do the copy on write operation. Its even better when we get something for free. With SEV sitting below the cache and FlashCopy, we can help to hide not only the SEV directory lookups, but FlashCopy doesn't need to know that its target vdisk is actually an SEVdisk, it simply issues a read from the source vdisk and a write to the target vdisk.
SEFC really does save you space. Now that we support up to 256 copies of a single source disk the innovative design of the SVC FlashCopy feature really starts to make a difference. SVC is one of the few products to support cascaded copies (copies of copies at later points in time) SVC's "Cascaded, Incremental, Multi-Target, Space-Efficient FlashCopy" isa true example of IBM's power at innovating. Lets looks at a real life example covering an application and its associated test environments.
In many cases, there can be multiple copies (“clones”) of production systems being used for various purposes such as development, test, QA and training. With traditional replication techniques, each copy occupies as much disk space as the original.
Using SEFC could significantly reduce the amount of space required because capacity would be used only for differences between the copies and the original data. Assume you use a fully-provisioned test master version of the data to create a copy that is physically isolated from the production data. From this test master copy, you create four test versions of the data. After a test run, you could “re-trigger” the copy from the test master to the test copy to reset its contents back to the original state for another test.
With traditional replication, the configuration would have required a total of five times the capacity of the production data (original data plus four copies) or even six times if you used a 'test master' copy as well. Using SEFC, that requirement would fall to a little over twice the production data (original data, test master copy, and some space for changes).
This example takes advantage of combining together the richness of the SVC FlashCopy function: space-efficient, cascaded, and multi-target FlashCopy are all being used. Of course you can add incremental to update the copies from the production data, where only the changed data from the previous full copy needs to be re-copied.
Anyway, here are a few of my quick thoughts. Make the grain size for target SEVdisks the same as the grain size for SEFC. That is, if you use 64K or 256K FC grain sizes, make the target SEVdisk use the same allocation size. If you plan to copy a large amount of data then 256K is the most 'efficient'. But if you plan to make a full copy of the source there is probably not much point in using an SEVdisk as the target. Because of SVC's layered architecture, you can use any combination of fully-provisioned and SEVdisks with all of the FlashCopy functions.
Thats my first part of whats new when you upgrade to 4.3.0, with respect to SEV and SEFC. In my next post I'll be covering Virtual Disk Mirroring (VDM) and the other enhancements we've made to SVC with the new release. I've tried to cover the answers without knowing your questions, ask away and I'll answer as best I can, rest assured if I don't know the answer, I know someone that does.
An exchange and discussion of Storage Virtualization
