Creating an ROI assessment for implementing IBM Rational solutions

Document options Document options requiring JavaScript are not displayed Rate this page Help us improve this content

Level: Introductory

Joshua Barnes, IT Process and Technology Consultant, Ajilon Consulting

15 Oct 2004

from The Rational Edge: This article offers techniques that project managers can use to quanitify the value of IBM Rational solutions in terms of return on investment. The approach described is based on the author's years of experience, and offers calculations and tips on presentation that decision makers find persuasive.

From The Rational Edge.

So... you are finally at that critical point: you have a good shot at implementing an IBM Rational solution, but executive management and the bean counters have asked the ROI question -- how much is this going to save us? If you are wondering how you can create an ROI assessment to answer such a question, this article is for you. I will give you an objective method for creating such an assessment as well as a supporting appendix full of the details.

This is the first of a series of articles providing real life experiences to help you implement the IBM Rational Unified Process®, or RUP®, and Rational tools for an enterprise. This "man-in-the-trenches" chronicle can arm you with the content and samples to enable your success.

Background

ROI estimates: what are they? During his review of this article, Walker Royce, Vice President of Rational Software Services, made the following comment:

Fundamentally, ROI estimates are not science; they are due diligence exercises to build a guiding coalition, and to gain buy-in on certain aspects of organizational change. Be wary of "unjustified precision" (i.e., averaging ten survey responses that provided a 1-5 rating is not a precise description of customer satisfaction) and objective representations of subjective or speculative estimates. Almost all ROI data in our industry is based on some dimension of speculation.

Software projects rarely do as well as planned and, as a result, far too often we have to visit our iron triangle of scope, cost, and time. How can you and your organization get better with planning and executing software projects consistently? IBM Rational may appear to be your ticket to this nirvana world, but, okay, it's not that easy! It is going to take time, money, and project leadership willing to commit to 15-25 percent productivity improvements over organizational averages (attempting more is usually too ambitious, because that's too much change for most organizations to absorb, and attempting less isn't worthwhile, because it doesn't have impact).

Enterprise implementations of IBM Rational products involve significant investments of capital. Consider all the components: licensing, maintenance, training, mentoring, consultants, new hardware, hardware upgrades, network infrastructure upgrades, employee overhead, etc. The total investment can paralyze unprepared implementers of RUP. The very real and scary thought of being in front of a committee of C-level executives and asking for a staggering dollar amount can be overwhelming.

But you are not going to be one of the unfortunate, unprepared, would-be RUP implementers whose vision of improved software development is left on the chopping block -- along with a part of your future success at that company.

Back to top

Know your organization

I've seen several common characteristics in software development organizations. Often, organizations investigating IBM Rational solutions have many varying process/methodologies in-house that are all serial or "waterfall" in nature. On top of that, the individual processes themselves are rarely followed consistently from one area of the organization to the next. Now, add in a smattering of point solution development tools that don't play very well together (let alone have “seamless" integration), and you have the setting for some real frustration and political maneuvering of resources who hide behind the inevitable chaos that has become the norm.

If this sounds familiar, you may be asking yourself, where am I going to start? Begin by inquiring about IT ROI specifics that are appropriate for your company and environment. Some companies have a culture that requires very specific formats for how calculations are performed; others do not. If you do not have any previous experience with IT ROI in your company, learn all you can before your first visible step, assembling your assessment team.

Back to top

Assembling the team

A dream team for an ROI assessment would have the following types of resources:

• Executive sponsor This is an empowered executive-level manager who will support the initiative with position power as well as allocation of budget and resources to set the stage for a successful implementation.
  
  
• Leader This is someone with proven experience implementing the RUP and IBM Rational products. Previous experience is crucial to identifying business drivers as well as key pain areas the organization is in need of resolving, and mapping these pain areas to the RUP and tools to objectively assess how much potential value add there is. If you do not have these skills, you should look to your organization for someone who does. If no such resource is currently on staff, you should consider bringing in someone externally who is qualified (e.g., an independent consultant, an IBM Rational consultant, or an IBM Rational partner).
  
  
• Financial member This is someone with knowledge of your organization's project and program budgets as well as standard overhead rates for resources.
  
  
• ROI member This is someone who is experienced in preparing IT ROI assessments and is well-versed in ROI calculations.
  
  
• Cruise director This is someone who "knows everyone." At most companies there are employees who have been around forever and know who to talk to, and who not to bother talking to. They know who will add value, who won't, and those special cases who may not add any direct value, but whom you need to talk to simply because they could create strong roadblocks if you failed to include them in this process.
  
  
• Communication member This is someone well versed in drafting IT communications. Part of estimating ROI is determining both tangible and intangible (less quantifiable) benefits. It is also about determining if there is a suitable baseline for measuring current actuals and derived metrics. Is there anyone you regularly deal with who really likes discussing metrics? Most likely not; therefore, you will need to put some positive spin on what you are doing to avoid impeding progress and success. Communication that continually highlights the positive aspects of this change and does not highlight current individual team or department performance/results can help pave the way for your progress.
  
  

The roles listed above are just that, roles. One resource may have the skills to fill more than one role and should, if possible. As I mentioned earlier, this would be a dream team, and kudos to you if you are fortunate to fill all the roles. The executive sponsor and leader are the two essential roles that must be filled; depending on your environment, the other roles will have varying levels of importance.

Back to top

Reviewing assessment input options

There are many tools at your disposal to use in gathering input to the ROI assessment. I will discuss some of the keys ones, such as vendor supplied tools, commercially available package applications, analyst reports, and Web searches.

IBM Rational-supplied assessment tools

A great starting place for an assessment option for RUP and Rational products is the IBM Rational Breakeven Productivity Model and IBM Rational Suite Economic Value Model. These are Excel-based tools, available from IBM Rational; they work very well for their intended purpose. ¹

The IBM Rational Breakeven Productivity Model is a workbook containing several spreadsheets. There is an "Inputs" sheet where all data items are entered. The rest of the pages in the model -- Business Case, Break-Even Analysis, Calculations, and Summary -- comprise a presentation based on the data inputs. This model has a very simple algorithm for determining the point at which the initiative breaks even as a result of realizing an increase in productivity, as follows:

The IBM Rational Economic Value Model will measure the value of implementing the Rational solution based on the COCOMO II ² software cost estimation model and your own business drivers. COCOMO II is an objective cost model for planning and executing software projects that supports return-on-investment decisions with a credible basis of estimate. This economic value model holds most COCOMO II parameters constant in order to isolate those most impacted by the IBM Rational solution. The Rational solution has a positive impact on many of the other parameters as well, but for the sake of simplicity, this model only deals with this subset.

There are benefits as well as some cautions to consider in presenting the IBM Rational-supplied assessment tools described above for management approval:

Benefits:

• These assessment tools are based on COCOMO II, the most widely recognized software cost model in the world.
  
  
• They will show an expected return as well as the productivity breakeven point.
  
  

Cautions:

• Executive management will normally not (and rightfully so) put too much faith into a vendor-supplied tool.
  
  
• COCOMO II is very complex -- is anyone at the executive level going to understand it?
  
  
• The tools can provide some high ROI figures which you can be asked to justify, or commit to in terms of benefits realization.
  
  

These two tools can add value to your overall assessment, but supplementing them with other input options is highly recommended.

There are additional tools that target only portions of the software development lifecycle, such as the IBM Rational CM (configuration management) Economic Value Model, ReqPro Economic Value Model, and Rose Economic Value Model. These can be useful for implementing portions of the full IBM Rational solution, or simply to generate useful ideas for individual tools. (These tools can also help in an upcoming step, identifying key areas where RUP and IBM Rational Solutions can add value.)

Commercially available packaged applications

There are more than 75 packaged applications commercially available for performing ROI analysis. These can be used for software, project estimating, and tracking return on investment. Some of these packages -- such as CIOView, Cost Xpert, and QSM Slim -- are very robust, taking a seemingly endless number of data points into consideration. ³

Benefits:

• Can provide many different views and levels of detail for differing audiences.
  
  
• Can evaluate numerous data points, many of which may not be considered without looking at such a package.
  
  
• Some are based on COCOMO II.
  
  

Cautions:

• Additional cost (some are quite pricy) including licensing, learning curve, etc.
  
  
• Some environments have a lengthy procurement process for "non-standard" software.
  
  
• Can be overkill for this type of an assessment where there is not a solid baseline to work from (in respect of current process and established metrics).
  
  

A packaged application can be an asset; but in my experience, unless you already have one in house, the real value lies in looking at various packages and gleaning ideas to incorporate into your assessment.

Analyst reports

There are many published reports by industry analysts that can be extremely useful in your decision-making process. But independent analyst reports are quite expensive -- $10 to $25 thousand per report is not uncommon. So unless you have unlimited budget, you might consider the freely available reports that IBM Rational commissioned from IDC, a global market intelligence and advisory firm in the information technology and telecommunications industries, which analyzes and predicts technology trends. These reports offer a wealth of information on ROI from IBM Rational process and tools. Here is a partial list of IDC reports (and there are probably others by other analyst firms as well):

• Achieving ROI with Rational Requirements Management Tools ⁴
  
  
• Dialect Solutions Group: Achieving ROI with IBM Rational ClearCase and IBM Rational ClearQuest® ⁵
  
  
• Achieving ROI with Rational ClearCase® ⁶
  
  

Benefits:

• Analyst reports offer a great starting point to look at what other companies have achieved and generate ideas where IBM Rational can add value to your organization.
  
  
• Many reports are available that can shed light on where RUP as well as each of the Rational tools individually (either as a point solution or integrated suite) have provided documented value to other companies.
  
  

Cautions:

• Although most analyst firms are reputable, vendor-sponsored reports that showcase capabilities will obviously not describe failures, and the purchase of independent analyst reports is expensive.
  
  
• Reports are typically based on customer-specific experience without sufficient detail for you to follow in doing your own ROI assessment.
  
  
• The environments that the reports are based on may not match yours.
  
  

Analyst reports and others written on the returns that have been measured and realized for RUP and Rational tools can add significant value from both a reference point as well as idea generators for you and your team.

Internet search

Performing a Google search on “IT ROI" will bring back enough information (over 5 million results last time I did this) to keep you busy for many lifetimes. However, this exhaustive source of information can help generate ideas for you to investigate in the next steps of this process: conducting the interviews during data collection.

Benefits:

• An abundance of articles in periodicals such as CIO, CTO, SD Magazine, etc. have ROI articles that can help formulate ideas for where you look for value from RUP and tools to be added.
  
  
• Stories offer ways to compare what others have done in this same situation (i.e., yours is not the first time an ROI assessment was done for a new technology).
  
  

Cautions:

• There is not an abundance of, specifically, RUP/Rational tool ROI assessments out there.
  
  

Back to top

Gathering data, selecting interviewees

Now that you have determined your best input options, you can begin the process of selecting resources to interview for gathering your data. There are four main qualities that I generally focus on ⁷ during this selection process:

Position power. Are enough key players selected, especially at the director level, so that those left out cannot easily block progress? This point ties back to an earlier one, where you may have to put some effort into a resource that will not add any value in order to remove any early road blocks to your ROI assessment, as well as to the entire implementation.

Expertise. Ensure ample resources are selected to capture the various points of view (i.e., not only managers, but software architects, analysts, implementers, and testers -- the project team in the trenches, feeling the pain).

Credibility. Does the selected interview population have enough people with good reputations to represent the real data you need for analysis as well as capitalize on communications?

Leadership. Does the group include enough proven leaders?

Back to top

Interviews

Once you have the short list of the population to interview, let the interviewing begin! The goal of interviewing is to gather data to feed your assessment. Here you are looking at the business drivers that are generating the need for an improved software development process along with a set of supporting development tools. The pain points associated with these drivers will be mapped to RUP and/or IBM Rational tools for the value proposition in your assessment. Examples of typical pain points are poor quality (numerous defects found by the users in production), poor requirements management methods (some development teams simply rely on email or Excel spreadsheets to handle requirements management, which can be a recipe for confusion), and configuration management issues (e.g., regression: defects fixed two builds ago have just reared their ugly heads again).

The assessment team needs to focus interviews only on productive data gathering that will benefit the assessment; these sessions can quickly become complaint sessions for some other issue, so you need to keep the interviewee on course. Now, I am not saying you should “lead the witness" as we see in movies in simulated courtroom examinations; you do not want to generate false positives in your interviewing that will skew your analysis and subsequent calculations.

Back to top

Analysis of key areas where RUP and Rational tools add value

From the first moment you begin analyzing your collected data, be as conservative as possible. Simply ensure that the most conservative ROI percentage you and your team calculates justifies the company's investment in the solution. If you can ensure this, you will have succeeded.

You, or the resource you have filling the lead role on the assessment team, will need to apply the experience and knowledge of RUP and Rational product implementations to the data from the interviews and analyze where value can be added. Some areas to consider concentrating focus on is:

Increased productivity

• Standard development process (RUP)
  
  
• Instantiation of best practices (RUP)
  
  
• Automated testing (IBM Rational Robot, IBM Rational XDE Tester®)
  
  
• Efficient communication (IBM Rational ClearQuest)
  
  
• Common language UML (IBM Rational Rose, IBM Rational Rose/XDE Modeler®)
  
  
• Component reuse IBM Rational ClearCase
  
  

Quality

• Increased coverage of regression testing by use of automated testing
  
  
• Traceability to requirements for clear understanding of what is being tested
  
  

Cost savings

• Proactive management of requirements (IBM Rational RequisitePro®)
  
  
• Integrated change management, requirements, and testing tools
  
  

Higher accuracy in estimates

• Defined roles and activities (RUP)
  
  
• Metrics (RequisitePro, ClearQuest, IBM Rational TestManager®, IBM Rational ProjectConsole®)
  
  
• Consistency of the approach is used across the organization
  
  

Reduction of project cycle times

• Less rework and scrap due to configuration management issues (ClearCase)
  
  

As part of your analysis, you will need to complete calculations as a percentage of estimated improvement on a discipline-by-discipline basis. For example, imagine that one of the pain points identified in your interviews is poor quality resulting in numerous defects being discovered in production. During analysis, you discover that very little regression testing is being completed because of time constraints involved with manual testing. A representative sample of manual tests are executed and timed. You then use Robot or XDE Tester to run the same tests. Taking into consideration the time to set up (and an estimate to maintain) you estimate that an 8 percent productivity increase of the total testing effort can be realized. Let's say you have analyzed three other contributors to the poor quality pain point and determined your overall increase in productivity is estimated at 25 percent for the testing discipline.

Back to top

Financial calculations

ROI is a simple calculation once you have the components; it is total benefit minus total cost. ROI expressed as a percentage is total benefit divided by total cost. For IT investment in the Rational solution, a three-year period (stated as 36 months, and not 3 calendar years) is a realistic and conservative amount of time to measure over. However, your organization may require a shorter or longer timescale for these types of IT investments. It's important to state your timescale in terms of months and not calendar years. Why? Because if you do not begin the actual implementation work until mid-April, you would be over a quarter down on the time you have to produce in your first-year results.

The approach I will walk you through has two financial calculation components: the total cost of ownership and the estimated benefit.

Estimated total cost of ownership (TCO)

The first financial calculation for you to work on is the total cost of ownership. This will include items such as: license cost, license maintenance cost (for all three years -- 36 months), hardware costs (new, upgrades, installation, etc.), training, mentoring, consulting, learning curve (initial dip in productivity to realize a subsequent increase in productivity), administrative costs, employee costs (time in training, new hires, etc.), and implementation team cost.

Your calculations must be conservative, so don't underestimate the cost of the implementation. Based on my experience, if data to reasonably estimate the "soft" cost (employee time, for instance) is not available, you should calculate the TCO for the implementation at two times the cost of the licenses and maintenance. Hence, if you are going to spend $1 million on license and maintenance over a 36-month period, then you should budget an additional $2 million for implementation costs. This is generally very conservative for Rational implementations and does not generate scrutiny over the TCO. Table 1 shows a list of TCO items for you to consider.

Table 1: Total cost of ownership (TCO) items to consider

Estimated benefit

The second financial calculation is your estimated benefit. The first step in calculating this is to obtain your total IT project budget for the year. Next, determine what percentage of current work effort maps to each of the RUP's nine disciplines. You may not be able to map to one or more of these disciplines; for example, your company may not be performing any business modeling activities currently. Therefore, do not get hung up trying to map to every one of the disciplines RUP has to offer if you are not currently doing work in that area; but remember that your mapping needs to add up to 100 percent for the disciplines you do map.

Now that you have your total IT project budget for the current year, subtract all project budget that the RUP and/or Rational tools will not be able to add value to (i.e., in a typical project, this may include budget for infrastructure upgrade projects, migration to a new release of operating system, etc.). The remaining budget has the potential for the RUP and tools to add value to.

Multiply the discipline breakdown percentages to the remaining budget to be applied to the expected benefit calculation. These values should represent the amount of funding that is spent in each discipline with your current processes and tools. Table 2 shows these calculations based on a massive project budget of $500 million.

Table 2: Calculations for estimated project benefit

Taking the estimated improvements for the analysis that was performed on the key pain points mapped to RUP and tools; you will extend the budget from each discipline. Start with the disciplines that will have improvements, their percentage of project budget, and estimated project budget. Then multiply the total estimated project improvement from the analysis, which will provide you with the potential savings applied to 100 percent of the projects for which RUP and the tools can add value. Obtaining 100 percent in the first year of an implementation would not be a Herculean feat; it would just not be a likely achievement. You must objectively look at your environment and its inherent dynamics -- between employees, budget release, and especially politics -- in order to conservatively estimate what percentage of the projects will actually involve RUP or tools (in the context of each discipline and upon the analysis you performed for each component of your overall estimated improvement). Multiply this percentage of projects you will be able to apply RUP and/or the tools to, and finally arrive at your first year-estimated benefit. An example of this result is shown in Table 3, where the requirements and testing disciplines are singled out as likely areas of improvement for the first year of the project.

Table 3: Determine what percentage of your projects will involve RUP or IBM Rational tools during the first year of a three-year implementation plan.

The next step will be to obtain projected IT project budgets for the next two years. If the next two years of projections are not available, look at the history of your overall IT project budget. Are they increasing at a steady rate per year, decreasing at a steady rate, or holding about the same? With the projected numbers or numbers that you derive from trend analysis of previous years and current market conditions for your company, repeat the exercise above for the next two years, as shown in Table 4. You must also re-evaluate how many of the projects are a potential for value add by RUP and/or the tools and increase the percentage accordingly.

Table 4: In this example, I have increased the project budget by 5 percent per year and increased the percentage of projects we can apply RUP and/or the tools to, progressively each year as the organization matures with the new process and technology.

Sum up your three-year (36 month) total estimated benefit, as shown in Table 5. Ask yourself: does this amount appear reasonable and achievable in your environment? Have you been conservative enough for your decision-makers to get behind these estimates and approve as well as champion you? If you have doubts, consider doing some additional analysis.

Table 5: Total benefit over a three-year interval.

If you have confidence in your 36-month totals, move on to the final step: your estimated ROI.

The basic steps in performing an ROI assessment on implementing IBM Rational technology.

Estimated ROI

Now that you have your estimated total 36-month cost and your estimated total 36-month benefit, the estimated return on investment is a simple calculation. The calculation for ROI is your total benefits minus your total cost. The calculation for ROI as a percentage is your total benefits divided by your total cost, as shown in Table 6.

Table 6: The calculation for ROI as a percentage is your total benefits divided by your total cost.

For those who would like to extend this to a more sophisticated model, consider adding in depreciation and the time value of money.

Back to top

Writing your assessment

Before you begin writing your assessment, consider your audience and your decision makers. This will be an assemblage of your company's executive IT management, so your time with them will likely be limited. Write the assessment as an executive summary, and target two to three pages in length at most. An example is provided at the end of this article as Appendix B.

Have an “executive summary" section on the first page that hits the critical success items you want them to focus on. Incorporate a visual on the first page that shows what you expect vs. what other companies and industry in general have proved to achieve. Keep all the details of the analysis you performed and the financial calculations out of the assessment; after all, this is an executive assessment. The details, all of them, should be contained in a supporting appendix (an example is provided at the end of this article as Appendix A) so that any or all of the decision makers can drill down successively into the calculations and reasoning behind your assessment.

From my experience, it's best to make the assessment and the appendices separate artifacts. If you are considering presenting them together in a bound format with tabs and full color glory, great! Have a section with the assessment, supporting appendix, meeting minutes of your individual interviews, and so on, but still have a separate two- to three-page version (an executive summary, as provided in my own Appendix B at the end of this article) of the assessment to distribute to them. Most executives will be receptive to a very short executive briefing, even if they are not very interested in the topic. When presented with a book, even if what they need to review is very short, more often than not the book will sit and only a select few will read your great work.

In lieu of textually describing the assessment and appendix, I have included a sample of both for you to use. These samples will be the cornerstone of this series of articles from the trenches to arm you with not just a nice write up of an approach, but a tangible set of samples for you to work on.

Back to top

Summary

Without real metrics derived from solid data, you do not have a baseline to work from. In the end, estimated ROI is just that, an estimate. It is a leap of faith that your decision makers must make, and you are going to have to get them comfortable enough to make that big step. Taking a conservative approach to your analysis and then performing financial calculations with a concrete approach will set the stage for this leap.

Once you have buy-in for your estimates, you will need to start putting the infrastructure in place to capture the data that you will analyze (i.e., utilizing IBM Rational RequisitePro for requirements management metrics, ClearQuest for change management metrics, etc.). Tracking your actual ROI and comparing to the estimate will provide you with a mechanism to steer the focus of your implementation and success. This topic will be part of the next installment in this series. Until then, if you have any questions about this approach, feel free to email me.

Back to top

Notes

¹Please contact me at joshbarnes@bellsouth.net if you are interested in obtaining either of these tools.

²See http://sunset.usc.edu/research/COCOMOII/

³For each of the three packages noted here, see, respectively: http://www.cioview.com/ , http://www.costxpert.com/ , and http://www.qsm.com/

⁴See http://www.dad.be/library/pdf/rational.pdf

⁵See http://whitepapers.silicon.com/0,39024759,60047345q,00.htm

⁶See http://whitepapers.silicon.com/0,39024759,60071840p-39000545q,00.htm

⁷All four of these are taken from John P. Kotter's book, Leading Change, Harvard Business School Press: 1996.

Back to top

Appendix A: Achieving ROI with Rational Unified Process and Rational Solutions

Editor's note:

This appendix is a sample of what the author typically provides clients as part of the ROI assessment. The tables you see below map almost identically to those presented in the main body of this article. Text contained within << double angle brackets >> indicate elements that the assessment team would customize for the client in an actual engagement.

This appendix provides details for the calculations presented in the Executive Summary, Achieving ROI with Rational Unified Process and Rational Solutions. The solution encompasses a modern software development process encompassing best practices and tools to support that process.

What we at << Your Company Name >> offer as a standard software development lifecycle process must be balanced with a set of supporting, fully integrated tools. Tools and process must be developed integral to one another. It is much easier to add process features if you do not need to support them in tools, which is why we are mitigating this risk by using best practices embodied in the IBM Rational tools.

The best practices that this initiative will implement form the foundation for the return on investment. They are proven to support business and technical decisions that balance tradeoffs and compromises that our software development teams face every day. Developing iteratively will enable our project teams to execute successive amounts of functionality at a regular pace and permit the team to use the lessons learned from each iteration to make informed corrections in response to the current business and technical risks. By taking an architecture-first approach, they will be able to tackle the highest technical risks in the system (balanced along with the highest business risks). Modeling represents analysis and design in visual form, permitting many levels of detail and views, which cannot be readily understood from code itself. Testing will occur continuously, with each new iteration representing a baseline against which the evolving application can be measured and traced to the requirements. Managing requirements and change arm the project team with the tools and process to keep pace with evolving user needs (showing an executable slice of functionality to a stakeholder can expose things they would have never considered by just reading text-based documents).

Back to top

Key areas to improve

We interviewed program, project, and functional managers to identify what key pain points our software development projects are experiencing. The goal of our interviews was to gather data to feed this assessment and let us focus in on where the RUP and Rational tools can provide immediate benefit as well as the most return on investment. Our conclusions in the Executive Summary were the primary result of how the adoption of IBM Rational solutions can improve the following key areas:

Standardized software development lifecycle process

Our organization has many varying processes and methodologies that are currently being utilized in our software development departments. Compounding the issue of not following a standard software development lifecycle across our development departments is the issue of these existing processes and methods not being consistently followed from department to department, or even within individual departments.

We have recognized the need for a standardized and consistently followed software development lifecycle process (with flexibility to right size the process for each department's project needs).

Requirements

Requirements gathering and requirements management practices currently range from ad hoc verbal or email communication to manual process captured in MS Word or Excel. This results in poor communication between the business users and our development department resources as well as the potential for miscommunication among the various IT roles. This has been a contributing factor to the delivery of incorrect and/or incomplete functionality, which at a minimum may not fully satisfy the intended business need(s) and, at a maximum, adversely impact the business.

Automated testing

Manual testing is prevalent in the organization. Manual tests are prone to errors in a greater degree than automated tests, while taking significantly more time to execute. Due to the typical deadlines we have for our projects, there is not enough time to provide sufficient test coverage of code that will be deployed to production, which increase post deployment development costs, diverts resources support and away from working on new projects.

<< Your Company Key Area to Improve >>

<< TBD >>

Back to top

Analysis of key areas where RUP and Rational tools add value

Analysis was preformed and calculations as a percentage of estimated improvements on a discipline-by-discipline based were completed. The details of the calculations are as follows.

Back to top

Calculations

Note that all ROI calculations are based on a 12-month period representing a “year", not a calendar year.

Estimated total cost of ownership

There are many items that we took into consideration for identifying the total cost of ownership (TCO) over the next thirty-six months. Table A1 lists the items of most significance we reviewed:

Table A1:

During our review of these items, we did not have a suitable baseline and/or enough experience as an organization with implementing a new, formal process for software development combined with a set of tightly coupled supporting tools (as opposed to varying point solutions from many different vendors). Therefore, taking all of the items above into consideration, we are estimating the TCO for this implementation conservatively to be 200% of our investment of licenses and first-year maintenance.

Our proposed cost for licenses and first-year maintenance is $6,200,000. Therefore, our conservative estimate for the TCO is 12,400,000 over a thirty-six month period.

Estimated benefit

There are nine disciplines in the Rational Unified Process: business modeling, requirements, analysis & design, implementation, test, deployment, configuration & change management, project management, and environment. Meeting with functional and project management, we mapped our current areas of work effort within software development into one of the nine RUP disciplines. Next we estimated the percentages of work allocation for the disciplines as represented in Table A2.

Table A2:

We started our calculations with an IT project budget next year of $500,000,000. Then we backed out 41% ($205,000,000) for all project budgets that the RUP and/or Rational tools would not add value to (i.e. infrastructure upgrade projects, migration to a new operating system version for our desktops, etc.). The remaining budget has the potential for the RUP and tools to add value. Applying the discipline breakdown percentages to the remaining budget resulted in Table A3.

Table A3:

Requirements management

Improvement in requirements elicitation, scope management, change impact analysis, and traceability between the development areas as well as the business is estimated to improve productivity of the requirements discipline, analysis & design discipline, and testing discipline. Using conservative percentages for productivity improvement based on the use-case method of capturing functional requirements, we estimate a 14% increase in productivity. Utilizing use cases will significantly reduce the ambiguity of our current requirements, resulting in a sharp decrease in the amount of rework and scrap we experience in analysis & design, implementation, and testing. By automating requirements management with RequisitePro, we estimate a 6% increase in productivity. Currently we are primarily using Excel spreadsheets to manage requirements, a very manually intensive and error-proned process. The ability to trace a functional requirement (detail level) up to a Feature (high level) as well as to a test case is estimated to increase productivity an additional 2.5% from the ability to manage the inevitable change in our requirements and perform timely impact analysis.

An overall productivity improvement of 22.5% for the area of requirements is estimated, as shown in Table A4.

Table A4:

Automated testing

Implementation of automation for unit testing, functional testing, performance testing, and regression testing would reduce project cycle times and substantially increase productivity of our testing. Building test plans and test cases based on requirements facilitates testing the intent of the application, not just the code. When test planning occurs in parallel with development, the time savings will be significant. Improving the scope and frequency of regression testing will provide us with an 18% increase in productivity. We will be able to plan, track, and execute the regression testing through Rational TestManager as well as advance to functional testing using the same tool. We estimate this will provide us with an additional 6% of time saved.

We estimate that implementation of automation for unit testing, functional testing, performance testing and regression testing will increase the productivity of testing efforts and yield an improvement of 25% overall, as shown in Table A5:

Table A5:

<< Your Company Additional Key Area to Improve >>

<< TBD >>

Second and third year estimated savings

The estimated savings above represent first year savings. We obtained project budget projections from the PMO for the next two years. The project budgets are expected to grow by 5% per year for the next two years, as shown in Table A6.

As we mature on the RUP and utilization of the Rational tools, we estimate the amount of project that we can apply the RUP and tools to will continue to increase. This increase will be reflected in our “percent of project applied to" column in the tables below. For our calculations, we have held our estimates in the “percent of project budget" column static. We expect these percentages to change over the course of our implementation and maturity level, and will track the changes with our actual measurements as we progress.

Table A6:

Total benefit over 36 month period

The total benefits over a 36-month period are the summation of the first, second, and third year estimated savings as shown in Table A7:

Table A7:

Estimated Return On Investment

The estimated return on investment over a 36-month period is as shown in Table A8:

Table A8:

Back to top

Notes

¹$38,350,000 * 22.5% = $8,628,750 | $8,628,750 * 30.0% = $2,588,625

²$76,700,000 * 25.0% = $19,175,000 | $19,175,000 * 33.3% = $6,385,275

³$40,267,500 * 22.5% = $ 9,060,188 | $ 9,060,188 * 55.0% = $ 4,983,103

⁴$80,535,000 * 25.0% = $20,133,750 | $20,133,750 * 70.0% = $14,093,625

⁵$42,185,000 * 22.5% = $ 9,791,625 | $ 9,491,625 * 90.0% = $ 8,542,463

⁶$84,370,000 * 25.0% = $21,092,500 | $21,092,500 * 95.0% = $20,037,875

⁷$2,588,625 + $4,983,103 + $8,542,463 = $13,784,191

⁸$6,385,275 + $14,093,625 + $20,037,875 = $40,516,775

Back to top

Appendix B: Executive Summary

Back to top

Executive summary: Achieving ROI with Rational Unified Process and Rational Solutions

Editor's note:

This appendix is a sample of what the author typically provides clients as part of the ROI assessment. Text contained within << double angle brackets >> indicate elements that the assessment team would customize for the client in an actual engagement.

This report highlights the potential ROI in the adoption of the Rational Unified Process (RUP) and Rational Solutions by << Your Company Name >>. There are many products in the industry designed to address the various areas of the software development lifecycle. However, using integrated products in an integrated environment can provide a tremendous return on investment. This assessment was performed by interviewing key resources within the organization and focuses on determining areas of risk and/or improvement where the application of IBM Rational solutions can have a significant impact in the following areas: increased productivity, better quality, and improved timeliness. Based upon this assessment, knowledge of software development best practices, and IDC industry averages, we conclude that by adopting the IBM Rational solution, << Your Company Name >> can yield significant benefits.

Deploying technology without changing process in an organization will create little impact -- and it often brings negative consequences. Naked technology wipes out productivity improvements, hurts return on investment, and dulls the bright edge of well-conceived strategies. Implementing a structured methodology that encompasses the entire software development life cycle can have a marked effect on an enterprise.

The best practices that this initiative will implement form the foundation for the return on investment. They are proven to support business and technical decisions that balance tradeoffs and compromises that our software development teams face every day. Developing iteratively will enable our project teams to execute successive amounts of functionality on a regular pace and permit the team to use the lessons learned from each iteration to make informed corrections in response to the current business and technical risks. By taking an architecture-first approach, they will be able to tackle the highest technical risks in the system (balanced along with the highest business risks). Modeling represents analysis and design in visual form, permitting many levels of detail and views, which cannot be readily understood from code itself. Testing will occur continuously, with each new iteration representing a baseline against which the evolving application can be measured and traced to the requirements. Managing requirements and change arm the project team with the tools and process to keep pace with evolving user needs (showing an executable slice of functionality to a stakeholder can expose things they would have never considered by just reading text-based documents).

We intend to measure success in economic values -- is this initiative providing an acceptable return on investment? In other words, is the amount we are spending, in consideration of resources and capital investments, providing quantifiable benefit that advances the mission of our company? Functionality, quality, and schedule will play a seminal role in this equation; missing functionality, poor quality, and missed schedules represent failures in delivering tangible value.

Back to top

Expected benefits

Requirements management

Improvement in requirements elicitation, scope management, change impact analysis, and traceability between the development areas as well as the business is estimated to improve productivity within the Requirements, Analysis & Design, and Testing disciplines. Using conservative percentages for productivity improvement with the use-case method of capturing functional requirements, we estimate a 14% increase in productivity. Utilizing use cases will significantly reduce the ambiguity of our current requirements, resulting in a sharp decrease in the amount of rework and scrap we experience in analysis & design, implementation, and testing. By automating requirements management with RequisitePro, we estimate a 6% increase in productivity. Currently we are primarily using Excel spreadsheets to manage requirements, a very manually intensive and error prone process. The ability to trace a functional requirement (detail level) up to a Feature (high level) as well as to a test case is estimated to increase productivity an additional 2.5% from the ability to manage the inevitable change in our requirements and perform timely impact analysis.

An overall productivity improvement of 22.5% for the area of requirements is estimated, resulting in a first year, 12-month savings of $2,588,625 and a three-year, 36-month savings of $13,784,191.

Automated testing

Implementation of automation for unit testing, functional testing, performance testing, and regression testing would reduce project cycle times and substantially increase productivity of our testing. Building test plans and test cases based on requirements facilitates testing the intent of the application, not just the code. When test planning occurs in parallel with development, the time savings will be significant. Improving the scope and frequency of regression testing will provide us with an 18% increase in productivity. We will be able to plan, track, and execute the regression testing through Rational TestManager as well as advance to functional testing using the same tool. We estimate this will provide us with an additional 6% of time saved.

We estimate that implementation of automation for unit testing, functional testing, performance testing and regression testing will increase the productivity of testing efforts and yield an overall improvement of 25% resulting in a first year, 12-month savings of $6,385,275 and a three-year, 36-month savings of $40,516,775.

<< Your Company Key Area to Improve >>
<<TBD >>

Back to top

Conclusion

Table B1

Back to top

Notes

¹Our percentages represent very conservative estimates, of which we expect our actual numbers to be much higher. Please see Appendix A for calculation details and justification

²Please see Appendix A for calculation details

Resources

• A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
  
  
• Global Rational User Group Community

About the author

		Joshua Barnes is a consultant with over a decade of experience implementing new process and technology, and he specializes in leading enterprise implementations of IBM Rational Unified Process and Rational tools. He is currently leading the implementation of the Rational software solution to over 1,500 resources at a client site. Some of his prior positions include senior director of application development of a Fortune 500 company and director of program management at a global software development company. He is the program director of the largest Rational User Group in the country, the North-Florida Rational User Group, as well as co-facilitator for the IBM developerWorks RUP and Software Lifecycle forums. Joshua enjoys speaking at conferences such as the IBM Rational Software Development User Conference, developerWorks Live!. He can be reached at joshbarnes@bellsouth.net.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Back to top