 | Level: Intermediate Razeyah Stephen, Manager, Domino Performance Team, IBM
Varadarajulu Manavalan, Domino IT Architect, IBM
Nirmala Venkatraman, Performance Architect, IBM
04 Oct 2005 In this article, we review the performance improvements we achieved by deploying Domino 7 on pSeries servers in a live IBM production environment.
During Lotus Domino 7 development, the Domino/Notes team worked very closely with the IBM Global Services (IGS) team as they upgraded or deployed 30+ Domino servers with Domino 7, supporting over 55,000 users. The joint goals were for Domino 7 to be of very high quality, and to validate the lab performance results in IBM deployment production environments.
In this article, we will describe the configuration of three pSeries servers in IBM's Domino 7 environment, share performance data collected, and discuss deployment and monitoring tips. For these three servers, we saw 55 percent CPU savings on one production server, and between 14 percent to 23 percent CPU savings on the other two production servers, when we upgraded to Domino 7. The information presented in this article should be very helpful for your Domino 7 deployment in general, and on pSeries in particular.
This article assumes you're an experienced Domino administrator.
Domino 7 mail servers within IBM
IBM started Domino 7 deployment in production in May, 2004. Currently, IBM Domino 7 servers serve over 55,000 Notes client users. Figure 1 shows the growth of the number of Notes users served by Domino 7 servers within IBM:
Figure 1. Domino 7 mail users served
The Domino development teams worked closely with the IBM deployment teams to monitor and improve quality and performance. The Domino 7 mail servers for the IBM early deployment environment include eight pSeries servers, three zSeries servers (two zOS and one zLinux), and four iSeries servers.
Domino 7 performance
This section describes three configurations used within IBM for Domino 7 deployment on pSeries. The first configuration supports IBM Research, the second supports IBM Global Services, and the third supports various IBM groups and departments. These production servers were running Domino 6.5.1 and Domino 6.5.3, and were upgraded to the currently available beta versions of Domino 7. The Domino 7 results we present in the graphs are based on Domino 7 beta 4, plus some fixes. (Note that Notes/Domino 7 has been released and is now available.)
Configuration 1: IBM Research
The first configuration hosted users from IBM Research. The following table shows the hardware/software for this configuration:
|
System model number
|
IBM 7038-6M2 (pSeries 650)
| | Processor type | PowerPC_Power4 | | Number of processors | 6 | | Processor clock speed | 1452 MHz | | CPU type | 64 bit | | Kernel type | 32 bit | | Memory size | 16384 MB | | Total paging space | 6144 MB | | Percent used (paging space) | 1 percent | | Active physical drives | 38 | | Active logical volumes | 74 | | File system type | JFS | | Operating system | AIX 5.3 ML1 | | Domino version | Domino 7 public beta 4 | | Number of logical partitions (LPARs) | 1 | | Number of Domino partitions (DPARs ) | 2 | | Registered users (DPAR 1) | 2120 | | Registered users (DPAR 2) | 1413 | | Percent active users (DPAR 1) | 33 percent | | Percent active users (DPAR 2) | 38 percent | | Average mail database size (DPAR 1) | 243 MB | | Average mail database size (DPAR 2) | 349 MB |
We enabled the following server tasks on this server:
ServerTasks=Tivoliv60,Router,Update,Amgr,Adminp,Sched,CalConn,mtbf,http,tmmscan,tmscan,
tmmscan,dbscan mail1.box,dbscan mail2.box
ServerTasksAt2=UpdAll updsystemnsfs.ind, updAll updmailfolders1.ind, updAll updmailfolders2.ind
ServerTasksAt1=Catalog
Our first Domino partition (DPAR 1) had 2120 registered users, 700 active users (33 percent), and averaged about 2,800,000 daily prime time transactions. The average number of daily transactions per user was 4,000. The second Domino partition (DPAR2) had 1413 registered users, 537 active users (38 percent), and averaged about 2,300,000 daily prime time transactions. The average number of daily transactions per user on DPAR2 was 4,283.
Domino Web Access was enabled on both partitions. (Both DPARs had sufficient available CPU resources to serve Domino Web Access users.) These user counts are for Domino 7; user counts for Domino 6.5.1 averaged three percent less.
Both DPARs were enabled for circular transaction logging. Transaction logging was disabled for the mail box via the Notes.ini setting MAILBOXDISABLETXNLOGGING=1. Busytime.nsf (Clubusy.nsf if clustered) was disabled via the SCHEDULE_DISABLETXNLOGGING=1 setting.
This configuration used the following mail templates (all numbers approximate):
- 500 users using a modified Std45Mail template.
- 900 users using a modified StdR50Mail template.
- 300 users using a modified StdR6iMail template. (This template supports both Notes and Domino Web Access functionality.)
- 1500 users using a modified StdR6Mail template.
In addition, 333 users used different kinds of templates (out of the box, user-modified, and so on). Approximately 42 percent of this user population used the modified StdR6Mail template.
In this configuration, users ran LotusScript, Java, and formula-based agents. Third-party applications running on the servers included:
- Trend Micro's Scanmail (antivirus security for Lotus Domino messaging), enabled for multiple mailboxes on each server.
- IBM Tivoli Monitoring for Messaging and Collaboration (ITMMC - Build Level 5.1.1.6).
- Document Expiration API. This ran every night against all mail databases, and deleted documents that met or passed their expiration date.
IBM Research performance data
Figure 2 shows weekly CPU percent usage in our IBM Research configuration. Each bar represents one week (7 AM through 5 PM Monday through Friday) of data.
Figure 2. IBM Research CPU usage
This configuration was upgraded to Domino 7 after the week of 10/29/04. The configuration ran various Domino 7 beta versions from the week of 10/29/04, and the data from the week of 8/12/05 to the week of 9/09/05 is for Domino 7 beta 4. From the data collected on this production server, Domino 6.5.1 was using an average 62 percent CPU, and when we upgraded to Domino 7 beta 4, the average CPU utilization dropped to 28 percent, a CPU savings of 55 percent.
Configuration 2: IBM Global Services
The second configuration hosted users from IBM Global Services (IGS). This table lists the hardware/software configuration for the IGS server:
|
System model number
|
IBM 7038-6M2 (pSeries 650)
| | Processor type | PowerPC_Power4 | | Number of processors | 8 | | Processor clock speed | 1452 MHz | | CPU type | 64 bit | | Kernel type | 32 bit | | Memory size | 32768 MB | | Total paging space | 4672 MB | | Percent used (paging space) | 1 percent | | Active physical drives | 34 | | Active logical volumes | 59 | | File system type | JFS | | Operating system | AIX 5.3 ML1 | | Domino version | Domino 7 Public Beta 4 | | Number of logical partitions (LPARs) | 1 | | Number of Domino partitions (DPARs ) | 4 | | Registered users (DPAR 1) | 4080 | | Registered users (DPAR 2) | 3992 | | DPAR 3 | Mail hub server | | DPAR 4 | Tivoli Storage Manager server | | Percent active users (DPAR 1) | 34 percent | | Percent active users (DPAR 2) | 32 percent | | Average mail database size (DPAR 1) | 162 MB | | Average mail database size (DPAR 2) | 129 MB |
We enabled the following server tasks on these servers:
ServerTasks=Tivoliv60,Router,Update,Amgr,Adminp,Sched,CalConn,mtbf,tmmscan,tmscan,
tmmscan,dbscan mail1.box,dbscan mail2.box
ServerTasksAt2=UpdAll updsystemnsfs.ind, updAll updmailfolders1.ind, updAll updmailfolders2.ind
ServerTasksAt1=Catalog
The first DPAR (listed as DPAR 1 in the preceding table) had 4080 registered users, 1387 active users (34 percent active), and performed about 7,500,000 transactions daily (prime time). The average number of daily transactions per user was 5,407. The second DPAR (DPAR 2) had 3992 registered users, 1277 active users (32 percent active), and handled approximately 6,500,000 daily "prime time" transactions. The average number of daily transactions per user was 5090. These numbers were obtained with Domino 7; the user counts for Domino 6.5.3 were 5 percent less.
Both DPARs were enabled for archive transaction logging. Transaction logging was disabled for the mail box via the Notes.ini setting MAILBOXDISABLETXNLOGGING=1. Busytime.nsf (Clubusy.nsf if clustered) was disabled via the SCHEDULE_DISABLETXNLOGGING=1 setting.
This configuration used the following mail templates:
- 272 users using a modified Std45Mail template.
- 800 users using the StdR50Mail template.
- 7000 users (87 percent of our total population) using a modified StdR6Mail template.
In this configuration, users predominantely ran LotusScript and formula-based agents. Third-party applications running on the servers were the same as listed previously for our IBM Research configuration.
IGS performance data
Figure 3 shows weekly CPU percent usage in our IGS configuration. Each bar represents one week (7 AM through 5 PM Monday through Friday) of data.
Figure 3. IBM Global Services CPU usage
This configuration was upgraded to Domino 7 after the week of 02/11/05. The configuration ran various Domino 7 beta versions starting the week of 02/11/05, and the data from the week of 8/12/05 to the week of 9/16/05 is for Domino 7 beta 4. From the data collected on this production server, Domino 6.5.3 was using an average 65 percent CPU. When we upgraded to Domino 7 beta 4, the average CPU utilization dropped to 56 percent, an improvement of 14 percent.
Domino 7 CPU improvements on this configuration were smaller that the research server because more users are on the modified StdR6Mail template (87 percent) and some new features in this template have additional CPU cost. Also, another factor may be the usage patterns of the users. Based on the average number of daily transactions, IGS users are more active and "heavier" than IBM Research users.
Configuration 3: various IBM groups
This third configuration hosted users from various IBM groups and departments. The following table lists the hardware/software used for this configuration:
|
System model number
|
IBM 7038-6M2 (pSeries 650)
| | Processor type | PowerPC_Power4 | | Number of processors | 8 | | Processor clock speed | 1452 MHz | | CPU type | 64 bit | | Kernel type | 32 bit | | Memory size | 32768 MB | | Total paging space | 6144 MB | | Percent used (paging space) | 1 percent | | Active physical drives | 36 | | Active logical volumes | 79 | | File system type | JFS | | Operating system | AIX 5.3 ML1 | | Domino version | Domino 7 Public beta 4 | | Number of logical partitions (LPARs) | 1 | | Number of Domino partitions (DPARs) | 4 | | Registered users (DPAR 1) | 3985 | | Registered users (DPAR 2) | 4001 | | DPAR 3 | Mail hub server | | DPAR 4 | Tivoli Storage Manager Server | | Percent active users (DPAR 1) | 31 percent | | Percent active users (DPAR 2) | 33 percent | | Average mail database size (DPAR 1) | 136 MB | | Average mail database size (DPAR 2) | 149 MB |
We enabled the following server tasks on this server:
ServerTasks=Tivoliv60,Router,Update,Amgr,Adminp,Sched,CalConn,mtbf,tmmscan,tmscan,
tmmscan,dbscan mail1.box,dbscan mail2.box
ServerTasksAt2=UpdAll updsystemnsfs.ind, updAll updmailfolders1.ind, updAll updmailfolders2.ind
ServerTasksAt1=Catalog
The first DPAR (DPAR 1) had 3985 registered users, 1235 active users (31 percent active), and handles about 7,000,000 daily prime time transactions. The average number of daily transactions per user was 5,668. DPAR 2 had 4001 registered users, 1320 active users (33 percent active), and performed an average of about 7,000,000 daily prime time transactions. The average number of daily transactions per user was 5303. These are Domino 7 counts; user numbers for Domino 6.5.3 were 6 percent less.
On both DPARs, we enabled archive transaction logging. As with our other configurations, we disabled transaction logging for the mail box via the Notes.ini setting MAILBOXDISABLETXNLOGGING=1. Busytime.nsf (Clubusy.nsf if clustered) was disabled via the SCHEDULE_DISABLETXNLOGGING=1 setting.
This configuration used the following mail templates:
- 306 users using a modified Std45Mail template.
- 726 users using the StdR50Mail template.
- 6954 users (87 percent of our total population) using a modified StdR6Mail template.
Users predominately ran LotusScript and formula-based agents. Third-party applications running on the servers were the same as described in our previous two configurations.
Configuration 3 performance data
Figure 4 shows weekly CPU percent usage in this configuration. Each bar represents data collected for one week (Monday through Friday).
Figure 4. Configuration 3 CPU usage
This configuration was upgraded to Domino 7 after the week of 02/25/05. The configuration ran various Domino 7 beta versions from the week of 02/25/05 on, and the data from the week of 8/12/05 to the week of 9/16/05 is for Domino 7 beta 4. From the data collected, Domino 6.5.3 was using an average of 66 percent CPU, and when we upgraded to Domino 7 beta 4, the average CPU utilization dropped to 51 percent, an improvement of 23 percent.
Domino 7 deployment tips
Based on our experiences deploying Domino 7 on these three configurations, we recommend the following.
Domino parameters
Run with multiple mail boxes (we used two mail boxes). Disable transaction logging for mail.boxes via the Notes.ini parameter MailBoxDisableTXNLogging=1. Disable transaction logging on Busytime.nsf (Clubusy.nsf, if clustered) by setting the Notes.ini parameter Schedule_Disabletxnlogging=1. In addition, set the following in the server's Notes.ini file:
Server_Max_Concurrent_Trans=1000
Server_Pool_Tasks=100
Disable_BCC_Group_Expansion=1
Server_Session_Timeout=31
Implement Extended Directory Catalog rather than Directory Catalog to avoid full-text indexing overhead (for instance, Edcww.nsf uses view indices versus dircat/ibmdclns.nsf uses full-text indices). At a minimum, implement circular transaction logging for greater data integrity and faster restarts. Turn off unneeded Domino server tasks. To improve server performance, limit the amount of debugging/logging that is written to the log file (Log.nsf) and the console.
We obtained better performance with separate file systems for system databases. Each Domino partition (DPAR) was configured with four SSA RAID 5 arrays for Domino Data directory (/notesdata) and one SSA RAID 5 array for Extended Directory Catalog. Each Domino partition has a mirrored pair of SCSI transaction log drives, hardware RAID 1 mirroring using the 2498 or similar SCSI RAID adapter.
Use the Domino Directory large directories (such as Names.nsf, Edcww.nsf, and Wwpdl.nsf). Set VIEW_REBUILD_DIR=/notesdata/viewbuild. Our default location was the Domino Data directory. When implemented, Domino uses the Optimized view rebuild instead of a standard rebuild. Moving to another location can remove high-access file contention (Log.nsf, Names.nsf, and so on).
Limit database growth to require less I/O and CPU resources (large files use more CPU; mail databases with more documents use more CPU). This helps speed up view updates and rebuilds.
AIX parameters
Enable IOCP (I/O completion ports). Enter smitty iocp and ensure the current state is "available." Also, make sure the "State to be configured at system restart" setting is "available." IOCP is enabled and the current state is set to "available" on all IBM servers discussed in our configurations.
Set maxuproc to 128. (Refer to release notes for further information.)
On our servers, maxperm% and maxclient% were set to 10 to ensure there was no paging. To set these to be persistent across system reboots, enter "vmo -p -o maxperm%=10 -o maxclient%=10." The settings maxperm% and maxclient% can be different from ours for your environment. In addition, on our server, total paging space was set to 6144 MB and 4672 MB respectively.
Set the following:
-
AIXTHREAD_COND_DEBUG=OFF turns off the automatic logging of the list of condition variables for use by the AIX debugger. The default is ON.
-
AIXTHREAD_RWLOCK_DEBUG=OFF turns off the automatic logging of the list of read-write locks for use by the AIX debugger. The default is ON.
-
AIXTHREAD_MUTEX_DEBUG=OFF turns off maintenance of list of active mutexes for use by the AIX debugger. The default is OFF.
These variables can be set in either .profile, or in the .kshrc or .chrc file of the Notes user.
Performance monitoring tips
This section lists a few performance monitoring tips for AIX and Domino servers.
AIX servers
These are the set of tools (run periodically), that we use for performance monitoring of AIX servers:
-
'ps -aux' to collect process information, such as process ID, percent CPU, and virtual memory footprint.
-
vmstat and iostat information to collect CPU, memory, and I/O performance data. Collect this periodically during peak and off-peak hours. For example, "vmstat 300 2" collects two five-minute samples.
-
filemon information, similar to iostat information, for disk/file I/O performance data.
-
tprof data to collect CPU profiling information. For instance, to collect a one-minute tprof sample and create a sample output domino.prof, run the following command: tprof -ske -r domino -x 'sleep 60'.
-
'show stat' for Domino statistics collection, using the remote console or through statrep.
Domino servers
These are the tools we use to monitor Domino servers after upgrading, to ensure that everything was working as expected:
-
Domino statistics. Events4.nsf and Statrep.nsf were used to configure, create, and report information from all Domino servers in the infrastructure.
-
Operating system statistics. The most essential statistics monitored were CPU utilization, memory utilization, disk I/O activity, and disk capacity. These statistics can be compared against data from a previous version and verify that there is no bottleneck on the system.
-
Activity trends. These were used to record and report statistics on user activity (clients) against databases on the Domino server. The reported statistics were trended over time, which provided a way to measure and compare workloads across servers. This information was used to identify the most active users and databases on each server. This tool is useful for short-term and long-term capacity planning.
-
Domino 7 Administrator client (server monitoring) is very useful and provides a quick overview of infrastructure, and identifies tasks that may have failed.
-
Domino logs. Check the Domino logs (Log.nsf and console logs) for problems with database corruption, process starts and stops, failed processes or error conditions, network traffic, agents, security, SMTP, HTTP, and so on, which might cause performance issues. This should be done on a daily basis to catch issues (if any) sooner rather than later.
Historical data collected using these tools help you monitor and troubleshoot performance issues through the upgrade process. We have used these tools extensively through the Domino 7 upgrade on IBM production servers, without incurring substantial overhead. We used these tools to build a baseline, and to collect historical data. This allowed us to find changes easily, and if needed to track down any performance issues.
Conclusion
In the IBM pSeries production environment we measured between 14 percent to 55 percent CPU savings, when compared to Domino 6.5.1/6.5.3. The 55 percent CPU improvement was on a pSeries server with two Domino partitions. This server hosted 3533 research users. On two other servers, one hosting IGS users and the other users from various departments, we measured 14 percent and 23 percent CPU savings, respectively. These two production servers hosted about 8,000 users each and had two Domino partitions each. We consider these two configurations more typical of customer environments.
In our lab environment we have measured about 25 percent CPU improvement with AIX on pSeries. So, our stringent lab results are proving to be a predictor of the trend in production. Of course, results will vary depending on factors such as mail file sizes, and user load, and so on.
Resources
About the authors | | | Razeyah Stephen, manager of the Domino Performance team, has been with Iris since 1998. She is a frequent presenter at Lotusphere. Previously Razeyah had five years of experience in the performance of storage subsystems. |
| | | Varadarajulu (Varada) Manavalan is a Domino IT Architect in IBM's IGS Service Delivery Organization. He started at IBM in 1997, and has worked in IT since the early 1990's. Varada can be reached at varlu@us.ibm.com. |
| | | Nirmala Venkatraman is a Senior Software Engineer at IBM, working as a Project Leader for the Domino Server Performance team. |
Rate this page
| |
