IBM uses a wide range of industry-standard benchmarks to evaluate server performance. The following descriptions have been compiled from the respective Web sites for each benchmark. For more detailed information, visit the Web site for the benchmark.
 
TPC Benchmark C (TPC-C), approved in July of 1992, is an on-line transaction processing (OLTP) benchmark. TPC-C simulates a complete computing environment in which a population of users executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses. While the benchmark portrays the activity of a wholesale supplier, TPC-C is not limited to the activity of any particular business segment, but, rather represents any industry that must manage, sell, or distribute a product or service. However, it should be stressed that it is not the intent of TPC-C to specify how to best implement an Order-Entry system.
The performance metric reported by TPC-C measures the number of orders that can be fully processed per minute and is expressed in transactions per minute (tpmC). Two other metrics are associated: $/tpmC and the date of availability for the priced system.
 More info
TPC Benchmark E (TPC-E) is an On-Line Transaction Processing (OLTP) workload developed by the TPC. The TPC-E benchmark simulates the OLTP workload of a brokerage firm. The focus of the benchmark is the central database that executes transactions related to the firm’s customer accounts. Although the underlying business model of TPC-E is a brokerage firm, the database schema, data population, transactions, and implementation rules have been designed to be broadly representative of modern OLTP systems.
The TPC-E metrics are tpsE (transactions per second E) and $/tpsE. The tpsE metric is the number of trade-result transactions the server can sustain over a period of time. The price/performance metric, $/tpsE, is the total system cost for hardware, software, and maintenance, divided by the performance.
More info
TPC Benchmark H (TPC-H) models a decision support system. Decision support systems are used to analyze OLTP information for use in business decisions. This benchmark illustrates decision support systems that examine large volumes of data, execute queries with a high degree of complexity, and give answers to critical business questions.
The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour Performance Metric (QphH@Size), and reflects multiple aspects of the capability of the system to process queries. These aspects include the query processing power when queries are submitted by a single user, and the query throughput when queries are submitted by multiple concurrent users. Because of its impact on performance, the size of the database against which the queries are executed is also included in the TPC-H metric.
More info
TPC Benchmark App (TPC-App) is an application server and Web services benchmark. The workload is performed in a managed environment that simulates the activities of a business-to-business transactional application server operating in a 24x7 environment. TPC-App showcases the performance capabilities of application server systems. The workload exercises commercially available application server products, messaging products, and databases associated with such environments.
Two performance metrics are reported by TPC-App. The first is the Web Service Interactions per second (SIPS) per Application Server system. The second is the Total SIPS, which is the total number of SIPS for the entire tested configuration (SUT). Multiple Web Service Interactions are used to simulate the business activity of an online supplier, and each interaction is subject to a response time constraint.
More info
SPECweb2005 emulates users sending browser requests over broadband Internet connections to a Web server. It provides three new workloads: a banking site (HTTPS), an e-commerce site (HTTP/HTTPS mix), and a support site (HTTP). Dynamic content is implemented in PHP and JSP. The current version is 1.10.
More info
SPECjbb2005 is a benchmark for evaluating the performance of servers running typical Java business applications, JBB2005 represents an order processing application for a wholesale supplier. The benchmark can be used to evaluate performance of hardware and software aspects of Java Virtual Machine (JVM) servers.
More info
SPECjAppServer2004 is designed to measure the performance of J2EE 1.3 application servers. This benchmark includes an enhanced workload by adding a Web tier, JMS, and other changes to SPECjAppServer2002.
More info
SPEC CPU2006 is designed to provide performance measurements that can be used to compare compute-intensive workloads on different computer systems. SPEC CPU2006 contains two benchmark suites: CINT2006 for measuring and comparing compute-intensive integer performance, and CFP2006 for measuring and comparing compute-intensive floating point performance.
More info
NotesBench provides an objective method for evaluating the performance of different platforms running Lotus Domino Server. NotesBench generates a transactions-per-minute (tpm) metric, called a NotesMark for each test, along with a value for the maximum capacity (number of users) supported, and the average response time. Both performance metrics have an associated price/performance metric based on the cost of the hardware and software used to run the benchmark. The price/performance is not based on total cost of ownership (TCO), which includes warranty and maintenance coverage.
More Info
|
|
|
The Oracle Applications Standard Benchmark is focused on ERP applications and represents a mixed workload intended to model the most common transactions operating on the seven most widely used enterprise application modules. Definitions of transactions that compose the benchmark load were obtained through collaboration with functional consultants and are representative of typical customer workloads, with batch transactions representing 25% of the total workload.
More info
Baan Enterprise Resource Planning (ERP) is a suite of client/server business solutions, which integrates a company's business transactions into a single software solution. Baan ERP software provides applications that customers use to manage financials, accounting, sales and distribution, materials management, production planning, quality management, plant maintenance and human resource functions.
The Baan Benchmark Methodology is used to measure the performance of different computer system configurations running the standard BaanERP benchmark suite in two-tier client/server mode, which determines the exact number of Baan Reference Users (BRUs) that can be supported on a specific vendor's computer system.
SAP Standard Application Benchmarks test and prove the scalability of mySAP.com solutions. The benchmark results provide basic sizing recommendations for customers by testing new hardware, system software components, and Relational Database Management Systems (RDBMS). They also allow for comparison of different system configurations. The original SAP Standard Application Benchmarks have been available since R/3 Release 1.1H (April, 1993) and are now available for many SAP components.
The benchmarking procedure is standardized and well defined. It is monitored by the SAP Benchmark Council made up of representatives of SAP and technology partners involved in benchmarking. Originally introduced to strengthen quality assurance, the SAP Standard Application Benchmarks can also be used to test and verify scalability, concurrency and multi-user behavior of system software components, RDBMS, and business applications. All performance data relevant to system, user, and business applications are monitored during a benchmark run and can be used to compare platforms and as basic input for sizing recommendations.
More info
|
|
|
LS-DYNA, developed by the Livermore Software Technology Corporation, is a general-purpose transient dynamic finite element program capable of simulating complex real world problems. It is optimized for shared and distributed memory Unix, Linux, and Microsoft Windows platforms. LS-DYNA is being used by automobile, aerospace, manufacturing and bioengineering companies.
More info
The Fluent Benchmarks can be used to compare performance of different hardware platforms running the FLUENT flow solver. The broad physical modeling capabilities of FLUENT have been applied to industrial applications ranging from air flow over an aircraft wing to combustion in a furnace, from bubble columns to glass production, from blood flow to semiconductor manufacturing, from clean room design to wastewater treatment plants. The ability of the software to model in-cylinder engines, aero-acoustics, turbo-machinery, and multiphase systems has served to broaden its reach.
More info
vConsolidate, launched in April 2007 at the Intel Developer Forum (IDF), is designed to simulate real-world server performance in a typical environment, and to enable clients to compare the performance of multi-processor platforms in a virtualized environment.
More info
This test measures the messaging throughput of a single server, single-site topology. Its purpose is to measure the maximum throughput of a Microsoft Exchange Server on a specific hardware configuration. The benchmark results can provide a basis for comparing hardware and/or software products. The MAPI Messaging Benchmark (MMB3) measures throughput in terms of a specific profile of user actions, executed over an 8-hour working day. This benchmark is different from the MMB2 setting that was used with Exchange 2000 in that the rate of client requests is significantly greater for the MMB3 profile.
More info
Retired Benchmarks
SPECweb99 is a software benchmark product developed by the Standard Performance Evaluation Corporation (SPEC). It is designed to measure a system's ability to act as a web server for static and dynamic pages. This benchmark was retired in October 2005 and replaced by the WEB2005 suite.
SPECweb99_SSL, released in April 2002, added Secure Sockets Layer (SSL) Protocol support to SPECweb99. It tested secure Web server performance using HTTP 1.0/1.1 over the SSL Protocol. It was an extension of, rather than a replacement for, SPECweb99. This benchmark was retired in October 2005 and replaced by the WEB2005 suite.
SPECjbb2000 (Java Business Benchmark) was SPEC's first benchmark for evaluating the performance of server-side Java. Retired in January 2006, the benchmark is still available for purchase but no additional result submissions are being accepted and support is no longer offered. The current version of the benchmark is JBB2005.
SPEC HPC2002 was retired in June 2007 and is no longer being distributed, and no additional result submissions are being accepted. Published results remain available here for historical purposes
SPEC CPU2000 was retired in February 2007 and replaced by the CPU2006 suite.
More info
ECperf was retired in July 2002 and replaced by SPECjAppServer.
More info
Microsoft Exchange 2000 MMB2 was replaced by Exchange Server 2003 MMB3.
More info
TPC Benchmark W (TPC-W) is a transactional Web benchmark. This benchmark was retired April 28, 2005.
More info
| 
More info 