IBM uses a wide range of industry-standard benchmarks to evaluate server performance. The following descriptions have been compiled from the respective Web sites for each benchmark. For more detailed information, visit the Web site for the benchmark.
TPC Benchmark C (TPC-C), approved in July of 1992, is an on-line
transaction processing (OLTP) benchmark. TPC-C simulates a complete computing
environment in which a population of users executes transactions against a
database. The benchmark is centered around the principal activities (transactions)
of an order-entry environment. These transactions include entering and delivering
orders, recording payments, checking the status of orders, and monitoring the
level of stock at the warehouses. While the benchmark portrays the activity
of a wholesale supplier, TPC-C is not limited to the activity of any particular
business segment, but, rather represents any industry that must manage, sell,
or distribute a product or service. However, it should be stressed that it
is not the intent of TPC-C to specify how to best implement an Order-Entry
system.
The performance metric reported by TPC-C measures the number of orders that
can be fully processed per minute and is expressed in transactions per minute
(tpmC). Two other metrics are associated: $/tpmC and the date of availability
for the priced system.
 More
info
TPC Benchmark E (TPC-E) is an On-Line Transaction Processing (OLTP) workload
developed by the TPC. The TPC-E benchmark simulates the OLTP workload of a
brokerage firm. The focus of the benchmark is the central database that executes
transactions related to the firm’s customer accounts. Although the underlying
business model of TPC-E is a brokerage firm, the database schema, data population,
transactions, and implementation rules have been designed to be broadly representative
of modern OLTP systems.
The TPC-E metrics are tpsE (transactions per second E) and $/tpsE. The tpsE
metric is the
number of trade-result transactions the server can sustain over a period of
time. The
price/performance metric, $/tpsE, is the total system cost for hardware, software,
and
maintenance, divided by the performance.
More info
TPC Benchmark H (TPC-H) models a decision support system. Decision support
systems are used to analyze OLTP information for use in business decisions.
This benchmark illustrates decision support systems that examine large volumes
of data, execute queries with a high degree of complexity, and give answers
to critical business questions.
The performance metric reported by TPC-H is called the TPC-H Composite Query-per-Hour
Performance Metric (QphH@Size), and reflects multiple aspects of the capability
of the system to process queries. These aspects include the query processing
power when queries are submitted by a single user, and the query throughput
when queries are submitted by multiple concurrent users. Because of its impact
on performance, the size of the database against which the queries are executed
is also included in the TPC-H metric.
More
info
TPC Benchmark App (TPC-App) is an application server and Web services benchmark.
The workload is performed in a managed environment that simulates the activities
of a business-to-business transactional application server operating in a 24x7
environment. TPC-App showcases the performance capabilities of application
server systems. The workload exercises commercially available application server
products, messaging products, and databases associated with such environments.
Two performance metrics are reported by TPC-App. The first is the Web Service
Interactions per second (SIPS) per Application Server system. The second is
the Total SIPS, which is the total number of SIPS for the entire tested configuration
(SUT). Multiple Web Service Interactions are used to simulate the business
activity of an online supplier, and each interaction is subject to a response
time constraint.
More info
SPECweb2005 emulates users sending browser requests over broadband Internet
connections to a Web server. It provides three new workloads: a banking site
(HTTPS), an e-commerce site (HTTP/HTTPS mix), and a support site (HTTP). Dynamic
content is implemented in PHP and JSP. The current version is 1.10.
More
info
SPECjbb2005 is a benchmark for evaluating the performance of servers running
typical Java business applications, JBB2005 represents an order processing
application for a wholesale supplier. The benchmark can be used to evaluate
performance of hardware and software aspects of Java Virtual Machine (JVM)
servers.
More
info
SPECjAppServer2004 is designed to measure the performance of J2EE 1.3 application
servers. This benchmark includes an enhanced workload by adding a Web tier,
JMS, and other changes to SPECjAppServer2002.
More
info
SPEC CPU2006 is designed to provide performance measurements that can be
used to compare compute-intensive workloads on different computer systems.
SPEC CPU2006 contains two benchmark suites: CINT2006 for measuring and comparing
compute-intensive integer performance, and CFP2006 for measuring and comparing
compute-intensive floating point performance.
More
info
NotesBench provides an objective method for evaluating the performance of
different platforms running Lotus Domino Server. NotesBench generates a transactions-per-minute
(tpm) metric, called a NotesMark for each test, along with a value for the
maximum capacity (number of users) supported, and the average response time.
Both performance metrics have an associated price/performance metric based
on the cost of the hardware and software used to run the benchmark. The price/performance
is not based on total cost of ownership (TCO), which includes warranty and
maintenance coverage.
More Info
|
|
|
The Oracle Applications Standard Benchmark is focused on ERP applications and
represents a mixed workload intended to model the most common transactions
operating on the seven most widely used enterprise application modules. Definitions
of transactions that compose the benchmark load were obtained through collaboration
with functional consultants and are representative of typical customer workloads,
with batch transactions representing 25% of the total workload.
More info
Baan Enterprise Resource Planning (ERP) is a suite of client/server business solutions, which integrates a company's business transactions into a single software solution. Baan ERP software provides applications that customers use to manage financials, accounting, sales and distribution, materials management, production planning, quality management, plant maintenance and human resource functions.
The Baan Benchmark Methodology is used to measure the performance of different computer system configurations running the standard BaanERP benchmark suite in two-tier client/server mode, which determines the exact number of Baan Reference Users (BRUs) that can be supported on a specific vendor's computer system.
SAP Standard Application Benchmarks test and prove the scalability of mySAP.com
solutions. The benchmark results provide basic sizing recommendations for customers
by testing new hardware, system software components, and Relational Database
Management Systems (RDBMS). They also allow for comparison of different system
configurations. The original SAP Standard Application Benchmarks have been
available since R/3 Release 1.1H (April, 1993) and are now available for many
SAP components.
The benchmarking procedure is standardized and well defined. It is monitored
by the SAP Benchmark Council made up of representatives of SAP and technology
partners involved in benchmarking. Originally introduced to strengthen quality
assurance, the SAP Standard Application Benchmarks can also be used to test
and verify scalability, concurrency and multi-user behavior of system software
components, RDBMS, and business applications. All performance data relevant
to system, user, and business applications are monitored during a benchmark
run and can be used to compare platforms and as basic input for sizing recommendations.
More
info
|
|
|
LS-DYNA, developed by the Livermore Software Technology
Corporation, is a general-purpose
transient dynamic finite element program capable of simulating complex real
world problems. It is
optimized for shared and distributed memory Unix, Linux, and Microsoft Windows
platforms. LS-DYNA is being used by automobile, aerospace, manufacturing and
bioengineering
companies.
More
info
The Fluent Benchmarks can be used to compare performance of different hardware
platforms
running the FLUENT flow solver. The broad physical modeling capabilities of
FLUENT have been
applied to industrial applications ranging from air flow over an aircraft wing
to combustion in a
furnace, from bubble columns to glass production, from blood flow to semiconductor
manufacturing, from clean room design to wastewater treatment plants. The ability
of the software
to model in-cylinder engines, aero-acoustics, turbo-machinery, and multiphase
systems has
served to broaden its reach.
More
info
vConsolidate, launched in April 2007 at the Intel Developer Forum (IDF), is
designed to simulate real-world server performance in a typical environment,
and to enable clients to compare the performance of multi-processor platforms
in a virtualized environment.
More
info
This test measures the messaging throughput of a single server, single-site
topology. Its purpose is to measure the maximum throughput of a Microsoft Exchange
Server on a specific hardware configuration. The benchmark results can provide
a basis for comparing hardware and/or software products. The MAPI Messaging
Benchmark (MMB3) measures throughput
in terms of a specific profile of user actions, executed over an 8-hour working
day. This benchmark is different from the MMB2 setting that was used with Exchange
2000 in that the rate of client requests is significantly greater for the MMB3
profile.
More
info
Retired Benchmarks
SPECweb99 is a software benchmark product developed by the Standard Performance
Evaluation Corporation (SPEC). It is designed to measure a system's ability
to act as a web server for static and dynamic pages. This benchmark was retired
in October 2005 and replaced by the WEB2005 suite.
SPECweb99_SSL, released in April 2002, added Secure Sockets Layer (SSL) Protocol
support to SPECweb99. It tested secure Web server performance using HTTP 1.0/1.1
over the SSL Protocol. It was an extension of, rather than a replacement for,
SPECweb99. This benchmark was retired in October 2005 and replaced by the WEB2005 suite.
SPECjbb2000 (Java Business Benchmark) was SPEC's first benchmark for evaluating
the performance of server-side Java. Retired in January 2006, the benchmark
is still available for purchase but no additional result submissions are being
accepted and support is no longer offered. The current version of the benchmark
is JBB2005.
SPEC HPC2002 was retired in June 2007 and is no longer being distributed,
and no additional result submissions are being accepted. Published results
remain available here for historical purposes
SPEC CPU2000 was retired in February 2007 and replaced by the CPU2006 suite.
More
info
ECperf was retired in July 2002 and replaced by SPECjAppServer.
More
info
Microsoft Exchange 2000 MMB2 was replaced by Exchange Server
2003 MMB3.
More
info
TPC Benchmark W (TPC-W) is a transactional Web benchmark. This benchmark was
retired April 28, 2005.
More
info
|
More
info
