Server performance indicators: dial out the server evaluation system fog

Users always want a simple and efficient metric to quantify the evaluation server system, as a basis for selection. But in fact, the server's system performance is difficult to measure with one or two indicators. A number of server evaluation systems, including TPC, SPEC, SAP SD, Linpack, and HPCC, provide a quantitative evaluation of processor performance, server system performance, and business application performance up to the performance of high-performance computers. In so many standards, how do users choose the evaluation system that best suits their application environment? Here, we have chosen TPC and SPEC with a wide range of applications for an in-depth introduction.

■ Going out of the wrong zone

Going deep into the TPC-C indicator

The TPC system is one of the most influential benchmarks, especially in the past two years, the domestic media reports on the TPC indicator Massive. But how many users really understand the meaning? This article uses TPC-C as an example to let users know more about this benchmark.

The tpmC value is widely used at home and abroad to measure the transaction processing capabilities of server systems. But what exactly is the tpmC value? I have asked this question to some users, professional media reporters and even some foreign large companies, but the accuracy of the answer is far from the popularity of tpmC. Many people mistakenly write it as TPMC, and even confuse it with TPC organizations.

TPC (Transaction Processing Performance Council) is a non-profit organization created by dozens of member companies and is headquartered in the United States. The members of TPC are mainly computer hardware and software manufacturers, not computer users. Their function is to develop standard specifications, performance and price metrics for business application benchmarks, and to manage the release of test results.

TPC does not give the code for the benchmark, but only the standard specification for the benchmark. Any manufacturer or other tester can optimally construct their own test system (test platform and test program) according to the specifications. In order to ensure the integrity of the test results, the testee (usually the manufacturer) must submit to the TPC a complete report (Full Disclosure Report), including the detailed configuration of the system under test, the classification price and the five-year maintenance fee. Total cost. The report must be verified by an auditor authorized by the TPC (the TPC itself does not perform an audit). TPC has fewer than 10 auditors worldwide, all in the United States.

TPC has launched 11 benchmark programs, namely TPC-App, TPC-H, TPC-C, TPC-W, outdated TPC-A, TPC-B, TPC-D and TPC. -R, and TPC-S (Server Special Test Benchmark Program), TPC-E (Large Enterprise Information Service Test Benchmark Program) and TPC-Client/Server, which were abandoned because they were not accepted by the industry. At present, TPC-C, which is the most popular & popular, is the benchmark program for online transaction processing (OLTP). It was completed in July 1992 and was gradually accepted by the industry.

TPC-C uses three performance and price metrics, where performance is measured by tpmC (transactions per minute, tpm) and C is the C benchmark in TPC. It is defined as the number of new orders processed by the system per minute. TPC-C is also often expressed in terms of system performance and price, in units of $/tpmC, which is derived from the total system price (in US dollars) /tpmC.

Interpreting tpmC

It is not difficult to know from the definition of TPC-C that this benchmark is used to measure the performance of the entire IT system, not the standard for evaluating a server or a hardware system. Moreover, the value of tpmC is directly affected by each link. The right table probably shows the impact of system settings on the tpmC test. The “IT system” here includes servers, peripherals (such as hard drives or RAID), server-side operating systems, database software, clients and their operating systems, database software and network connections. Therefore, how to interpret the tpmC value will vary greatly depending on the procurement needs.

tpmC test indicators and hardware correlations Transaction type Complexity Frequency of occurrence Access to the number of tables (memory, disk IO related) Average logical IO number CPU load (%) New orders Complex transactions 45% 8 46 53 Payments Complex transactions 43% 4 8 11 Delivery 4% 4 70 8 Order inquiry Simple transaction 4% 3 12 1 Stock inquiry Complex transaction 4% 3 401 27 Of the above 5 transactions, except for the delivery transaction is an after-the-fact batch The other four are online transactions. It should be noted that while processing new orders, the system has to deal with other four types of transaction requests. In general, a new order request cannot exceed 45% of a full transaction request, so when a system's performance is 1000tpmC, it actually processes more than 2,000 requests per minute. Source: www.tpc.org

Take the server as an example. In many manufacturers' TPC test systems, the price of the server is only 25% or less of the total price of the system, and the price of the hard disk may account for more than 30% of the total price, because the TPC-C requires the system to be tested to be saved for 180 days. The transaction record (this trend will be intensified from some of the latest TPC-C test results). If the same server is used in the user's environment, the tpmC value reported by the manufacturer is of little significance, because the actual system of the user is quite different from the system originally used by the manufacturer for TPC testing. When the same host is used in different systems, the tpmC value may vary considerably, and many users are not aware of this yet.

Especially for server procurement users, the tpmC metric is more about measuring the performance area from the client to the terminal network (as shown on the left), rather than the server-to-enterprise network that is usually mistaken for the server. performance. It can be seen that if the user is to establish a new business system, then it is better to draw on the performance indicators of tpmC. If you only purchase some or some hardware devices, you need to refer to more indicators.

For users who are fascinated by tpmC values, a real question is whether the cost of implementing high tpmC metrics can be tolerated. After all, TPC-C is a very complicated benchmark. It takes a lot of resources to do a rigorous test. Manufacturers usually don't report all the money and time spent on TPC.

In fact, because tpmC (the other evaluation indicators are mostly the same) is greatly affected by system optimization, the industry has been controversial about whether it should be optimized in the test. In 1993, experts from the three major IT companies raised objections to the optimization of OLTP performance testing — DEC's Jim Gray (now TPC-C committee expert) and Walt Kohler, Tianten (TANDEN) Charles Levine and IBM's Steve Kiss jointly commented on whether OLTP performance test metrics should be artificially optimized:

& ldquo; A good performance test metric does not depend on optimization in special situations. The ideal situation should be: the nature and capabilities of a test metric during the test, and the optimizations made to the test can indeed provide performance improvements for real-world users. However, contrary to expectations, some manufacturers are forced by the huge market pressure brought by the standard test indicators. They spend a lot of energy on how to improve the test results while reducing the price ratio, regardless of whether all the optimizations made are given. Users bring real benefits. ”

Evaluation organization website:

http://tpc.org/

http://www.spec.org/

Storage:

http://www.storageperformance.org