Benchmarks are something of a curate’s egg. The underlying principles of testing are almost always sound, but the relevance of results to real-world situations is not always what it might be.

The most apposite example is probably the venerable server benchmark TPC-C, which its creator, the Transaction Processing Council, has now decided to replace with something that has more bearing on the operational issues modern enterprises face.

TCP-C’s place will be taken by TPC-E, which alters the basic premise of the benchmark to something more applicable to a transaction-focused database environment. TPC-E should give a much better analogy of real-world internet-based transaction management issues where many different events can create time-shifted, fractured transactions that need to be completed and assessed.

The benchmark also uses data culled from such sources as the US Census and the New York Stock Exchange to generate names, addresses and company information, with the aim of inducing the types of operational skews found in working IT environments.

TPC-E is also targeting the Achilles heel of the TPC-C benchmark: hardware utilisation. This issue has been open to manipulation by server manufacturers in the past, to the point where significantly unreal server installations were created purely to achieve a good benchmark result.

For example, TPC-C results could be improved by putting a larger number of smaller hard disk drives into a server – more than any IT manager (or chief finance officer) would countenance.

Because this type of infrastructure is so complex it required an army of pointy-headed rocket-scientists to tweak and cajole the system to perform at all, let alone well. As such, only the major vendors could even contemplate the cost of attempting a TPC-C benchmark, for the installation would require several thousand disk drives, and a price-tag running into eight digits.

TPC-E will insist on a more balanced mix of disk and processor usage, at the same time cutting the cost of running the benchmark and making the process faster to run and audit.

The results will be published as tpsE – transaction per second and so a measure of server performance – and $/tpsE, which divides the total cost of hardware, software and maintenance by the performance number.

All of which should be able to produce a better CTTRW (Closeness To The Real World) rating.