- About Us
Despite a variety of open source testing tools, until recently there wasn't an easy way to measure and compare the performance of two Linux-powered machines. Phoronix Test Suite (PTS), released this month, addresses this -- and how! Using the suite you can gauge and compare multiple Linux-powered machines to find out if a particular setup is better than another for a particular task, such as hosting a Web server or playing games.
Linux.com: Is the Phoronix Test Suite the same test tool you use internally for testing hardware on Phoronix? What was the motivation behind releasing the test suite?
Michael Larabel: With a few exceptions, yes, the public version of the Phoronix Test Suite is the same as what's being used internally. For instance, the Phoronix Test Suite has sensor monitoring support via ACPI and LM_Sensors. This includes support for monitoring the +3.33V, +5.00V, and +12.00V rails. However, as these sensors aren't always 100% accurate and there isn't uniform support across all motherboards, we still prefer using digital multimeters when monitoring the voltage rails in, say, power supply reviews. There are also a few areas left that aren't yet automated and covered by the Phoronix Test Suite, such as hooking in with Bootchart for measuring the boot performance, focusing on the qualitative side of video playback, and testing audio playback and recording. For an overwhelming majority of the articles, however, the Phoronix Test Suite is what's being used.
One of the motivations behind the Phoronix Test Suite was to open up our testing protocols so that the community can more easily run benchmarks under Linux.... It's disgusting to see bloggers and those on forums using glxgears and calling it a benchmark. The driving motivation, however, has been from independent hardware and software vendors. A number of them are open to having their engineering and quality assurance teams use Linux more in pre-production testing. However, because Linux isn't the primary OS of their customers, they aren't willing to devote many resources to it -- both for the human testing and then to develop a test suite. A number of manufacturers though have agreed to use the Phoronix Test Suite if it provides them with a way for autonomous testing, easy setup and maintainability, and a collection of tests that represent common applications used by those on the Linux desktop. Phoronix Test Suite 1.0 now provides all of that plus more.
Lc: What's involved in putting together a test profile?
ML: There is a brief tutorial included with the documentation for writing your own test profile. There is also a list of all supported XML tags. For a new test maintainer, the documentation is a good start, and then just looking at the existing test profile to see how those work.
For example, the profile from X-Plane, the multiplatform flight simulator, was the first test profile where the software vendor (Laminar Research) had worked with us to make X-Plane 9 a nice option for benchmarking. X-Plane 9.00 is a licensed pay program, though they do offer a free demo download, but even that is very large in size (somewhere around 900MB or so). What one of the developers did at Laminar Research was strip out all of the code and files (excess scenery, etc.) that aren't needed if all you're going to do is run benchmarks. He then created a standard timed demo file. This resulted in a version of X-Plane 9 specifically for benchmarking and is about half the demo's size.
Not only did Laminar produce this customized version, but it went on to document the various testing options. These options were how to run an FPS test, command-line arguments for adjusting various settings, and so forth. With the customized version of X-Plane and the information the company provided us, we were able to create this profile in just a matter of minutes.
[Another profile to look at] is one we created based upon LAME for measuring MP3 encode time. All of the profiles can easily be found and examined in pts/test-profiles/* and pts/test-resources/* [from the source tarball].
Lc: How's the feedback from hardware vendors and software developers?
ML: I've personally been very pleased with the feedback from hardware and software vendors surrounding the Phoronix Test Suite. Aside from Ben Supnik with Laminar Research providing X-Plane, Stefan Doesinger of CodeWeavers had provided a Wine compatibility test early on, and there are at least six other vendors we're working with at this time to push their own tests and suites. This includes software vendors who will be bringing their Windows benchmarks to Linux via PTS and one major company that has committed to publishing its regression test harness for Linux within PTS. There are also about 10 other independent contributors that have been working on test profiles thus far. Right now in git we're up to 58 test profiles.
Aside from vendors using the Phoronix Test Suite for hardware validation and using the software to automate their Linux tests, other companies are looking to this software for more reliable bug reports and customer testing, because it can serve as a common test environment between end users and companies. A great example of this can be found on our forums, where an end user believed he had experienced a problem with a driver and was able to clearly show a problem using PTS. The vendor could now look at this and its engineers could replicate the same software configuration and test settings. It'd be as simple as running "phoronix-test-suite benchmark ivanovic-22332-15328-27752" to see if they can reproduce the problem.
Lc: Any time frame on a searchable PTS Global interface? Apart from that, how will the suite be developed in forthcoming versions?
ML: There is a basic search interface, but a nice Web 2.0 version should arrive later this year. This includes many AJAX features for sorting the lists, paging, searching within a specific list, and more. PTS Global will also pick up some other features for further enhancing the collaboration possibilities and encouraging software projects to adopt the Phoronix Test Suite for their testing platform. I don't have a more defined time frame than "later this year," as the dedicated PTS server and the other Phoronix servers that PTS is piggybacking on are already being clobbered and chewing through globs of bandwidth for PTS Global, PTS downloads, and mirroring many of the tests that are downloaded using PTS. Once this infrastructure gets upgraded, I'll be more comfortable rolling out rich Internet features in PTS Global.
The first major update to Phoronix Test Suite "Trondheim" will incorporate a module/plugin framework (currently in development) as well as support for new areas of testing. The module framework will allow the capabilities of pts-core to be expanded and opened up in new ways. Two of the planned modules include one for LCDproc for reporting the test being run and its results to an LCD panel or keyboard display. The other plugin is for setting up PTS to run in batch mode whenever there is a new commit in a git tree for the software being tested (say the kernel, Mesa, or a driver). The new tests support will focus upon areas such as qualitative video playback comparisons by capturing and saving the screen, algorithms for attempting to determine screen artifacts and other rendering oddities, and finer areas of graphics such as anti-aliasing and anisotropic filtering. Through the interaction with the community and various hardware and software vendors, we'll also be working to better define the Phoronix Certification and Qualification Suites, as well as a set of stress tests for pushing systems to their limits.