Evaluation Previous Work

Previous work in Subjective Audio Testing, to be used for DAFxTra Evaluation 2008

Graham Coleman


1. Mushram (2005) and ITU BS.1534 (2003)

The user-documentation for Mushram, a Matlab toolbox for subjective testing, implements and cites a standard MUSHRA (MUlti Stimulus test with Hidden Reference and Anchor) from the International Telecommunications Union (ITU) - ITU BS.1534. The user documentation also explains some aspects of the standard for the purpose of "the comparison of high quality reference sounds with several lower quality test sounds". Principles stressed in the manual include selection of the test material, selection of the test subjects, separate training and evaluation phases of the test, the requirement of at least one test sound being rated 100 (ostensibly the reference sound), anchor sounds computed similarly across all experiments, a post-screening to reject over- or under-critical subject results, and finally a statistical analysis to produce the results of the test. The standard itself is available in electronic form from the ITU for a fee.

2. SQAM (1988)

t3253 Sound Quality Assessment Material Recordings for Subjective Tests - User's Handbook for the EBU SQAM Compact Disk - first edition 198

The Sound Quality Assessment Material (SQAM) is primarily a set of sounds produced by the European Broadcasting Union (EBU) intended for testing digital audio reproduction systems. The sounds were selected to cover a range of input material (voice, instrument, music) and to measure certain types of distortion commonly produced by such systems (A/D-D/A nonlinearity, aliasing distortion, bit errors, bitrate reduction, dynamic range or frequency range  reduction, post-processing overload, program modulation noise, or modification in stereo image). The most recent page, including sounds, can be found here: Tech3253 (report) (older page, broken link to report). The report mainly discusses the sounds and does not give referencesm or discuss the evaluation methodology, which perhaps are discussed in another tech report.

However, there seem to be some other EBU tech3000 reports that may possibly be of interest:

t3309 Evaluations of Cascaded Audio Codecs 2005
t3270 Euroradio Measurements CD for testing stereophonic sound programme circuits 1995
t3276 Listening Conditions for the Assessment of Sound Programme Material: Monophonic and Two-channel Stereophonic - second edition 1998
t3276-s1-2004 Listening Conditions for the Assessment of Sound Programme Material - Supplement 1, Multichannel Sound - second edition 2004
t3286 Assessment Methods for the Subjective Evaluation of the Quality of Sound Programme Material - Music - first edition 1997
t3286-s1 Assessment Methods for the Subjective Evaluation of the Quality of Sound Programme Material - Supplement 1, Multichannel - first edition 2001
t3287 Parameters for the subjective Evaluation of the Quality of Sound Programme Material - Music (PEQS) 1997
t3324 EBU evaluations of multichannel audio codecs 2007

3. PEAQ (2001)

Another standard from the ICU, BS.1387-1 (Method for objective measurements of perceived audio quality) Perceptual Evaluation of Audio Quality (PEAQ) appears to provide objective methods (numerical tests) rather than subjective methods of quality evaluation. The standard is downloadable for a fee.


Perceptual Evaluation of Speech Quality (PESQ) seems to be a speech-specific successor to PEAQ, and provides a quality rating of telephony systems.

An academic description of the standard is available in a paper by Rix:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=941023

(NOTE: expand sections 4 and on)


4. PERCEPTUAL SPEECH QUALITY ASSESSMENT – A REVIEW Anthony Rix link

A brief review of subjective tests and so-called intrusive models (objective or numerical tests).

5. Tech 3286-E Assessment methods for the subjective evaluation of the quality of
sound programme material Supplement 1- Multichannel

Gives in-depth discussion, including questionaires and evaluation dimensions for multichannel sound system evaluation.

6. Listening Evaluation - Sound and Video Contractor

Popular article in trade press describes in detail subjective loudspeaker testing methodology and some results, including references to published studies.

7. Lecture notes on Sound Quality from Prof. Matti Karjalaine.

Professor from Helsinki University of Technology / Laboratory of Acoustics and Audio Signal Processing gives an overview of sound quality measurement.

8. Thesis of Thilo Thiede

Relating to perceptual objective measures. Perfe recommends chapter 5 and 6, which mention his objective quality testing methodology.

Other Related Evaluation Contests

1. MIREX

http://www.music-ir.org/mirex/2008/index.php/Main_Page

2. NIPS Challenges

http://www.nipsfsc.ecs.soton.ac.uk/

3. An ICA'99 source separation challenge

http://sound.media.mit.edu/ica-bench/