Previous work in Subjective Audio Testing, to be used for DAFxTra Evaluation 2008

Graham Coleman

1. Mushram (2005) and ITU BS.1534 (2003)

The user-documentation for Mushram, a Matlab toolbox for subjective testing, implements and cites a standard MUSHRA (MUlti Stimulus test with Hidden Reference and Anchor) from the International Telecommunications Union (ITU) - ITU BS.1534. The user documentation also explains some aspects of the standard for the purpose of "the comparison of high quality reference sounds with several lower quality test sounds". Principles stressed in the manual include selection of the test material, selection of the test subjects, separate training and evaluation phases of the test, the requirement of at least one test sound being rated 100 (ostensibly the reference sound), anchor sounds computed similarly across all experiments, a post-screening to reject over- or under-critical subject results, and finally a statistical analysis to produce the results of the test. The standard itself is available in electronic form from the ITU for a fee.

2. SQAM (1988)

t3253

Sound Quality Assessment Material Recordings for Subjective Tests - User's Handbook for the EBU SQAM Compact Disk - first edition

198

The Sound Quality Assessment Material (SQAM) is primarily a set of sounds produced by the European Broadcasting Union (EBU) intended for testing digital audio reproduction systems. The sounds were selected to cover a range of input material (voice, instrument, music) and to measure certain types of distortion commonly produced by such systems (A/D-D/A nonlinearity, aliasing distortion, bit errors, bitrate reduction, dynamic range or frequency range  reduction, post-processing overload, program modulation noise, or modification in stereo image). The most recent page, including sounds, can be found here: Tech3253 (report) (older page, broken link to report). The report mainly discusses the sounds and does not give referencesm or discuss the evaluation methodology, which perhaps are discussed in another tech report.

However, there seem to be some other EBU tech3000 reports that may possibly be of interest:

t3309

Evaluations of Cascaded Audio Codecs

2005

t3270

Euroradio Measurements CD for testing stereophonic sound programme circuits

1995

t3276

Listening Conditions for the Assessment of Sound Programme Material: Monophonic and Two-channel Stereophonic - second edition

1998

t3276-s1-2004

Listening Conditions for the Assessment of Sound Programme Material - Supplement 1, Multichannel Sound - second edition

2004

t3286

Assessment Methods for the Subjective Evaluation of the Quality of Sound Programme Material - Music - first edition

1997

t3286-s1

Assessment Methods for the Subjective Evaluation of the Quality of Sound Programme Material - Supplement 1, Multichannel - first edition

2001

t3287

Parameters for the subjective Evaluation of the Quality of Sound Programme Material - Music (PEQS)

1997

t3324

EBU evaluations of multichannel audio codecs

2007

3. PEAQ (2001)

Another standard from the ICU, BS.1387-1 (Method for objective measurements of perceived audio quality) Perceptual Evaluation of Audio Quality (PEAQ) appears to provide objective methods (numerical tests) rather than subjective methods of quality evaluation. The standard is downloadable for a fee.

Perceptual Evaluation of Speech Quality (PESQ) seems to be a speech-specific successor to PEAQ, and provides a quality rating of telephony systems.

An academic description of the standard is available in a paper by Rix:

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=941023

(NOTE: expand sections 4 and on)

4. PERCEPTUAL SPEECH QUALITY ASSESSMENT – A REVIEW Anthony Rix link

A brief review of subjective tests and so-called intrusive models (objective or numerical tests).

5. Tech 3286-E Assessment methods for the subjective evaluation of the quality of
sound programme material Supplement 1- Multichannel

Gives in-depth discussion, including questionaires and evaluation dimensions for multichannel sound system evaluation.

6. Listening Evaluation - Sound and Video Contractor

Popular article in trade press describes in detail subjective loudspeaker testing methodology and some results, including references to published studies.

7. Lecture notes on Sound Quality from Prof. Matti Karjalaine.

Professor from Helsinki University of Technology / Laboratory of Acoustics and Audio Signal Processing gives an overview of sound quality measurement.

8. Thesis of Thilo Thiede

Relating to perceptual objective measures. Perfe recommends chapter 5 and 6, which mention his objective quality testing methodology.

Other Related Evaluation Contests

1. MIREX

2. NIPS Challenges

http://www.nipsfsc.ecs.soton.ac.uk/

3. An ICA'99 source separation challenge

http://sound.media.mit.edu/ica-bench/