Sound and Music Computing

Home » Wiki » Dafxtra2008

DAFxTRa 2008

Welcome to the planning wiki for DAFxTRa 2008.

DAFxTRa 2008 (DAFx Transformation RAting) is a new initiative aimed at evaluating and comparing algorithms for audio effects. Our goal is to have the main evaluation in September 2008 during DAFx-08 but to make it happen we need the involvement of the audio effects research community.
Most audio effects tasks do not afford an objective measure of ground truth, like in MIR, and thus we have to define specific evaluation strategies. We want to do it by involving the developers of algorithms interested in participating and by specifying with them the evaluation process. The participants should learn from the process and the whole DAFx community should benefit from it.

Our initial aims are the following:

  1. Propose several audio effects categories. Initial proposals: time-scaling, pitch-shifting, source separation, morphing, distortion effects and deconstruction.
  2. Select the categories for which there is a sufficient number of participants.
  3. Select sounds from Freesound.org to be used as test sounds for each category.
  4. Define the evaluation procedure for each category.
  5. Ask the participants to submit the transformed sounds (not the algorithms).
  6. Perform the evaluation both live at DAFx-08 and also online in Freesound.
  7. Publish the results of the evaluation.

Proposed timeline:

The evaluations proposed for this year's DAFx conference have been postponed to better plan for next year. Instead, we will meet at the conference on Wed, Sept 3rd during the 17:30-18:30 slot to discuss the future of DAFxtra.

Slides from DAFx08 presentation

Ideas for Categories:

Here are some categories that we are thinking of running:

  1. Time-scaling
  2. Pitch-shifting
  3. Freeform Cross-Synthesis
  4. Freeform Non-linear effects/ Distortion
  5. Freeform Deconstruction
  6. Voice Conversion
Evaluation Previous Work

These will change depending on who wants to participate and run evaluations. Some of the initial ones we have tabled for various reasons: Source Separation, superceded by SiSEC 2008, community-based source sep eval; Audio Coding, already done, but an important source for inspiration.

 

If you are interested in participating or in getting involved in the process join the DAFxTRa mailing list for an open discussion and introduce yourself in DafxtraPeople. Results of the discussion and organizational details of the evaluation will be posted here.

This wiki page is maintained by Graham Coleman