Dan Ellis: Research Projects:

Audio Feature Toolkit

It is generally recognized that the soundtrack to video carries extensive information of value to content analysis and indexing applications. Many labs are investigating these kinds of issues, and several have looked at soundtrack analysis as a source of cues. However, many more researchers might like to use audio information, but find the prospect of developing a whole new set of primitives, to look at a modality that is not their primary concern, rather discouraging.

Thanks to the long history of speech recognition, computer music, audio compression etc., there are a number of different useful feature types that have been defined. While this is still an active research area, there are plenty of applications where the development of novel features is not needed: using a well-established feature would probably be fine.

In addition to features, there are numerous classification and segmentation algorithms, many derived from speech recognition, that are relatively mature, and would be useful to a wide range of researchers if an easy-to-use implemetation was available.

This project is about developing an integrated suite of sound analysis and classification tools with the goal of providing something convenient and useful for multimedia content researchers who wish to avoid the investment of developing their own code. Thus, the resulting programs should be:

• Flexible to support a wide range of known and unforseen applications. This can be achieved by a modular structure, with different program pieces that can be plugged together in different sequences, perhaps.
• Extensible to allow the addition of new processing schemes with the minimum additional effort. (We always say this, but the number of people who end up actually writing code to extend a system is often pretty low!)
• Efficient: Since the target application is processing of large multimedia archives, the toolkit must be able to scale well. This means that the signal processing should be efficient and lightweight, and the resulting data files should be small. Running in much longer than real time is probably not much use.
• General-purpose: We are not looking only at studio speech, but want to be able to do something useful with any kind of soundtrack -- background music, home video, crowd noise, commercials, whatever.
• Robust: Algorithms that break when the input strays outside some narrow space of signals (for instance, speech recognition) aren't particularly valuable in a domain where there are very few guarantees about what content will be encountered. Better to do something less clever, but that works reasonably well in all possible circumstances.
• Mixture-friendly: Real sound almost always consists of multiple sound sources mixed together, and the more a representation can respect this -- either by calculating global attributes that vary sensibly when presented with mixtures, or by working towards the separation of attributes of different sources -- the more valuable it will be.

Starting points

A place to start would be to be able to reproduce the common speech/music/other segmentation that has been reported by various authors (e.g. Hain et al.'s description of Broadcast News segmentation) via manually-labeled Gaussian Mixture Models and Hidden Markov Model decoding.

We already have feacalc to calculate low-order warped-frequency cepstra (including MFCCs), although it was built for large collections of short clips (speech databases) rather than a few huge soundtracks.

Related work

Other people have identified the value of a common set of audio processing tools, and also It's a natural urge to package up one's tools and hope that all the invested effort might benefit someone else. There are a number of current related efforts that this work might beneficially exist within or alongside:

• MPEG-7: The MPEG metadata standardization effort includes specification of a wide range of abstract descriptors for audio, at all kinds of levels, including (I believe) example implementations of code to extract them.
• Silvia Pfeiffer's MPEG Maaate is a framework with very similar goals to the project described here.