photo

Ed Schofield: Publications

PhD Thesis

[*] Fitting maximum-entropy models on large sample spaces. Imperial College London, 2006

Conference Papers

[6] Detailed multidimensional analysis of our acoustical environment. Forum Acusticum 2005

[5] Automated image annotation using global features and robust nonparametric density estimation. CIVR 2005

[4] Fast parameter estimation for joint maximum entropy language models. Interspeech 2004

[3] A speech interface for open-domain question answering. ACL 2003

[2] Language models for questions. EACL 2003

[1] On interfaces for mobile information retrieval. MobileHCI 2002

Tutorials

[1] Concepts in pattern recognition, November 2001

Technical Reports

[1] Skin tracking for virtual reality systems, August 2001

Details

PhD Thesis

Fitting maximum-entropy models on large sample spaces

Department of Computing, Imperial College London, June 2006; revised January 2007

Download thesis (2.8MB)

Abstract:

This thesis investigates the iterative application of Monte Carlo methods to the problem of parameter estimation for models of maximum entropy, minimum divergence, and maximum likelihood among the class of exponential-family densities. It describes a suite of tools for applying such models to large domains in which exact computation is not practically possible.

The first result is a derivation of estimators for the Lagrange dual of the entropy and its gradient using importance sampling from a measure on the same probability space or its image under the transformation induced by the canonical sufficient statistic. This yields two benefits. One is the flexibility to choose an auxiliary distribution for sampling that reduces the standard error of the estimates for a given sample size. The other is the opportunity to re-weight a fixed sample iteratively to reduce the computational burden for each iteration.

The second result is a derivation of matrix–vector expressions for these estimators. Importance-sampling estimates of the entropy dual and its gradient can be computed efficiently from a fixed sample; the computation is dominated by two matrix–vector products involving the same matrix of sample statistics.

The third result is an experimental study of the application of these estimators to the problem of estimating whole-sentence language models. The use of importance sampling in conjunction with sample-path optimization is feasible whenever the auxiliary distribution does not too severely under-represent any linguistic features under constraint. Parameter estimation is rapid, requiring a few minutes with a 2006-vintage computer to fit models under hundreds of thousands of constraints. The procedure is most effective when used to minimize divergence (relative entropy) from existing baseline models, such as n-grams estimated by traditional means, rather than to maximize entropy under constraints on the probabilities of rare n-grams.

BibTeX:

@phdthesis{schofield06fitting,
    author = "Edward J. Schofield",
    title = "Fitting maximum-entropy models on large sample spaces",
    school = "Department of Computing, Imperial College London",
    address = "London, England",
    month = jun,
    year = 2006,
    url = "www.edschofield.com/publications/schofield06fitting.pdf",
}

Conference Papers

[6] Detailed multidimensional analysis of our acoustical environment. With Márian Képesi and Luis Weruaga.

Proceedings of Forum Acusticum, September 2005, Budapest.

Download paper

Abstract:

The most often-used time-frequency analysis tool, the Short-Time Fourier Transform, suffers from blurry harmonic representation when analysing acoustic sources with changing frequency. This phenomenon is always present in the analysis of underwater and geophysical signals, the sounds of bats, whales, birds and other animals, engine noises, and human conversations. For time-frequency analysis of real-world acoustic signals like these this paper introduces a technique based on the the Short-Time Harmonic Chirp Transform (STHChT) with a noise-level-independent feedback-like estimation of frequency changes. The basis of this adaptive transform comprises quadratic chirps that follow the frequency trajectory segment by segment. The frequency tracking method is based on the harmonic structure of the source, derived from its spectral representation, whose performance is in turn strongly enhanced by the use of the STHChT. This combination of analysis tools offers a more precise time-frequency representation of acoustic signals than state-of-the- art time-frequency analysis techniques. Comparative evaluation results between the proposed STHChT and popular time-frequency techniques reveal an improvement in time-frequency localisation and a finer spectral representation.

BibTeX:

@inproceedings{kepesi05detailed,
    title = "Detailed multidimensional analysis of our acoustical environment",
    author = "Mari\'an K\'epesi and Edward J. Schofield and Luis Weruaga",
    booktitle = "Proceedings of Forum Acusticum",
    month = sep,
    year = 2005,
    address = "Budapest, Hungary",
    url = "www.edschofield.com/publications/kepesi05detailed.pdf",
}

[5] Automated image annotation using global features and robust nonparametric density estimation. With Alexei Yavlinsky

Proceedings of the Computer Image and Video Retrieval conference (CIVR), July 2005, Singapore

Download paper

Abstract:

This paper describes a simple framework for automatically annotating images using non-parametric models of distributions of image features. We show that under this framework quite simple image properties such as global colour and texture distributions provide a strong basis for reliably annotating images. We report results on subsets of two photographic libraries, the Corel Photo Archive and the Getty Image Archive. We also show how the popular Earth Mover's Distance measure can be effectively incorporated within this framework.

BibTeX:

@inproceedings{yavlinsky05automated,
    author = {Alexei Yavlinsky and Edward J. Schofield and Stefan R\"uger},
    title = {Automated Image Annotation Using Global Features and Robust Nonparametric Density Estimation},
    booktitle = {Proceedings of the 4th International Conference on Image and Video Retrieval (CIVR)},
    pages = {507--517},
    year = 2005,
    month = jul,
    address = {Singapore},
    editor = {D. Polani and B. Browning and A. Bonarini and K. Yoshida},
    publisher = {Springer-Verlag}, 
    volume = 3568,
    series = {Lecture Notes in Computer Science (LNCS)},
    isbn = {3-540-27858-3},
    url = "www.edschofield.com/publications/yavlinsky05automated.pdf",
}

[4] Fast parameter estimation for joint maximum entropy language models

Proceedings of Interspeech / ICSLP, October 2004, Jeju

Download paper

Abstract:

This paper discusses efficient parameter estimation methods for joint (unconditional) maximum entropy language models such as whole-sentence models. Such models are a sound framework for formalizing arbitrary linguistic knowledge in a consistent manner. It has been shown that general-purpose gradient-based optimization methods are among the most efficient algorithms for parameter estimation for several tasks in natural language processing. This paper applies gradient methods to whole-sentence language models and other domains whose sample spaces are infinite or practically innumerable and require simulation. It also presents Open Source software for easily fitting and testing joint maximum entropy models.

[3] A speech interface for open-domain question answering. With Zhiping Zheng

Proceedings of the Association of Computational Linguistics conference, (ACL) 2003, Sapporo

Download paper

Abstract:

Speech interfaces to question-answering systems offer significant potential for finding information with phones and mobile networked devices. We describe a demonstration of spoken question answering using a commercial dictation engine whose language models we have customized to questions, a Web-based text-prediction interface allowing quick correction of errors, and an open-domain question-answering system, AnswerBus, which is freely available on the Web. We describe a small evaluation of the effect of recognition errors on the precision of the answers returned and make some concrete recommendations for modifying a question-answering system for improving robustness to spoken input.

BibTeX:

@inproceedings{schofield03aspeech,
    author = "Edward J. Schofield and Zhiping Zheng",
    title = "A speech interface for open-domain question answering",
    year = 2003,
    month = jul,
    booktitle = "Proceedings of the Association for Computational Linguistics (ACL)",
    address = "Sapporo, Japan",
    url = "www.edschofield.com/publications/schofield03aspeech.pdf",
}

[2] Language models for questions

Proceedings of the European Association for Computational Linguistics conference (EACL) 2003, Budapest

Download paper

Abstract:

Natural-language question answering is a promising interface for retrieving information in mobile contexts because it by-passes the problem of presenting documents and interim search results on a small screen. This paper considers language-models suitable for rapid predictive text-input and spoken input of natural-language questions. It describes a varied corpus of fact-seeking questions posed by users online and analyzes its structure. We find it to be highly constrained lexically despite its wide spectrum of topics, with a per-word perplexity less than 47 with around 2.6% of words in the test set out-of-vocabulary. One implication is that predictive interfaces can greatly speed up the input of natural-language questions with a keypad or stylus. Another is that automatic speech-recognition of such questions can be quite accurate.

BibTeX:

@inproceedings{schofield03language,
    author = "Edward J. Schofield",
    title = "Language models for questions",
    year = 2003,
    month = apr,
    booktitle = "Proceedings of the European Association for Computational Linguistics (EACL)",
    address = "Budapest, Hungary",
    url = "www.edschofield.com/publications/schofield03language.pdf",
}

[1] On Interfaces for Mobile Information Retrieval. With Gernot Kubin

Proceedings of the conference on Human-Computer Interaction with Mobile Devices (MobileHCI), September 2002, Pisa

Download paper

Abstract:

We consider the task of retrieving online information in mobile environments. We propose question answering as a more appropriate interface than page-browsing for small displays. We assess different modalities for communicating using a mobile device with question-answering systems, focusing on speech. We then survey existing research in spoken information retrieval, present some new findings, and assess the feasibility of the endeavor.

BibTeX:

@inproceedings{schofield02oninterfaces,
  author = "Edward J. Schofield and Gernot Kubin",
  title =  "On Interfaces for Mobile Information Retrieval",
  booktitle = "Proceedings of the 4th International Symposium on Human Computer Interaction with Mobile Devices ({MobileHCI})",
  year = 2002,
  editor = "Fabio Patern\`{o}",
  number = 2411,
  series = "Lecture Notes in Computer Science (LNCS)",
  pages = "383--387",
  month = sep,
  publisher = "Springer-Verlag",
  url = "www.edschofield.com/schofield02oninterfaces.pdf",
}

Tutorials

[1] Concepts in pattern recognition

A presentation to the Telecommunications Research Center Vienna, November 2001.

Download slides

Abstract:

The recognition of patterns is essential for many budding applications of technology, including computer vision, speech, and financial forecasting. This talk presents an overview of some of the concepts in automatic pattern recognition. The discussion focuses on two examples in computer vision and econometrics, describing their relation and motivating a general mathematical framework for the learning of patterns. It introduces kernels and support vector-methods, and reviews some recent research suggesting that feed-forward neural networks are obsolete.