Classification and recognition with direct segment models

Segment based direct models have recently been used to improve the output of existing state-of-the-art speech recognizers. To date, however, they have relied on an existing HMM system to provide segment boundaries. This paper takes initial steps at using these models on their own, first by developing a segment-based maximum entropy phone classifier, and then by utilizing the features in a segmental conditional random field for recognition. To produce a feature representation that is independent of segment length, we utilize a set of ngram features based on vector-quantized representations of the acoustic input. We find that the models are able to integrate information at different granularities and from different streams. Contextual information from around the segment boundaries is particularly important. We obtain competitive results for TIMIT phone classification, and present initial recognition results.