Text segmentation and organisation

The next three chapters of the book deal with how to extract linguistic information from the text input. This chapter can covers various pre-processing issues, such as how to find whole sentences in running text or how to handle text various markup or control information in the text. Chapter 5 describes the main processes of text analysis itself, such as how to resolve homograph ambiguity. 

Finally, Chapter 6 describes how to predict prosody information from an often impoverished text input. In many ways, this subject shares similarities with text analysis. There is an important difference however in that while we can view text analysis as a decoding problem with a clear right and wrong, prosody prediction has no strict right and wrong as we are attempting to determining prosody from an underspecified input. 

The job of the text analysis system is to take arbitrary text as input and convert this to a form more suitable to subsequent linguistic processing. This can be thought of as an operation where we try to bring a sense of order to the often quite daimting range of effects present in raw text. If we consider 

Pre-processing possible identification of text genre, character encoding issues, possible multi-lingual issues. 

Sentence splitting segmentation of the document into a list of sentences. 

---