Natural-language text

Having dealt with non-natural language text, we now turn our attention to the issue of how to find the words fi om the text when that text does in fact encode natural language. For many words, this process is relatively straightforward if we take a word such as walk, we will have its textual form walk listed in the lexieon, and upon finding the a token walk in the text we can be fairly sure that this is in fact a vmtten representation of walk. At worst we might be presented with a capitalised version Walk or an upper ease version WALK but these are not too diffieult to deal with either. If all words and all text behaved in this manner the problem would be solved, but of eourse not all cases are so simple, and we ean identify the main reasons for this  

Alternative spellings A number of words have more than one spelling. Often in English this is related to American vs non-American conventions so that we have tyre and tire and honour and honor. In many cases these differences are systematic, for instance with the suffix -ISE which can be encoded as organise or organize. Often it is possible to list 

Abbreviations Many words occur in abbreviated form, so for instance we have dr for doctor and St for street. In many cases these are heavily conventionalised such that they can be listed as alternative spellings, but sometimes the writer has used a new abbreviation or one that we may not have listed in the lexicon. 

Unknown words There will always be words that are not in the lexicon and when we find these we have to deal with them somehow. 

Homographs are cases where two words share the same text form. In cases where the words differ in prommciation, we must find the correct word. In natural language homographs arise for three reasons  

In general, verbalisation functions for cases like these are not too difficult to construct all that is required is a careful and thorough evaluation of the possibilities. The main difficulty in these cases is that tiiere is no one correct expansion from (38) we see at least seven ways of saying the date, so which should we pick While sometimes the original text can be a guide (such that 10/12/67 would be verbahsed as ten twelve sixty seven), we find that human readers often do not do this, but rather pick a verbahsation that they believe is appropriate. The problem for us as TTS system builders is that sometimes the verbalisation we use is not the one that the reader expects or prefers, and this can be considered an error. 

LETTER-P LETTER-A LETTER-T FOUR ZERO AT CAM DOT LETTER-A LETTER-C DOT LETTER-U LETTER-K 

Having dealt with non-natural-language text, we now turn our attention to the issue of how to find the words from the text when that text does in fact encode natural language. 

---

Hodge, G., Nelson, T., and Vleduts-Stokolov, N. 1989. Automatic recognition of chemical names in natural language text. Paper presented at the 198th American Chemical Society National Meeting, Dallas, TX, April 7-9. 

The challenges for data integration for clinical genomics are these the data to be accessed are multidimensional and heterogeneous, the majority of data are unstructured and exist in natural language text, the data contains personal information that is subject to statutory requirements for protection of privacy, the ownerships of data lie with multiple stakeholders in multiple organizations,... 

The task of text decoding is to take a tokenised sentence and determine the best sequence of words. In many situations this is a classical disambiguation problem there is one, and only one correct sequence of words which gave rise to the text, and it is our job to determine this. In other situations, especially where we are dealing with non-natural language text such as numbers and dates and so on, there may be a few different acceptable word sequences. 

While we like to imagine that beautiful well constructed theories underly our algorithms, we frequently find that when it comes to text classification many systems in praetiee simply use a hodge-podge of rules to perform the task. This is particularly common in approaches to semiotic classification in genres where the amount of non-natural language text is very low, and so only a few special cases (say dealing with munbers) are required. 

For text genres dominated by natural language text, and in situations where very little training data is available, this approach may be adequate for semiotic classification. This is less likely to work for homographs, but again depending on the language, the level of errors may be acceptable. 

The final step in handling non-natural language text is to convert it into words and this process is often called verbalisation. If we take our decoded date example, we see that we have values for... 

The main problem in natural-language text decoding is dealing with homographs, whether tiiey be accidental, part-of-speech or abbreviation homographs. Many systems choose to handle each of these separately, and in particular abbreviation homographs are often dealt with at the same time or in the same module as semiotic classification. 

---