Flat files

According to an elegant remark by Davies [5], "Modem scientific data handling is multitechnique, multisystem, and manufacturer-independent, with results being processed remotely from the measuring apparatus. Indeed, data exchange and storage are steps of the utmost importance in the data acquisition pathway. The simplest way to store data is to define some special format (i.e., collection of rules) of a flat file. Naturally, one cannot overestimate the importance of databases, which are the subject of Chapter 5 in this book. Below we discuss three simple, yet efficient, data formats. 

Predictive Model Markup Language (PMML) is far more than just another format of a data container flat file [7]. As is clear from the name, it is an XML-based markup language delivering all the power of XML. Readers are recommended to consult Section 2.4.5 and the website www.xml.org for more details on XML and its applications in chemistry. 

In a flat-file sy.stcm the database is called a file. 

Figure 5-3. a) Main organization of a database or container the basic units of a field are bits and bytes, b) Example of data organization in a flat-file. 

Each object has the ability to serialize itself and also to initialize itself from a serialized representation. If the programming language has a reflective facility, you can write a single piece of code to determine the structure of the object and perform serialization and initialization. Java serialization works this way. Of course, flat files do not provide any of the multi-user, concurrency, meta-data, schema evolution, transaction, and recovery facilities that a database provides. 

FIGURE 4 A typical directory structure found in a flat-file data system. 

Figure 7 shows the relationship between a raw data channel and its associated metadata. If we were to choose the item highlighted Instrument Method , the embedded relational database would retrieve the exact version of the instrument method that was used to acquire the raw data. All this occurs in a fraction of a second. Imagine how long it would take using a conventional flat-file system (see Figure 8). 

The first collection of the powder diffraction data appeared in 1938, known as ASTM cards. Later, the International Centre for Diffiaction Data (ICDD) issued the data in electronic form, which became known as the Powder Diffraction File (PDF). Majority of the users currently use the so called Pdf-2 version, updated in different years. It contains both measured and computed diffiaction data in a flat-file structure. Beside some additional... 

The core of the EntityDictionaryDao is in the retrieve...() methods. Here we assume the entity dictionaries are stored in a relational database. They can also be accessed from other types of data sources, such as web service, XML, and flat files. The point is to transform them into something that can be accessed easily and quickly by CRS. Take a closer look at the retrievePersonnel() method. Like most other retrieve...() methods, retrievePersonnel() returns a Map. What is in the Map depends on what kind of lookups the clients want to use to access the personnel dictionary. In the context of CRS, the personnel data can be accessed by its entirety, the research site where the person is located, person id, person s full name, or person s username. Therefore, the Map that retrievePersonnel() returns has four Collections—an entire personnel list, a site-people map, a person id-person map, a person s full name-person map, and a username-person map. 

Use needle files to smooth the rough edges. Use flat files for flat areas and outside curves, and use semicircular and round files for inside curves. 

Databases are electronic filing cabinets that serve as a convenient and efficient means of storing vast amounts of information. An important distinction exists between primary (archival) and secondary (curated) databases. The primary databases represent experimental results with some interpretation. Their record is the sequence as it was experimentally derived. The DNA, RNA, or protein sequences are the items to be computed on and worked with as the valuable components of the primary databases. The secondary databases contain the fruits of analyses of the sequences in the primary sources such as patterns, motifs, functional sites, and so on. Most biochemical and/or molecular biology databases in the public domains are flat-file databases. Each entry of a database is given a unique identifier (i.e., an entry name and/or accession number) so that it can be retrieved uniformly by the combination of the database name and the identifier. 

The kind of information managed, whether it is sales data, electronic docnments, clinical trial data, or recipes for a manufacturing execution system, is fairly independent of the database type (althongh no one wonld build a flat file database for any of these). The choice of relational vs. hierarchical vs. network is primarily dependent on business needs. 

Pre-1980 —Flat File Storage of Chemical Structures. Computers consisted of mainframe machines (e.g., IBM 3090) and small minicomputers (Digital, Prime). Users connected through low speed serial connections, using "dumb" terminals (no graphics capability) or monochrome vector graphics terminals such as Tektronix and Imlac. Chemical structures were mainly stored as either (l)individ-ual structure files, indexed by name, and handled one or a few structures at a time or (2) in a flat-file database accessed by record number (26). A typical corporate database contained up to a few tens of thousands of structures. 

Relational databases can be combined, giving the whole system immense flexibility. The older flat-file databases store information in files which can be searched and sorted, but cannot be linked to other databases. 

In SRS, meta definition is used to describe objects which the SRS core uses. In the case of a database, a library object must be defined. This object contains the name of the library, what sort of library it is (i.e. what group of databanks it belongs to), the name and whereabouts of the flat files containing the data. It also contains a link to a file containing a list of rules which describe the internal syntax of the databank. These syntax rules will be described below. 

All flat file databases are semi-structured, containing a list of entries, with each entry containing a list of data-fields (e.g an Id, an Accession number, key words, a sequence etc.). Figure 1.6 shows a sample of a database entry. Each of the data-fields consists of strings or tokens. The set of productions for each database must describe how to divide the database into entries and then further into fields and then into the strings or tokens within that data field. It is these tokens within each field which are inserted into an index. 

The EMBL library object, (Figure 1.8), defines the EMBL library as being part of the sequence library group, with the format being described in the EMBL FORMAT object (Figure 1.9) and the files which make up the EMBL database being all those files with the extension dat in the EMBL flat file directory. 

Once a set of entries which match the search terms has been retrieved, various operations can be performed on that set. The simplest is to view each whole entry in it, or to just view various fields from the entries. Predefined views, either part of the SRS package, or defined by the local SRS administrator, allow for the data to be presented in other formats. Figure 1.13 shows the default entry view of a whole SWISS-PROT entry. This looks considerably different to the text in the flat file, as shown in Figure 1.14, with the information easier to read. Either a pre-defined view of the entries can be used to view just some of the fields, or alternatively a view of the data... 

Other attempts to solve the problem of integrating Molecular Biology resources can be divided into two possible approaches, either using relational databases to store and retrieve data or to use database specific programs to parse flat files. 

The NCBI does use a flat file approach to parse and retrieve the data in their databases and present it on the web. While the NCBI continutes to add databases, there are not as many databases available as some SRS servers, and hence it is difficult to find relationships that may exist between the data displayed and data in other databases. Since the NCBI present their data on their web site it is also not possible for other academic institutions or companies to bring the software in-house for integration of their proprietry data. 

SRS [18] is an indexing system for flat-file libraries such as EMBL or UniProt. Originally developed at EMBL, SRS was later acquired by LION Bioscience AG and released as a licensed product. It remains freely available for academics. 

---