Large Scale Data

Abstract In this chapter the problems of using spectral dimensionality reduction with large scale datasets are outlined along with various solutions to these problems. The computational complexity of various spectral dimensionality reduction algorithms are looked at in detail. There is also often much overlap between the solutions in this chapter and what has been discussed previously with regards to incremental learning. Finally, some parallel and GPU based implementation aspects are discussed. 

---

Computational methods have been applied to determine the connections in systems that are not well-defined by canonical pathways. This is either done by semi-automated and/or curated literature causal modeling [1] or by statistical methods based on large-scale data from expression or proteomic studies (a mostly theoretical approach is given by reference [2] and a more applied approach is in reference [3]). Many methods, including clustering, Bayesian analysis and principal component analysis have been used to find relationships and "fingerprints" in gene expression data [4]. 

In the case of the acquisition of large-scale data sets, the benefits of using reference materials are self-evident. In the past, whenever such data sets have been acquired without using suitable reference materials, a great deal of effort has subsequently been needed to adjust the data to a common scale. But the benefit of comparability is not restricted to large programs. Matrix-based reference materials that can be exchanged between different laboratories will enable researchers to better understand their own techniques and the information they provide. 

Bork, P. (2002). Comparative assessment of large-scale data sets of protein-protein interactions. Nature 417,... 

Systems medicine requires the integration of many different types of data into models that have predictive behavior. We discuss below several of the high-throughput platforms that are generating large-scale data for a systems approach to disease. 

In Eq. (15-135), is the specific wall surface (cmVcm ) and flp is the specific packing surface (cmvcm ). This term is dropped for a spray column (Cl = 0). The model coefficients are summarized in Table 15-19. Most of the axial mixing data available in the literature are for the continuous phase dispersed-phase axial mixing data are rare. Becker recommends assuming HDU = HDU, when dispersed-phase data are not available. Becker presents a parity plot (Fig. 15-33) based on small- and large-scale data for packed and spray columns. 

Following publication, large-scale data can be uploaded to search data repositories. The two most popular are arrayExpress and the Gene Expression Omnibus (GEO). Community-generated data stored in ArrayExpress and GEO, maintained by the European Bioinformatics Institute and the National Center for Biotechnology Institute, respectively, are searchable and readily available for download and analysis. Verification analyses can be performed using these previously published datasets. 

Von Merrng, C., Krause, R., Snel, B., Cornell, M., Oliver, S.G., Fields, S., and Bork, P. 2002. Comparative assessment of large-scale data sets of protein-protein interactions. Nature 417, 399-403. 

ViSTA-FlowLib is a software framework developed at the Institute for Scientific Computing of RWTH Aachen University [386]. It comprises algorithms for the interactive visualization of data sets produced by Computational Fluid Dynamics (CFD). Special attention is paid to unsteady and large-scale data sets. 

As the PubChem data content grows, there is an ever increasing need for facile methods of efficienf large-scale data management and analysis. 

A further consideration with large-scale data is the time taken to perform the selection process. As a highly repeated function within the algorithm, even small differences in execution time will make a large difference to the overall execution time of the algorithm. 

Buzzi-Ferraris, G. and Manenti, F. (2011b) Outlier detection in large-scale data sets. Comput. Chem. Eng, 35, 388-390. 

Frequently, the field of application for these methods and tools is confined to large-scale data processing... 

---