The report of the 2008 Extremely Large Database (XLDB) Workshop, a conference that addresses methods, architectures and best practices for data intensive science, observed, “For the largest-scale datasets, there is no debate that computation must be moved close to where the data resides, rather than moving the data to the computation.” New programming models such as MapReduce for data processing on large clusters have shown excellent performance (e.g., in Google applications) with rather low programming overhead. A widely used open source implementation of MapReduce programming model is Hadoop.

Simultaneously, there has been an implementation of active disks in the form of Netezza Performance Server (NPS), a data intensive supercomputer (DISC). The results from NPS DISC have proven to be quite remarkable, with data analysis delivered orders of magnitude faster than the currently used platforms.

The Center for Computational Research (CCR) at UB provide extensive on-line documentation and a wide variety of training, including hosting workshops. Topics include fundamentals of parallel computing, introduction to CCR, debugging and profiling tools, bioinformatics resources etc. Software packages (in the fields of bioinformatics, Chemistry/Biochemistry, Engineering and Physics) currently installed and maintained by CCR Staff can be found here.