Data Cleaning (and Transformation)

February 12 by Bina

Today's plan:

1. Lets review Lab1 ques and answers
2. Intorduction to RStudio and some of its features
3. Data cleaning of real estate sales data
4. Data for today's demo
Data cleaning is not easy..

Please see recitaion and office hours schedule posted on Piazza

We are in Chapter 2 of Doing Data Science text. Today we will look at Data Cleaning and EDA for another real business: http://www.realdirect.com It is "Rethinking real estate": how it is bought/sold. This is a real estate business based on a subscription model. Traditional model involves a broker and a hefty commision for the broker selling the house (anywhere between 7-10%). RealDirect will provide.

1. Broker support and interface and data-analytics intelligence
2. Seller interface: sellers "subscribe" to realdirect (flat fee $365 month)
3. Sellers get brokers at a lower rate (1%-2%)
4. Buyers can use realdirect interface/platform to search, negotiate, and buy the available listing. (active, offer made, in contract, offer rejected etc.)

See pp.46-50 of your Doing Data Science text book for a full description of the problem. We need a data strategy? What is your data strategy give a bsic collection of sales data? Given: Rolling sales for Brooklyn and Manhattan counties, study the data and provide a data strategy.

Cleaning Data: Basic steps

1. Convert unknown file types to csv. For example, excel does read well into R, so convert it to csv and read it in as csv file.
2. Remove characters such as $ and , embedded within numeric data
3. Turn all (or select) variable names as well as values to lowercase for easy comparison and processing
4. Remove outliers, zeros and N/A (blanks)
5. Transform characters to numeric code
6. Binning or categorizing data: add variable(s) that will "bin" or "categorize" orginal data. Example: age$Cat (-Inf,0, 10,30,50,inf)
7. Add new information columns: "class" or "levels"
8. Remove unwanted observations, and variables
9. Add meaningful labels to support charting (this is more an EDA step)
10. Turn character/quantitative data into factors

How can you improve your expertise and be prepared for the exam1? Just go through chapter 2 and walk thorugh the two examples, NY Times click data and the Brooklyn exercise.

Algorithms for Data Processing I

February 14 by Bina

Today's plan:

1. Lets review Lab1 questions and answers: RShiny, web server, optional.
2. Algorithms for data processing: we will explore some of these today
3. Linear regression (linear model): a versatile algorithmic approach
4. Synthesizing data using R methods; contrarian "lm" demo

Please see recitaion and office hours schedule posted on Piazza

Motivation 1 for "lm": Here is a list of Top ten machine learning algorithms.

Motivation 2 for "lm": Bloomberg's oil price prediction method.

We are in Chapter 3. Read Chapter 3. Three important data processing algorithms are discussed in this chapter. We will look into three of these today: Linear Regression.

Problem 1: Consider the data

S = {(x,y)= (1,25), (10,250), (100,2500), (200,5000)}

What next? What is value for X = 40?
We observe that

• There is a linear pattern.
• Coefficient relating x and y is 25
• Seems deterministic
• and y = 25*x is the relationship, an obvious linear pattern.

Linear model is defined by the functional form:
y = β0 + β1*x

Lets work on 2 demos:

A simple analysis of "cars" data already available in R environment Another demo to understand the evaluation metric for an "lm" . Goal is to visualize and understand lm and its statistics.
1. The second demo is on synthetic data to understand the importance of the evalaution metrics and also to understand "overfitting"

For each demo we are interested in:

1. Cleaning and getting a data frame
2. Plot the data points
3. Visually assess the data plot
4. Model using "lm"
5. "lm" by default seeks to find a trend line that minimizes the sum of squares of the vertical distances between the approximated or predicted and observed y's.
6. We want to learn both the "trend" and the "variance"
7. Evaluate the measure of goodness of our model in R-squared and p: R-square measures the the proportion of variance. p-value assesses the significance of the result. We discuss both these measures: we want R-sqaured to high (0.0-1.0 range) and p to be low or <0.05. R-squared is 1-(total predicted error-squared/total mean error squared) (see p.67)
8. Always evalaute the error and goodness of fit using standard methods: sum of sqaures is a operation in the evalaution process.
.

See Demo code here

Algorithms for Data Processing II: Parameters

February 19 by Bina

Today's plan:

1. Evaluation of goodness of fit: with R Studio and LM
2. Pick it up where we left off: K-means -- Hand trace an example
3. K-means experiment stats: goodness of fit, number of iterations, convergence, termination criteria, closeness criteria
4. K-nn: complete Iris demo, and discuss common mis-iterpretations

Please see recitaion and office hours schedule posted on Piazza

See here for sample questions Sample
See Demo code here

Data Infrastructures

February 21 by Bina

Introduction

In the 1990's during the early days of the Internet, the access mode for home computing was through a humungous modem that made this characteristic noise. People had one of the two or three service providers: Prodigy, or Compuserve and probably the web and the search was the only application. Even that was slow. The "web page" was not fancy since most people had only line-based terminals for accessing the Internet. One of the popular browsers was called "Mosaic". When you query for the something it took such a long time that you can clean up your room by the time the query-results arrived.

In late 1990's Larry Page and Sergei Brin (Standford graduate) students worked on the idea considering the web as a large graph and ranking the nodes of this connected web. This became the core idea for their thesis and the company they founded, Google. Google that has since become the byword for "search". The core idea is an algorithm called "Pagerank". This is well explained in a paper [1] written about this thesis.

Until Google came about the focus was on improving the algorithms and the data infrastructure was secondary. There were questions about the storage use for supporting this mammoth network of pages and their content that are being searched. In 2004 two engineers from Google presented an overview paper [2] of the Google File System (GFS) and a novel parallel processing algorithm "MapReduce". A newer version of the paper is available at [3] .

Of course, Google did not release the code as open source. Doug Cutting an independent programmer who was living in Washington State reverse engineered the GFS and named it after the yellow toy elephant that belonged to his daughter. Yahoo hired him and made Hadoop open source.

In 2008 Yahoo, National Science Foundation and Computing Community Consortium gathered together educators, industries, and other important stakeholders for the inaugural Hadoop Summit. Look at the list of attendees here I learned Hadoop from Doug Cutting. It is imperative that we study Hadoop and MapReduce together to learn their collective contribution.

Let us understand the foundation of the Big Data Infrastructure that has become integral part of most computing environments. We will approach this study by processing unstructured data as discussed in text [4] .

References

1. K. Bryan and T. Leise. The $25,000,000,000 Linear Algebra Behind Google. here.
2. Dean, J. and Ghemawat, S. 2004. MapReduce: Simplified data processing on large clusters. In Proceedings of Operating Systems Design and Implementation (OSDI). San Francisco, CA. 137-150. MR, last viewed 2017.
3. Newer version in ACM of the above paper here.
4. J. Lin and C. Dyer. Data-Intensive Text Processing with MapReduce, Synthesis Lectures on Human Language Technologies, 2010, Vol. 3, No. 1, Pages 1-177, (doi: 10.2200/S00274ED1V01Y201006HLT007). Morgan & Claypool Publishers. An online version of this text is also available through UB Libraries since UB subscribes to Morgan and Claypool Publishers. Online version available here