A History of Data Science

Chapter Summary

This is a very quick overview of the eight "parent" disciplines that contribute to the new Data Science discipline. It suggests generic questions that data scientists should ask as they work through solving problems.

Discussion

As mentioned in Chapter 1, Data Science is a mash-up of several different disciplines. We also noted that an individual data scientist is most likely an expert in one or two of these disciplines and proficient in another two or three. There is probably no living person who is an expert in all these disciplines, and an extremely rare person would be proficient in 5 or 6 of these disciplines. This means that data science must be practiced as a team where, across the membership of the team, there is expertise and proficiency across all disciplines. Let us explore what these disciplines are and how they contribute to data science.

Data Engineering

Data Engineering is the data part of data science. According to Wikipedia, Data Engineering involves acquiring, ingesting, transforming, storing, and retrieving data. Data engineering also includes adding metadata to the data. Because all these activities are interrelated, a data engineer must solve these issues as an integrated whole. For example, we must understand how we plan to store and retrieve the data in order to create a good ingestion process. Data engineering requires a thorough understanding of the general nature of the data science problems to be solved in order to formulate a robust data acquisition and management plan. Once the plan is well developed, data engineers can begin to implement it into data management systems.

Acquiring - This is the process of laying our hands on the data. The data engineer part of the data scientist needs to ask the questions, "where is the data coming from?", "what does the data look like?". and "how does our team get access to the data?" The data could come from many places such as RSS feeds, a sensor network or a preexisting data repository. The data could be numbers, text documents, images, or videos. The data can be collected by the team or purchased from a vendor. For example, if we are going to investigate highways, we could have sensors on a stretch of freeway that measures how fast cars are going. These sensors send us the data as text messages that include the date, time, lane, and speed of every car that crosses the sensors.

Ingesting - This is the process of getting the data from the source into the computer systems we will use for our analysis. The data engineer part of the data scientist needs to ask the questions, "how much data is coming?", "how fast is it coming?", "where are we going to put the data?", "do we have enough disk space for the data?", and "do I need to filter the incoming data in any way?" Data is measured in bytes. A byte is roughly equivalent to one character of a written word. A one-page document is about 1,000 bytes or one kilobyte (1K). For example, if we are going to investigate highways, we could be receiving car speed data at a rate of 10,000 bytes per second for a 1-week period. There are 604,800 seconds in a week. This means you will receive 6,048,000,000 bytes (6 gigabytes) of data in one week. No problem. That will fit on a thumb drive.

Transforming - This is the process of converting the data from the form in which it was collected to the form it needs to be in for our analysis. The data engineer part of the data scientist needs to ask the questions, "what is the form of the raw data?" and "what does the form of the processed data need to be?" A common raw data format is comma-separated values (CSV) which looks like this:

20120709,135214,157,3,57.4
20120709,135523,13,2,62.1

Scientific Method

1. Every life form that everyone knows of depends on liquid water to exist.
2. Therefore, all known life depends on liquid water to exist.

1. All men are mortal.
2. Socrates is a man.
3. Therefore, Socrates is mortal.

• The classic example is the trial of Galileo. At the time (1633), the Catholic church held to Aristotle's logical argument that the earth was the center of the cosmos. Galileo's observations with his newly invented telescope provided evidence of Copernicus's assertion that the earth revolved around the sun. The outcome of the trial was that Galileo was sentenced to house arrest for heresy. In 2000, Pope John Paul II apologized for the injustice done to Galileo.

• A defendant is considered not guilty as long as his or her guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough charging evidence the defendant is convicted. At the start of the procedure, there are two hypotheses: "the defendant is not guilty", and "the defendant is guilty". The first one is called the null hypothesis and is accepted for the time being. The second one is called the alternative hypothesis. It is the hypothesis one tries to prove. The hypothesis of innocence is only rejected when an erroneous conviction is very unlikely because one doesn't want to convict an innocent defendant.

• One prominent example is the "inclined plane," or "ball and ramp experiment". In this experiment, Galileo used an inclined plane and several steel balls of different weights. With this design, Galileo was able to slow down the falling motion and record, with reasonable accuracy, the times at which a steel ball passed certain markings on a beam. Galileo disproved Aristotle's assertion that weight affects the speed of an object's fall. According to Aristotle's Theory of Falling Bodies, the heavier steel ball would reach the ground before the lighter steel ball. Galileo's hypothesis was that the two balls would reach the ground at the same time.

Math

Statistics

Advanced Computing

/**
 * Traditional "Hello World" program In Java
 */

class HelloWorldApp {</span><br><span class="pln"> </span><span class="kwd">public</span><span class="pln"> </span><span class="kwd">static</span><span class="pln"> </span><span class="kwd">void</span><span class="pln"> main</span><span class="pun">(</span><span class="typ">String</span><span class="pun">[]</span><span class="pln"> args</span><span class="pun">)</span><span class="pln"> </span><span class="pun">{</span><br><span class="pln"> </span><span class="typ">System</span><span class="pun">.</span><span class="kwd">out</span><span class="pun">.</span><span class="pln">println</span><span class="pun">(</span><span class="str">"Hello World!"</span><span class="pun">);</span><span class="pln"> </span><span class="com">// Display the string.</span><br><span class="pln"> </span><span class="pun">}
}

#
# Traditional "Hello World" program in Python 2.x
#

print "Hello World!"

Visualization

Hacker mindset

• A famous example is Steve Wozniak's hand-made Apple I computer. It was built from parts scrounged from Hewlett-Packard's trash and from electronic surplus supply stores. Wozniak wanted to give the plans away, but his partner, Steve Jobs, convinced him that they should sell ready-made machines. The rest, as they say, is history.
• Another example is the Carnegie Mellon Internet Coke Machine. In the early days of the internet before the web, students at Carnegie Mellon instrumented and wired their local Coke Machine to the internet. The students could check to see which internal dispenser columns had been loaded most recently, so they could be sure to buy cold, not warm, sodas. This was important because the machine sold one Coke every 12 minutes and was re-loaded several times a day.

Domain Expertise

• Edwin Chen, a data scientist at Twitter, computed and visualized the geographic distribution of tweets that refer to soft drinks as "soda," as "pop," and as "coke". Just observing that the Midwest uses "pop" and the Northeast uses "soda" is interesting but lacks an explanation. In order to understand WHY these geographic divisions exist, we would need to consult with domain experts in sociology, linguistics, US history, and maybe anthropology - none of whom may know anything about data science. Why do you think these geographic linguistic differences exist?
• Nate Silver is a statistician and domain expert in US politics. His blog regularly combines both the data and an explanation of what it means. In his posting, "How Romney’s Pick of a Running Mate Could Sway the Outcome," he not only tells us what the differences are based on his mathematical model, he explains why those outcomes fell out the way they did.

Assignment/Exercise

• Print a copy and read over Google's R Style Guide. Right now, most of the guide will not make a lot of sense, but it will make more sense as we progress through the book. Keep the printed copy for future reference.
• Search the web for "introduction to R," "R tutorial," "R basics," and "list of R commands". Pick four or five of these websites to work on. Try working through the first few examples of each site. Many of the introductions go too fast or assume too much prior knowledge, so when it gets too confusing just try another site.
• Try the commands:
  

• Write a short 5- to 7-line program that will execute without errors and save it. Be sure to include the names of all those who contributed in the comment section.
• Make a list of the sites the team worked from, and indicate which was the most helpful.
• Make a list of the top 10 unanswered questions the team has at the end of the study session.

Chapter Summary

Discussion

What is Data?

What is a Data Point?

What is a Data Set?

What are Data Types?

Data Types in Mathematics

1. Integers - According to Wikipedia, integers are numbers that can be written without a fractional or decimal component, and fall within the set {..., −2, −1, 0, 1, 2, ...}. For example, 21, 4, and −2048 are integers; 9.75, 5½, and $\sqrt{2}$ are not integers.
2. Rational Numbers - According to the Wikipedia, rational numbers are those that can be expressed as the quotient or fraction p/q of two integers, with the denominator q not equal to zero. Since q may be equal to 1, every integer is a rational number. The decimal expansion of a rational number always either terminates after a finite number of digits or begins to repeat the same finite sequence of digits over and over. For example, 9.75 2/3, and 5.8144144144… are rational numbers.
3. Real Numbers - According to Wikipedia, real numbers include all the rational numbers, such as the integer −5 and the fraction 4/3, plus all the irrational numbers such as $\sqrt{2}$ (1.41421356... the square root of two), $π (3.14159265..)$., and e (2.71828..).
4. Imaginary Numbers - According to Wikipedia, imaginary numbers are those whose square is less than or equal to zero. For example, $\sqrt{-25}$ is an imaginary number and its square is -25. An imaginary number can be written as a real number multiplied by the imaginary unit i, which is defined by its property $i ^2 = −1$. Thus, $\sqrt{-25} = 5i$.

Data Types in Statistics

1. Nominal - Nominal data are recorded as categories. For this reason, nominal data is also known as categorical data. For example, rocks can be generally categorized as igneous, sedimentary, and metamorphic.
2. Ordinal - Ordinal data are recorded in the rank order of scores (1st, 2nd, 3rd, etc). An example of ordinal data is the result of a horse race, which says only which horses arrived first, second, or third but includes no information about race times.
3. Interval - Interval data are recorded not just about the order of the data points, but also the size of the intervals in between data points. A highly familiar example of interval scale measurement is temperature with the Celsius scale. In this particular scale, the unit of measurement is 1/100 of the temperature difference between the freezing and boiling points of water. The zero point, however, is arbitrary.
4. Ratio - Ratio data are recorded on an interval scale with a true zero point. Mass, length, time, plane angle, energy, and electric charge are examples of physical measures that are ratio scales. Informally, the distinguishing feature of a ratio scale is the possession of a zero value. For example, the Kelvin temperature scale has a non-arbitrary zero point of absolute zero.

Data Types in Computer Science

1. Bit - A bit (a contraction of a binary digit) is the basic unit of information in computing and telecommunications; a bit represents either 1 or 0 (one or zero) only. This kind of data is sometimes also called binary data. When 8 bits are grouped together we call that a byte. A byte can have values in the range 0-255 (00000000-11111111). For example, the byte 10110100 = 180.
   
   • Hexadecimal - Bytes are often represented as Base 16 numbers. Base 16 is known as Hexadecimal (commonly shortened to Hex). Hex uses sixteen distinct symbols, most often the symbols 0–9 to represent values zero to nine, and A, B, C, D, E, F (or alternatively a–f) to represent values ten to fifteen. Each hexadecimal digit represents four bits, thus two hex digits fully represent one byte. As we mentioned, byte values can range from 0 to 255 (decimal), but may be more conveniently represented as two hexadecimal digits in the range 00 to FF. A two-byte number would also be called a 16-bit number. Rather than representing a number as 16 bits (10101011110011), we would represent it as 2AF3 (hex) or 10995 (decimal). With practice, computer scientists become proficient in reading and thinking in hex. Data scientists must understand and recognize hex numbers. There are many websites that will translate numbers from binary to decimal to hexadecimal and back.
2. Boolean - The Boolean data type encodes logical data, which has just two values (usually denoted "true" and "false"). It is intended to represent the truth values of logic and Boolean algebra. It is used to store the evaluation of the logical truth of an expression. Typically, two values are compared using logical operators such as .eq. (equal to), .gt. (greater than), and .le. (less than or equal to). For example, $b = (x .eq. y)$ would assign the boolean value of "true" to "b" if the value of "x" was the same as the value of "y," otherwise it would assign the logical value of "false" to "b".
3. Alphanumeric - This data type stores sequences of characters (a-z, A-Z, 0-9, special digits) in a string--from a character set such as ASCII for western languages or Unicode for Middle Eastern and Asian languages. Because most character sets include numeric digits, it is possible to have a string such as "1234". However, this would still be an alphanumeric value, not the integer value 1234.
4. Integers - This data type has essentially the same definition as the mathematical data type of the same name. In computer science, however, an integer can either be signed or unsigned. Let us consider a 16-bit (two-byte) integer. In its unsigned form, it can have values from 0 to 65535 (2¹⁶-1). However, if we reserve one bit for a (negative) sign, then the range becomes -32767 to +32768 (-7FFF to +8000 in hex).
5. Floating Point - This data type is a method of representing real numbers in a way that can support a wide range of values. The term floating point refers to the fact that the decimal point can "float"; that is, it can be placed anywhere relative to the significant digits of the number. This position is indicated separately in the internal representation, and floating-point representation can thus be thought of as a computer realization of scientific notation. In scientific notation, the given number is scaled by a power of 10 so that it lies within a certain range - typically between 1 and 10, with the decimal point appearing immediately after the first digit. The scaling factor, as a power of ten, is then indicated separately at the end of the number. For example, the revolution period of Jupiter's moon Io is 152853.5047 seconds, a value that would be represented in standard-form scientific notation as 1.528535047×10⁵ seconds. Floating-point representation is similar in concept to scientific notation. The base part of the number is called the significand (or sometimes the mantissa) and the exponent part of the number is unsurprisingly called the exponent.
   
   
   • The two most common ways in which floating point numbers are represented are either in 32-bit (4 byte) single precision or in 64-bit (8 byte) double precision. Single precision devotes 24 bits (about 7 decimal digits) to its significand. Double precision devotes 53 bits (about 16 decimal digits) to its significand.
6. List - This data type is used to represent complex data structures. In its most simple form, it has a key-value pair structure. For example, think of a to-do list:

Data Types in R

1. NULL - for something that is nothing
2. logical - for something that is either TRUE or FALSE (on or off; 1 or 0)
3. character - for alphanumeric strings
4. integer - for positive, negative, and zero whole numbers (no decimal place)
5. double - for real numbers (with a decimal place)
6. complex - for complex numbers that have both real and imaginary parts (e.g., square root of -1)
7. date - for dates only
8. POSIX - for dates and times (dates are internally represented as the number of days since 1970-01-01, with negative values for earlier dates)
9. list - for storing complex data structures, including the output of most of the built-in R functions

• Just a note about lists in R. R likes to use the list data type to store the output of various procedures. We generally do not perform statistical procedures on data stored in list data types--with one big exception. In order to do statistical analysis on lists, we need to convert them to tables with rows and columns. R has a number of functions to move data back and forth between table-like structures and list data types. The exception we just referred to, is called the data.frame list object. List objects of the class data.frame store rows and columns of data in such a specifically defined way as to facilitate statistical analysis. We will explain data frames in more detail below.

What are Variables and Values?

Assignment/Exercise

Find Data Types in R

 > a <- as.integer(1)
 > typeof(a)
 > a

 > b <- as.double(1)
 > typeof(b)
 > b

 > d <- as.character(1)
 > typeof(d)
 > d

 > e <- as.logical("true")
 > typeof(e)
 > e

 > f <- as.complex(-25)
 > typeof(f)
 > f

 > g <- as.null(0)
 > typeof(g)
 > g

 > h <- as.Date("2012-07-04")
 > typeof(h)
 > class(h)
 > h

 > i <-as.POSIXct("2012/07/04 10:15:59")
 > typeof(i)
 > class(i)
 > i

 > j <-as.POSIXlt("2012/07/04 10:15:59")
 > typeof(j)
 > class(j)
 > j

 > k <- list("Get haircut", "Buy Groceries", "Take shower") 
 > typeof(k)
 > k

Objects, Variables, Values, and Vectors in R

 > name <- c("Maria", "Fred", "Sakura") 
 > typeof(name)
 > name

 > age <- as.integer(c(24,19,21))
 > typeof(age)
 > age

 > name[1] 
 > name[2]
 > name[3]
 > age[1]
 > age[2]
 > age[3]

Data Sets and Data Frames

 > project <- data.frame(name, age)
 > typeof(project)
 > class(project)
 > ls(project)
 > str(project)

 > project$age.n <- as.double(project$age)
 > str(project)

 > project
     name age age.n
 1  Maria  24    24
 2   Fred  19    19
 3 Sakura  21    21

 > typeof(project$name)
 > typeof(project$age)
 > typeof(project$age.n)

• Because all statistical computations are done on numbers, R gave each value of the variable "name" an arbitrary integer number. It calls these arbitrary numbers levels. It then labeled these levels with the original values, so we would know what is going on. So under the covers, project$namehas the values: 2 (labeled "Maria), 1 (labeled "Fred"), and 3 (labeled Sakura). We can convert project$name back into the character data type, but we won't be able to perform statistical calculations on it.

 > project$name.c <- as.character(project$name)
 > typeof(project$name.c)
 > str(project)
 'data.frame':    3 obs. of  4 variables:
  $ name  : Factor w/ 3 levels "Fred","Maria",..: 2 1 3
  $ age   : int  24 19 21
  $ age.n : num  24 19 21
  $ name.c: chr  "Maria" "Fred" "Sakura"