CS250: Python for Data Science

An important initial step in the data life cycle is to express the question of interest in the context of the subject area and consider the connection between the question and the data collected to answer that question. It's good practice to do this before even thinking about the analysis or modeling steps because it may uncover a disconnect where the question of interest cannot be directly addressed with these data. As part of making the connection between the data collection process and the topic of investigation, we identify the population, the means of accessing the population, instruments of measurement, and additional protocols used in the collection process. These concepts help us understand the scope of the data, whether we aim to gain knowledge about a population, scientific quantity, physical model, social behavior, etc.

The target population consists of the collection of elements that you ultimately intend to describe and draw conclusions about. By element we mean those individuals that make up our population. The element may be a person in a group of people, a voter in an election, a tweet from a collection of tweets, or a county in a state. We sometimes call an element a unit or an atom.

The access frame is the collection of elements that are accessible to you for measurement and observation. These are the units by which you can access the target population. Ideally, the frame and population are perfectly aligned; meaning they consist of the exact same elements. However, the units in an access frame may be only a subset of the target population; additionally, the frame may include units that don't belong to the population. For example, to find out how a voter intends to vote in an election, you might call people by phone. Someone you call, may not be a voter so they are in your frame but not in the population. On the other hand, a voter who never answers a call from an unknown number can't be reached so they are in the population but not in your frame.

The sample is the subset of units taken from the access frame to measure, observe, and analyze. The sample gives you the data to analyze to make predictions or generalizations about the population of interest.

The contents of the access frame, in comparison to the target population, and the method used to select units from the frame to be in the sample are important factors in determining whether or not the data can be considered representative of the target population. If the access frame is not representative of the target population, then the data from the sample is most likely not representative either. And, if the units are sampled in a biased manner, problems with representativeness also arise.

You will also want to consider time and place in the data scope. For example, the effectiveness of a drug trial tested in one part of the world where a disease is raging might not compare as favorably with a trial in a different part of the world where background infection rates are lower. Additionally, data collected for the purpose of studying changes over time, like with the monthly measurements of CO2 in the atmosphere and the weekly reporting of Google searches for predicting flu trends have a temporal structure that we need to be mindful of as we examine the data. At other times, there might be spatial patterns in the data. For example, the environmental heath data, described later in this section, are reported for each census tract in the State of California, and we might, say, make maps to look for spatial correlations.

And, if you didn't collect the data, you will want to consider who did and for what purpose. This is especially relevant now since more data is passively collected instead of collected with a specific goal in mind. Taking a hard look at found data and asking yourself whether and how these data might be used to address your question can save you from making a fruitless analysis or drawing inappropriate conclusions.

For each of the following examples, we begin with a general question, narrow it to one that can be answered with data, and in doing so, we identify the target population, access frame, and sample. These concepts are represented by circles in a diagram of data scope, and the configuration of their overlap helps reveal key aspects the scope. Also in each example, we describe relevant temporal and spatial features to the data scope.

What makes members of an online community active? Content on Wikipedia is written and edited by volunteers who belong to the Wikipedia community. This online community is crucial to the success and vitality of Wikipedia. In trying to understand how to incentivize members of online communities, researchers carried out an experiment with Wikipedia contributors as subjects. A narrowed version of the general question asks: do informal awards increase the activity of Wikipedia contributors? For this experiment, the target population is the collection of active contributors - those who made at least one contribution to Wikipedia in the month before the start of the study. Additionally, the target population was further restricted to the top 1% of contributors. The access frame eliminated anyone in the population who had received an informal incentive that month. The access frame purposely excluded some of the contributors in the population because the researchers want to measure the impact of an incentive and those who had already received one incentive might behave differently. (See Figure 2.2).

Fig. 2.2 The access frame does not include the entire population because the experiment included only those contributers who had not already received an incentive. The sample is a randomly selected subset from the frame.

The sample is a randomly selected set of 200 contributors from the frame. The sample of contributors were observed for 90 days and digital traces of their activities on Wikipedia were collected. Notice that the contributor population is not static; there is regular turnover. In the month prior to the start of the study more than 144,000 volunteers produced content for Wikipedia. Selecting top contributors from among this group limits the generalizability of the findings, but given the size of the group of top contributors, if they can be influenced by an informal reward to maintain or increase their contributions that is a valuable finding.

In many experiments and studies, we don't have the ability to include all population units in the frame. It is often the case that the access frame consists of volunteers who are willing to join the study/experiment.

Who will win the election? The outcome of the US presidential election in 2016 took many people and many pollsters by surprise. Most pre-election polls predicted Clinton would beat Trump by a wide margin. Political polling is a type of public opinion survey held prior to an election that attempts to gauge who people will vote for. Since opinions change over time, the focus is reduced to a "horse-race" question, where respondents are asked for whom would they vote in a head-to-head race if the election were tomorrow: Candidate A or Candidate B.

Polls are conducted regularly throughout the presidential campaign, and the closer to election day, we expect the polls to get better at predicting the outcome, as preferences stabilize. Polls are also typically conducted statewide, and later combined to make predictions for the overall winner. For these reasons, the timing and location of a poll matters. The pollster matters too; some have consistently been closer to the mark than others 1.

In these pre-election surveys, the target population consists of those who will vote in the election, which in this example was the 2016 US presidential election. However, pollsters can only guess at whether someone will vote in the election so the access frame consists of those deemed to be likely voters (this is usually based on their past voting record, but other factors may be used to determine this), and since people are contacted by phone, the access frame is limited to those who have a landline or mobile phone. The sample consists of those people in the frame who are chosen according to a random dialing scheme. (See Figure 2.3).

Fig. 2.3 This representation is typical of many surveys. The access frame does not cover all of the population and includes some who are not in the population.

Later, we discuss the impact on the election predictions of people's unwillingness to answer their phone or participate in the poll.

How do environmental hazards impact an individual's health? To address this question, the California Environmental Protection Agency (CalEPA), the California Office of Health Hazard Assessment (OEHHA), and the public developed the CalEnviroScreen project. The project studies connections between population health and environmental pollution in California communities using data collected from several sources that includes demographic summaries from the U.S. census, health statistics from the California Office of Statewide Health Planning and Development, and pollution measurements from air monitoring stations around the state maintained by the California Air.

Ideally, we want to study the people of California, and assess the impact of these environmental hazards on an individual's health. However, in this situation, the data can only be obtained at the level of a census tract. The access frame consists of groups of residents living in the same census tract. So, the units in the frame are census tracts and the sample is a census–all of the tracts–since data are provided for all of the tracts in the state. (See Figure 2.4).

Fig. 2.4 The grid in the access frame represents the census tracts. The population, frame, and sample cover all Californians, but the grid limits measurments to the level of census tract.

Unfortunately, we cannot deaggregate the information in a tract to examine what happens to an individual person. This aggregation impacts the question we can address and the conclusions that we can draw.

These examples have demonstrated some of the configurations a target, access frame, and sample might have, and the exercises at the end of this chapter provide a few more examples. When a frame doesn't reach everyone, we should consider how this missing information might impact our findings. Similarly we ask what might happen when a frame includes those not in the population. Additionally, the techniques for drawing the sample can affect how representative the sample is of the population. When you think about the generalizability of your data findings, you also want to consider the quality of the instruments and procedures used to collect the data. If your sample is a census that matches your target, but the information is poorly collected, then your findings will be of little value. This is the topic of the next section.