Research Skills Tutorial

Read this tutorial, which discusses appropriate questions to ask to determine whether a source is credible and reliable. As you read, make a list of important questions to ask. Make notes under each question about why that question is important. Also, write down any tips to consider when answering each question.

Validity

Validity asks the question, "How do we know what we know?" Every field of study answers that question differently, but there are some ideas that are generally considered invalid, no matter whether an information source is scholarly or non-scholarly, and no matter what discipline or subject area it falls under.

In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The concepts of ideology, agenda and bias are related to that.

    • Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad. And just because an author subscribes to an ideology does not mean that the information source necessarily has an ideological agenda.

    • Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it is not strictly informational or educational. It is probably actually a persuasive information source or a piece of propaganda, even if it is pretending to be otherwise.

    • Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

A credible information source will not try to tell you how to feel about the information.


Validity by Discipline


Humanities

Literature, the Arts, History, Philosophy, Theology

In the humanities, the author is either constructing a worldview, or (more often) adding to, refining, and correcting a worldview that other scholars have created. That worldview needs to be self-consistent and consistent with the evidence. It needs to be able to support new discoveries and insights. And it often needs to be beautiful.

Here are some criteria for validity that can be generalized in the humanities:

    • Good use of primary sources. Appropriate sources are chosen. They are translated and interpreted correctly. Provenance is clear.
    • Anecdotes are fine as long as the author makes it very clear how they represent general principles.
    • Drawing connections and creating a persuasive (and beautiful) argument in favor of them.
    • No logical fallacies.
    • Exhaustive knowledge of relevant secondary sources, both the ones who support the author's arguments and the ones who oppose them. Addressing and participating in the multi-threaded conversation among thinkers is very important in the humanities.


Social Sciences

Psychology, Sociology, Anthropology, Linguistics, Economics 


The social sciences try to explain human minds and human societies using scientific method, but they are limited in how they can apply scientific method because of the practical and ethical problems that arise when you try to experiment on or observe human beings, communities, and cultures under controlled conditions. 

    • As with the humanities, social science criteria for valid information sources include a good background in the work of other scholars in the subject area, and no logical fallacies.
    • Statistics are very important for deriving measurable information from the inputs. A valid social science information source will tell you exactly what statistical instruments were used and will present not only the numbers, but also a measure of how certain those numbers are (standard deviation, plus or minus language.)
    • Research methods are important. A valid social science information source will have a whole section that describes methods, and will address things like how the sample was selected, how representative the sample was, and how variables were controlled for. The weaknesses and ambiguities will be addressed.
    • Avoiding bias - usually unconscious - on the part of the experimenter/observer, or in the responses of study participants - is very important.


Sciences

Mathematics, Physics, Chemistry, Biology, Earth Science, Astronomy, Environmental Science


The sciences pursue data about natural phenomena, with the goal of formulating theories that explain and predict those phenomena.

    • Just like the humanities and social sciences, a science information source should include a good background in the work of other scholars in the subject area, and no logical fallacies.
    • Scientific validity requires that the claims be generalizable (externally valid), reproducible and  falsifiable. 
    • Issues of metrology come up in the sciences - how accurately and precisely were they capable of measuring? How much can we trust their instruments? Were they actually measuring what they thought they were measuring?
    • Scientific research methods are extremely rigorous, even compared to social science methods. This is because scientific studies focus on phenomena that can be measured with much less ambiguity. No allowance is made for extra variables. 
    • Just like the social sciences, a science information source needs to avoid bias and use good statistical and research methods. 


Applied Fields


There is probably an applied field for every academic field. Especially consider business, engineering, computer science, nursing, education...

The applied fields are all about carrying out the activities of the everyday world. In the applied fields, consider the criteria for validity that are used by the related academic disciplines. In education, consider the validity criteria for psychology and sociology. In engineering, consider the validity criteria for chemistry and physics. 

Also bear in mind that the applied fields are practical, and so they care about what works in practice.


Some More about Validity


Validity is the question, "How do we know what we know?" It's tricky because every field of study has a different set of value judgments that they make about validity. But there are some things that are generally considered invalid, no matter whether an information source is scholarly or non-scholarly, and no matter what discipline or subject area it is in.


Agenda and Bias = Not Valid


In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The topics of agenda and bias are related to that.

    • Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad! And just because an author subscribes to an ideology does not mean that the information source necessarily has an ideological agenda.

    • Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it is not strictly informational or educational. It is probably actually a persuasive information source or a piece of propaganda, even if it is pretending to be otherwise.

    • Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

This set of targets demonstrates what bias means.

    • On the left is a target where all five shots were accurate. That represents research conclusions that represent the way things actually are. 

    • On the right is a target where all five shots were all over the place. That represents research conclusions that were incorrect, but not biased. The errors didn't systematically skew in any one direction.

    • In the middle is a target where all five shots are clustered over on one side. That represents research conclusions that are not only incorrect, but biased. The errors all systematically skew in one direction.

Three targets. On the left, one is labeled precise and has five shots clustered in and close to the bulls-eye. In the middle, one labeled biased has five shots clustered over to the top right of the bulls-eye. On the right, one labeled random has five shots scattered all over the target.

So while an author may belong to a certain ideology, that can still be OK. It's when the information source has an agenda, such that it's possible to detect bias, that's when you do not want to use the information source for research.

A few common ways bias happens:

    • Confirmation bias is when people, including writers and readers, fail to notice or remember things that do not fit into their assumptions, and of notice and remember things that do fit into their assumptions. (This gets even worse when Google and social media use your clicks and "likes" to decide what to show you!)

    • Confounding bias is when researchers fail to rule out the effects of another variable, that us different from the one they are studying. Confounding bias can be accidental, negligent, or intentional.

    • Selection bias refers to the need for researchers to select a sample from the overall population, when they observe or conduct an experiment. Their sample can fail to represent the entire population for many reasons. Selection bias can be accidental, negligent, or intentional.

    • Observer effect is when people (or animals) behave differently when they realize they are being observed, even if they do not mean to.

    • Publication bias refers to how it can be hard for writers and researchers to report that "nothing happened," or "It did not do what we thought it was going to do." By reading the news or scholarly literature, you can get a distorted view.


Emotional Manipulation = Not Valid


Emotional manipulation means that the information source has not stopped trying to persuade you with facts and logic, but is trying to bypass your mind and get to your "heart" and your "gut."