Research Skills Tutorial

Credible Sources

Evaluating information for research is just a specialized, advanced form of the same critical thinking skills you already use.

Why do we need to critically evaluate information sources?

No source of information is guaranteed to be trustworthy. You always need to use your own educated judgment, even with scholarly articles from library databases.
Some sources of information are more trustworthy than others, but it can be hard to tell from appearances.
Evaluating information using critical thinking will save time and effort by filtering out materials you should not use.
Your critical thinking will show up in your writing and you will get better grades.

Your professors may tell you to find credible information sources. This is a subjective term with many definitions, but the general consensus is that credibility is a combination of reliability, authority, validity and accuracy.

Reliability means that the entities that sponsored, supported, or published the information source have a reputation for quality, and integrity.
Authority means that the creator of the information source is an expert in the field.
Validity means that the research in the information source was conducted in ways that are commonly accepted for that field of study.
Accuracy means that you have ways of determining the correctness of the information in the information source.

Source: SUNY Empire State College Librarians, https://subjectguides.esc.edu/researchskillstutorial
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.

What Does the Information Source Tell You About Itself?

Scholarly Information Sources

If you have a scholarly information source like a monograph or a peer reviewed journal article, you are in luck. When these kinds of information sources are published, they have your needs in mind.

So you know:

It was either peer reviewed or editorially reviewed (and you can find out more by going to the publisher's or journal's web site.)
It was produced by a subject expert and for an audience of subject experts.
It was produced for the purpose of informing and educating. It is supposed to be objective and unbiased. (It is not perfect, but it is still better than if there were no such expectation!)

Plus, they make it easy to find out:

Who wrote this?
When was this written?
Who provided funding for this?
Who published this?
What potential conflicts of interest are there?
What kinds of sources did they get their information from?

Non-scholarly but Formally Published Information Sources

Here are some types of information sources that are formally published but non-scholarly:

Newspapers;
Documentaries;
Magazines;
Books from commercial publishers.

For these kinds of information sources, you will easily be able to find out:

When it was written?
Who wrote it?
Who published it?
Maybe what sources they got their information from.

But non-scholarly information sources are not held to the same standards as scholarly ones. There is a whole range, from highly credible to thoroughly disreputable.

They may or may not be objective and unbiased.
They may or may not attempt to mislead or manipulate you.
They may or may not use good sources and check their facts.

You will have to make a judgment based on two things:

The reputation of the organization that published the information source (book publisher, magazine, etc.)
The actual contents of the information source.

Information Sources that were Never Formally Published

It used to be that without the resources of a large organization like a journal or book publisher, it was hard for an author to get their information source out to an audience. So almost all research was done using books and articles (and of course, primary sources, which were sometimes very hard to get to.)

Now we have the Internet, and if you have a connection, you can make a web page and put whatever you want up there. If you go to college, your college may give you an opportunity to put your papers into an online repository. Organizations and companies of all kinds put many of their documents online - everything from policies and procedures, to data and reports. Ephemera like brochures and pamphlets that used to be throw-away pieces of paper now linger on the web for years after they have gone out of date. Famous authors have blogs, and so do politicians, subject experts, and schoolchildren, and business people trying to make a name for themselves.

All of these things are available to you as information sources. Some of them may be good for research, and many others are not.

They do not have a standard way to tell you who created them (and they may not tell you at all.)
They do not have a standard way to tell you when they were created (and they may not tell you at all.)
They do not have a standard way to disclose any sources of financial support or biases (and they may not tell you at all.)
They do not have a standard way to tell you what information sources they used, if any (and they may not tell you at all.)
They may actually be plagiarizing text or images or even copying whole other sources and not giving you any way to find out.
Depending on who produced the content, there may not be oversight from a larger organization with a reputation to worry about, which might have helped to ensure quality and ethical integrity of the content.

Plus the useful informational content is mixed in with content that was made to sell you things, manipulate you into changing your mind or changing your actions, and straight up deceive you.

Long story short, you have to be really careful with information sources that were never formally published. Usually that means "websites," but the same rules apply to:

self-published or vanity press books;
public access TV shows;
pamphlets, newsletters, etc. produced by an organization to promote itself or its cause.

Reliability

In general, reliability means consistency, quality and integrity.

In the case of a scholarly information source, like a monograph or scholarly journal article, you know:

It was either peer reviewed or editorially reviewed (and you can find out more by going to the publisher's or journal's website).
It was produced by a subject expert for an audience of subject experts.
It was produced for the purpose of informing and educating. It is supposed to be objective and unbiased (though this is sometimes not the case).

Plus, with scholarly information sources, it is easy to identify:

Who wrote it?
When was it written?
Who provided funding?
Who published it?
What potential conflicts of interest are there?
What kinds of sources were used in the research?

You can learn a lot about an information source by looking at the the organization(s) responsible for producing it. Some questions to ask when encountering non-scholarly information sources:

Is their reputation for putting out good information their first priority, or do they have other priorities? In particular, be wary of any organization that is trying to sell something, raise money for something, win an election, win a court battle, win a war, win a battle of public opinion, save souls, or save the world. Not that there is anything wrong with those things necessarily, but it is very hard for anyone to put objective scholarship first when they have those priorities.
Do they value honesty and integrity?
Do they value research and scholarship?

Authority

How do we know that the author or authors are experts? We can examine their curriculum vitae (C.V.), their publication history, and the response their work has received from other experts in the field.

Examine the Author's Curriculum Vitae

From the author's resumé or C.V. we can learn that:

From the author's publication history we can learn whether:

A curriculum vitae is a combination resumé and publication list for people who work in academia. Most people put their curriculum vitae online, whether on a personal website, a jobs website, or their biographical page on their institution's website. To locate one, you can usually just do a Web search (e.g., "G.J. Barker-Benfield" "curriculum vitae")

What to look for:

First, you want to make sure there is a resumé or C.V. Unless the author died or retired before 2000, there should be one online. If not, they may not actually be a scholar.
Second, you want to read about their education. Do they have one or more PhDs?
Third, look at their employment history. Have they gotten jobs researching and teaching at the college level in their subject area?
Fourth, what kinds of honors they have achieved?
Fifth, how much they have published about your topic of interest? Also, what publishers and journals have accepted their work? Some publishers and journals have stricter standards than others, and it really says something good about an author's research if is being published in them.

Use Google Scholar to Find Citation Counts

There are special (and expensive) tools that scholars can use to measure impact factor, but you will not need those. To get a general idea of how many times a particular work has been cited (its impact factor), you can use Google Scholar. Search for the author(s). Then, under the works listed in the search results, look for the phrase "Cited by" follow by a number (highlighted in red below). This is the number of times other published scholars have referenced this work in their research. You can click it to view a list of the works that cited it. Run some searches on articles on the same topic, published at the same time to get a basis for comparison.

Screenshot of a Google Scholar search results page showing both books and articles by author barker-benfield. Under each search result is a Cited By number (which you can click to see what information sources actually cited it.)

Use Databases to Find Reviews of Monographs in Journals

A scholarly book of any significance is going to be reviewed in scholarly journals in that subject area, and whether a work has received a negative or positive review matters less than the actual strengths and weaknesses that the review identifies.

To find scholarly reviews: [if you have access to a database such as OneSearch, through your academic institution or otherwise]

Enter title of the book (in quotation marks) and search.
Use the navigation bar on the left side of the search results to narrow by Source Type to Reviews.
If your book has a very common title, you might want to narrow things down.

Book reviews usually appear as follows in the search results list:

Screenshot of the OneSearch search results list with two book reviews highlighted. One says The Culture of Sensibility (Book Review) where "book review" is in parentheses. The other says "The Culture of Sensibility: Sex and Society in Eighteenth-Century Britain (Book), where book is in parentheses.

Finding Reviews of Non-scholarly Books

Most non-scholarly sources are never reviewed. You will not find reviews for websites, blogs, or the many kinds of gray literature that are found on the Web. However, many high quality non-fiction books will be reviewed in places such as the New York Times Book Review.

Use OneSearch to Find Critical Responses to Scholarly Articles

When you have a scholarly article that covers one point of view about a topic, you may wish to find articles that criticize or rebut it.

To find these kinds of critical responses: [if you have access to a database such as OneSearch, through your academic institution or otherwise]

Enter the article title (in quotation marks).
At the top of your list of preliminary search results, click the Advanced Search link underneath the search box.
Look at the date that the original article was published and limit your search to that year and beyond. For example, if the article came out in 2009, set your search to 2009-2012 (or the current year.)
Set the pull-down menu next to the search box to Title (TI).
Search for (“author’s last name” OR “short title of the article”) AND (“critique of” OR “response to” OR “response from”).

For example, if your original article is “Two Faces of Power” by Peter Bachrach and Morton Baratz, enter this:

(“two faces of power” OR (bachrach AND baratz)) AND (“critique of” OR “response to” OR “response from”)

Bear in mind that not every article receives direct responses. If you have tried several variations on the search and have not had any luck, even with a librarian’s help, you are probably better off just searching for articles on the same topic to see what others are saying about it.

Validity

Validity asks the question, "How do we know what we know?" Every field of study answers that question differently, but there are some ideas that are generally considered invalid, no matter whether an information source is scholarly or non-scholarly, and no matter what discipline or subject area it falls under.

In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The concepts of ideology, agenda and bias are related to that.

Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad. And just because an author subscribes to an ideology does not mean that the information source necessarily has an ideological agenda.
Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it is not strictly informational or educational. It is probably actually a persuasive information source or a piece of propaganda, even if it is pretending to be otherwise.
Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

A credible information source will not try to tell you how to feel about the information.

Validity by Discipline

Humanities

Literature, the Arts, History, Philosophy, Theology

In the humanities, the author is either constructing a worldview, or (more often) adding to, refining, and correcting a worldview that other scholars have created. That worldview needs to be self-consistent and consistent with the evidence. It needs to be able to support new discoveries and insights. And it often needs to be beautiful.

Here are some criteria for validity that can be generalized in the humanities:

Good use of primary sources. Appropriate sources are chosen. They are translated and interpreted correctly. Provenance is clear.
Anecdotes are fine as long as the author makes it very clear how they represent general principles.
Drawing connections and creating a persuasive (and beautiful) argument in favor of them.
No logical fallacies.
Exhaustive knowledge of relevant secondary sources, both the ones who support the author's arguments and the ones who oppose them. Addressing and participating in the multi-threaded conversation among thinkers is very important in the humanities.

Social Sciences

Psychology, Sociology, Anthropology, Linguistics, Economics

The social sciences try to explain human minds and human societies using scientific method, but they are limited in how they can apply scientific method because of the practical and ethical problems that arise when you try to experiment on or observe human beings, communities, and cultures under controlled conditions.

As with the humanities, social science criteria for valid information sources include a good background in the work of other scholars in the subject area, and no logical fallacies.
Statistics are very important for deriving measurable information from the inputs. A valid social science information source will tell you exactly what statistical instruments were used and will present not only the numbers, but also a measure of how certain those numbers are (standard deviation, plus or minus language.)
Research methods are important. A valid social science information source will have a whole section that describes methods, and will address things like how the sample was selected, how representative the sample was, and how variables were controlled for. The weaknesses and ambiguities will be addressed.
Avoiding bias - usually unconscious - on the part of the experimenter/observer, or in the responses of study participants - is very important.

Sciences

Mathematics, Physics, Chemistry, Biology, Earth Science, Astronomy, Environmental Science

The sciences pursue data about natural phenomena, with the goal of formulating theories that explain and predict those phenomena.

Just like the humanities and social sciences, a science information source should include a good background in the work of other scholars in the subject area, and no logical fallacies.
Scientific validity requires that the claims be generalizable (externally valid), reproducible and falsifiable.
Issues of metrology come up in the sciences - how accurately and precisely were they capable of measuring? How much can we trust their instruments? Were they actually measuring what they thought they were measuring?
Scientific research methods are extremely rigorous, even compared to social science methods. This is because scientific studies focus on phenomena that can be measured with much less ambiguity. No allowance is made for extra variables.
Just like the social sciences, a science information source needs to avoid bias and use good statistical and research methods.

Applied Fields

There is probably an applied field for every academic field. Especially consider business, engineering, computer science, nursing, education...

The applied fields are all about carrying out the activities of the everyday world. In the applied fields, consider the criteria for validity that are used by the related academic disciplines. In education, consider the validity criteria for psychology and sociology. In engineering, consider the validity criteria for chemistry and physics.

Also bear in mind that the applied fields are practical, and so they care about what works in practice.

Some More about Validity

Validity is the question, "How do we know what we know?" It's tricky because every field of study has a different set of value judgments that they make about validity. But there are some things that are generally considered invalid, no matter whether an information source is scholarly or non-scholarly, and no matter what discipline or subject area it is in.

Agenda and Bias = Not Valid

In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The topics of agenda and bias are related to that.

Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad! And just because an author subscribes to an ideology does not mean that the information source necessarily has an ideological agenda.
Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it is not strictly informational or educational. It is probably actually a persuasive information source or a piece of propaganda, even if it is pretending to be otherwise.
Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

This set of targets demonstrates what bias means.

On the left is a target where all five shots were accurate. That represents research conclusions that represent the way things actually are.
On the right is a target where all five shots were all over the place. That represents research conclusions that were incorrect, but not biased. The errors didn't systematically skew in any one direction.
In the middle is a target where all five shots are clustered over on one side. That represents research conclusions that are not only incorrect, but biased. The errors all systematically skew in one direction.

So while an author may belong to a certain ideology, that can still be OK. It's when the information source has an agenda, such that it's possible to detect bias, that's when you do not want to use the information source for research.

A few common ways bias happens:

Confirmation bias is when people, including writers and readers, fail to notice or remember things that do not fit into their assumptions, and of notice and remember things that do fit into their assumptions. (This gets even worse when Google and social media use your clicks and "likes" to decide what to show you!)
Confounding bias is when researchers fail to rule out the effects of another variable, that us different from the one they are studying. Confounding bias can be accidental, negligent, or intentional.
Selection bias refers to the need for researchers to select a sample from the overall population, when they observe or conduct an experiment. Their sample can fail to represent the entire population for many reasons. Selection bias can be accidental, negligent, or intentional.
Observer effect is when people (or animals) behave differently when they realize they are being observed, even if they do not mean to.
Publication bias refers to how it can be hard for writers and researchers to report that "nothing happened," or "It did not do what we thought it was going to do." By reading the news or scholarly literature, you can get a distorted view.

Emotional Manipulation = Not Valid

Emotional manipulation means that the information source has not stopped trying to persuade you with facts and logic, but is trying to bypass your mind and get to your "heart" and your "gut."

Site:	Saylor Academy
Course:	PHIL102: Introduction to Critical Thinking and Logic
Book:	Research Skills Tutorial

Printed by:	Guest user
Date:	Sunday, 6 April 2025, 10:52 AM

Description

Table of contents