Data Collection Techniques
Data gathering is the interaction between the software engineer (in this case a business analyst) and the customers (including users). There are many techniques for gathering data, including interviews, meetings, observations, questionnaires, reviewing software, reviewing internal documents, and reviewing external documents. Read this section to differentiate between them and pay attention to the main advantages and disadvantages of each one of these techniques.
1. Single Interview
There are seven techniques we use for data gathering during application development. They are interviews, group meetings, observation, temporary job assignment, questionnaires, review of internal and outside documents, and review of software. Each has a use for which it is best served, and each has limitations to the amount and type of information that can be got from the technique. The technique strengths and weaknesses are summarized in Table 4-2, which is referenced throughout this section.
In general, you always want to validate the information received from any source through triangulation. Triangulation is obtaining the same information from multiple sources. You might ask the same question in several interviews, compare questionnaire responses to each item, or check in-house and external documents for similar information. When a discrepancy is found, you reverify it with the original and triangulated sources as much as possible. If the information is critical to the application being correctly developed, put the definitions, explanations, or other information in writing and have it approved by the users separately from the other documentation. Next, we discuss each data collection technique.
TABLE 4-2 Summary of Data Collection Techniques
Interviews | |
Strengths | Weaknesses |
Get both qualitative and quantitative information | Takes some skill |
Get both detail and summary information | May obtain biased results |
Good method for surfacing requirements | Can result in misleading, inaccurate, or irrelevant information
|
Requires triangulation to verify results |
|
Not useful with large numbers of people to be interviewed (e.g., over 50) |
|
Group Meetings | |
Strengths | Weaknesses |
Decisions can be made | Decisions with large number of participants can take a long time |
Can get both detail and summary information | Wastes time |
Good for surfacing requirements | Interruptions divert attention of participants |
Gets many users involved | Arguments about turf, politics, etc. can occur |
Wrong participants lead to low results | |
Observation | |
Strengths | Weaknesses |
Surface unarticulated procedures, decision criteria, reasoning processes | Might not be representative time period |
Not biased by opinion | Behavior might be changed as a result of being observed |
Observer gets good problem domain understanding | Time consuming |
Review Software | |
Strengths | Weaknesses |
Good for learning current work procedures as constrained or guided by software design | May not be current |
Good for identifying questions to ask users about functions-how they work and whether they should be kept | May be inaccurate |
Time consuming |
TABLE 4-2 Summary of Data Collection Techniques (Continued)
Questionnaire | |
Strengths |
Weaknesses |
Anonymity for respondent | Recall may be imperfect |
Attitudes and feelings might be more honestly expressed | Unanswered questions mean you cannot get the information
|
Large numbers of people can be surveyed easily | Questions might be misinterpreted |
Best for limited response, closed-ended questions | Reliability or validity may be low |
Good for multicultural companies to surface biases, or requirements and design features that should be customized to fit local conventions | Might not add useful information to what is already known |
Temporary Assignment | |
Strengths |
Weaknesses |
Good to learn current context, terminology, procedures, problems |
May not include representative work activities or time period |
Bases for questions you might not otherwise ask | Time consuming |
May bias future design work |
|
Review Internal Documents | |
Strengths | Weaknesses |
Good for learning history and politics | May bias future design work |
Explains current context | Saves interview /user time |
Good for understanding current application | Not useful for obtaining attitudes or motivations |
Review External Documents | |
Strengths |
Weaknesses |
Good for identifying industry trends, surveys, expert opinions, other companies' experiences, and technical information relating to the problem domain | May not be relevant |
lnformation may not be accurate |
|
May bias future design work |
TABLE 4-3 Steps to Conducting a Successful Interview 1. Make an appointment that is at the convenience of the interviewee. 2. Prepare the interview; know the interviewee. 3. Be on time. 4. Have a planned beginning to the interview. a. Introduce yourself and your role on the project. 5. Have a planned middle to the interview.
b. Use open-ended general questions to begin the discussion. c. Be interested in all responses, pay attention. a. Combine open-ended and closed-ended questions to obtain the information you want.
b. Follow-up comments by probing for more detail. c. Provide feedback to the interviewee in the form of comments, such as, "Let me tell you what I think you mean, ... " d. Limit your notetaking to avoid distracting the interviewee.
6. Have a planned closing to the interview.
a. Summarize what you have heard. Ask for corrections as needed.
b. Request feedback, note validation, or other actions of interviewee.
c. If a follow-up interview is scheduled, confirm the date and time.
|
A good interview has a beginning, middle, and end. In the beginning, you introduce yourself and put the interviewee at ease. Begin with general questions that are inoffensive and not likely to evoke an emotional response. Pay attention to answers both to get cues for other questions, and to get cues on the honesty and attitude of the interviewee. In the middle, be businesslike and stick to the subject. Get all the information you came for, using the techniques you chose in advance. If some interesting side information emerges, ask if you can talk about it later and then do that. In closing, summarize what you have heard and tell the interviewee what happens next. You may write notes and ask him or her to review them for accuracy. If you do notes, try to get them back for review within 48 hours. Also, have the interviewee commit to the review by a specific date to aid in your time planning. If you say you will follow up with some activity, make sure you do.
Interviews use two types of questions: open-ended and closed-ended. An open-ended question is one that asks for a multi-sentence response. Open-ended questions are good for eliciting descriptions of current and proposed application functions, and for identifying feelings, opinions, and expectations about a proposed application. They can also be used to obtain any lengthy or explanatory answers. An example of open-ended question openings are: "Can you tell me about ... " or "What do you think about ... " or "Can you describe how you use ... ".
A closed-ended question is one which asks for a yes/no or specific answer. Closed-ended questions are good for eliciting factual information or forcing people to take a position on a sensitive issue. An example of a closed-ended question is: "Do you use
the monthly report?" A 'yes' response might be followed by an open-ended question, "Can you explain how?"
The questions can be ordered in such a way that
the interview might be structured or unstructured
(see Table 4-4). A structured interview is one in
which the interviewer has an agenda of items to
cover, specific questions to ask, and specific information desired. A mix of open and closed questions
is used to elicit details of interest. For instance, the
interview might start with "Describe the current
rental process." The respondent would describe the
process, most often using general terms. The interviewer might then ask specific questions, such as,
"What is the daily volume of rentals?" Each structured interview is basically the same because the
same questions are asked in the same sequence. Tallying the responses is fairly easy because of the
structure.
An unstructured interview is one in which the
interview unfolds and is directed by responses of
the interviewee. The questions tend to be mostly
open-ended. There is no set agenda, so the interviewer, who knows the information desired, uses the
responses from the open-ended questions to develop
ever more specific questions about the topics. The same questions used above as examples for the
structured interview might also be used in an unstructured interview; the difference is that above,
they are determined as a 'script' in advance. In an
unstructured situation, the questions flow from the
conversation.
TABLE 4-4 Comparison of Structured and Unstructured Interviews
Strengths | |
Structured | Unstructured |
Uses uniform wording of questions for all respondents | Provides greater flexibility in question wording to suit respondent |
Easy to administer and evaluate | Can be difficult to conduct because interviewer must listen carefully to develop questions about issues that arise spontaneously from answers to questions |
May surface otherwise overlooked information | |
More objective evaluation of respondents and answers to questions | Requires practice |
Requires little training | |
Requires little training | |
Weaknesses | |
Structured | Unstructured |
Cost of preparation can be high | May waste respondent and interviewer time |
Respondents do not always accept high level of structure and its mechanical posing of questions | Interviewer bias in questions or reporting of results is is more likely |
High level of structure is not suited to all situations | Extraneous information must be culled through |
Reduces spontaneity and ability of interviewer to follow up on comments of interviewee | Analysis and interpretation of results may be lengthy |
Takes more time to collect essential facts |
Structured interviews are most useful when you know the information desired in advance of the interview (see Table 4-4). Conversely, unstructured interviews are most useful when you cannot anticipate the topics or specific outcome. A typical series of interviews with a user client begins with unstructured interviews to give you an understanding of the problem domain. The interviews get progressively structured and focused as the information you need to complete the analysis also gets more specific.
User interview results should always be communicated back to the interviewee in a short period of time. The interviewee should be given a deadline for their review. If the person and/or information are critical to the application design being correct, you should ask for comments even after the deadline is missed. If the person is not key in the development, the deadline date signifies a period during which you will accept changes, after the date you continue work, assuming the information is correct.
It is good practice to develop diagram( s) as part of the interview documentation. At the beginning of the next interview session, you discuss the diagram(s) with the user and give him or her any written notes to verify at a later time. You get immediate feedback on the accuracy of the graphic and your understanding of the application. The benefits of this approach are both technical and psychological. From a technical perspective, you are constantly verifying what you have been told. By the time the analysis is complete, both you and the client have confidence that the depicted application processing is correct and complete. From a psychological perspective, you increase user confidence in your analytical ability by demonstrating your problem understanding. Each time you improve the diagram and deepen the analysis, you also increase user confidence that you will build an application that answers his or her needs.
Interviews are useful for obtaining both qualitative and quantitative information (see Table 4-2). The types of qualitative information are opinions, beliefs, attitudes, policies, and narrative descriptions. The types of quantitative information include frequencies, numbers, and quantities of items to be tracked or used in the application.
Interviews, and other forms of data collection, can give you misleading, inaccurate, politically motivated, or irrelevant information (see Table 4-2). You need to learn to read the person's body language and behavior to decide on further needs for the
same information. Table 4-5 lists respondent behaviors you might see in an interview and the actions you might take in dealing with the behaviors.
For instance, if you suspect the interviewee of lying or 'selectively remembering' information, try to cross-check the answers with other, more reliable sources. If the interview information is found to be false, ask the interviewee to please explain the differences between his or her answers and the other information. The session does not need to be a confrontation, rather, it is a simple request for explanation. Be careful not to accuse or condemn, simply try to get the correct information.
Persistence and triangulation are key to getting complete, accurate information. You are not required to become 'friends' with the application users, but interviews are smoother, yield more information for the time spent, and usually have less' game-playing'
if you are 'friendly' than if you are viewed as distant, overly-objective, or uninterested.
Source: Sue Conger, https://resources.saylor.org/CS/CS302/OER/The_New_Software_Engineering.pdf
This work is licensed under a Creative Commons Attribution 3.0 License.