Data Collection Techniques

Data gathering is the interaction between the software engineer (in this case a business analyst) and the customers (including users). There are many techniques for gathering data, including interviews, meetings, observations, questionnaires, reviewing software, reviewing internal documents, and reviewing external documents. Read this section to differentiate between them and pay attention to the main advantages and disadvantages of each one of these techniques.

1. Single Interview

There are seven techniques we use for data gathering during application development. They are interviews, group meetings, observation, temporary job assignment, questionnaires, review of internal and outside documents, and review of software. Each has a use for which it is best served, and each has limitations to the amount and type of information that can be got from the technique. The technique strengths and weaknesses are summarized in Table 4-2, which is referenced throughout this section. 

In general, you always want to validate the information received from any source through triangulation. Triangulation is obtaining the same information from multiple sources. You might ask the same question in several interviews, compare questionnaire responses to each item, or check in-house and external documents for similar information. When a discrepancy is found, you reverify it with the original and triangulated sources as much as possible. If the information is critical to the application being correctly developed, put the definitions, explanations, or other information in writing and have it approved by the users separately from the other documentation. Next, we discuss each data collection technique.

TABLE 4-2 Summary of Data Collection Techniques

Strengths Weaknesses
Get both qualitative and quantitative information Takes some skill
Get both detail and summary information May obtain biased results
Good method for surfacing requirements Can result in misleading, inaccurate, or irrelevant information

Requires triangulation to verify results

Not useful with large numbers of people to be interviewed (e.g., over 50)
Group Meetings
Strengths Weaknesses
Decisions can be made Decisions with large number of participants can take a long time
Can get both detail and summary information Wastes time
Good for surfacing requirements Interruptions divert attention of participants
Gets many users involved Arguments about turf, politics, etc. can occur

Wrong participants lead to low results
Strengths Weaknesses
Surface unarticulated procedures, decision criteria, reasoning processes Might not be representative time period
Not biased by opinion Behavior might be changed as a result of being observed
Observer gets good problem domain understanding Time consuming
Review Software
Strengths Weaknesses
Good for learning current work procedures as constrained or guided by software design May not be current
Good for identifying questions to ask users about functions-how they work and whether they should be kept May be inaccurate

Time consuming

TABLE 4-2 Summary of Data Collection Techniques (Continued)

Anonymity for respondent Recall may be imperfect
Attitudes and feelings might be more honestly expressed Unanswered questions mean you cannot get the information
Large numbers of people can be surveyed easily Questions might be misinterpreted
Best for limited response, closed-ended questions Reliability or validity may be low
Good for multicultural companies to surface biases, or requirements and design features that should be customized to fit local conventions Might not add useful information to what is already known
Temporary Assignment
Good to learn current context, terminology, procedures, problems
May not include representative work activities or time period
Bases for questions you might not otherwise ask Time consuming

May bias future design work
Review Internal Documents
Strengths Weaknesses
Good for learning history and politics May bias future design work
Explains current context Saves interview /user time
Good for understanding current application Not useful for obtaining attitudes or motivations
Review External Documents
Good for identifying industry trends, surveys, expert opinions, other companies' experiences, and technical information relating to the problem domain May not be relevant

lnformation may not be accurate

May bias future design work

TABLE 4-3 Steps to Conducting a Successful Interview

1. Make an appointment that is at the convenience of the interviewee.
2. Prepare the interview; know the interviewee.
3. Be on time.
4. Have a planned beginning to the interview.
a. Introduce yourself and your role on the project.
b. Use open-ended general questions to begin the discussion.
c. Be interested in all responses, pay attention.
5. Have a planned middle to the interview.
a. Combine open-ended and closed-ended questions to obtain the information you want.
b. Follow-up comments by probing for more detail.
c. Provide feedback to the interviewee in the form of comments, such as, "Let me tell you what I think you mean, ... "
d. Limit your notetaking to avoid distracting the interviewee.
6. Have a planned closing to the interview.
a. Summarize what you have heard. Ask for corrections as needed.
b. Request feedback, note validation, or other actions of interviewee.
      • Give him or her a date by which they will receive information for review.
      • Ask him or her for a date by which the review should be complete.
c. If a follow-up interview is scheduled, confirm the date and time.

A good interview has a beginning, middle, and end. In the beginning, you introduce yourself and put the interviewee at ease. Begin with general questions that are inoffensive and not likely to evoke an emotional response. Pay attention to answers both to get cues for other questions, and to get cues on the honesty and attitude of the interviewee. In the middle, be businesslike and stick to the subject. Get all the information you came for, using the techniques you chose in advance. If some interesting side information emerges, ask if you can talk about it later and then do that. In closing, summarize what you have heard and tell the interviewee what happens next. You may write notes and ask him or her to review them for accuracy. If you do notes, try to get them back for review within 48 hours. Also, have the interviewee commit to the review by a specific date to aid in your time planning. If you say you will follow up with some activity, make sure you do.

Interviews use two types of questions: open-ended and closed-ended. An open-ended question is one that asks for a multi-sentence response. Open-ended questions are good for eliciting descriptions of current and proposed application functions, and for identifying feelings, opinions, and expectations about a proposed application. They can also be used to obtain any lengthy or explanatory answers. An example of open-ended question openings are: "Can you tell me about ... " or "What do you think about ... " or "Can you describe how you use ... ".

A closed-ended question is one which asks for a yes/no or specific answer. Closed-ended questions are good for eliciting factual information or forcing people to take a position on a sensitive issue. An example of a closed-ended question is: "Do you use the monthly report?" A 'yes' response might be followed by an open-ended question, "Can you explain how?"

The questions can be ordered in such a way that the interview might be structured or unstructured (see Table 4-4). A structured interview is one in which the interviewer has an agenda of items to cover, specific questions to ask, and specific information desired. A mix of open and closed questions is used to elicit details of interest. For instance, the interview might start with "Describe the current rental process." The respondent would describe the process, most often using general terms. The interviewer might then ask specific questions, such as, "What is the daily volume of rentals?" Each structured interview is basically the same because the same questions are asked in the same sequence. Tallying the responses is fairly easy because of the structure.

An unstructured interview is one in which the interview unfolds and is directed by responses of the interviewee. The questions tend to be mostly open-ended. There is no set agenda, so the interviewer, who knows the information desired, uses the responses from the open-ended questions to develop ever more specific questions about the topics. The same questions used above as examples for the structured interview might also be used in an unstructured interview; the difference is that above, they are determined as a 'script' in advance. In an unstructured situation, the questions flow from the conversation.

TABLE 4-4 Comparison of Structured and Unstructured Interviews

Structured Unstructured
Uses uniform wording of questions for all respondents Provides greater flexibility in question wording to suit respondent
Easy to administer and evaluate Can be difficult to conduct because interviewer must listen carefully to develop questions about issues that arise spontaneously from answers to questions

May surface otherwise overlooked information
More objective evaluation of respondents and answers to questions Requires practice
Requires little training
Requires little training
Structured Unstructured
Cost of preparation can be high May waste respondent and interviewer time
Respondents do not always accept high level of structure and its mechanical posing of questions Interviewer bias in questions or reporting of results is is more likely
High level of structure is not suited to all situations Extraneous information must be culled through
Reduces spontaneity and ability of interviewer to follow up on comments of interviewee Analysis and interpretation of results may be lengthy

Takes more time to collect essential facts

Structured interviews are most useful when you know the information desired in advance of the interview (see Table 4-4). Conversely, unstructured interviews are most useful when you cannot anticipate the topics or specific outcome. A typical series of interviews with a user client begins with unstructured interviews to give you an understanding of the problem domain. The interviews get progressively structured and focused as the information you need to complete the analysis also gets more specific.

User interview results should always be communicated back to the interviewee in a short period of time. The interviewee should be given a deadline for their review. If the person and/or information are critical to the application design being correct, you should ask for comments even after the deadline is missed. If the person is not key in the development, the deadline date signifies a period during which you will accept changes, after the date you continue work, assuming the information is correct. 

 It is good practice to develop diagram( s) as part of the interview documentation. At the beginning of the next interview session, you discuss the diagram(s) with the user and give him or her any written notes to verify at a later time. You get immediate feedback on the accuracy of the graphic and your understanding of the application. The benefits of this approach are both technical and psychological. From a technical perspective, you are constantly verifying what you have been told. By the time the analysis is complete, both you and the client have confidence that the depicted application processing is correct and complete. From a psychological perspective, you increase user confidence in your analytical ability by demonstrating your problem understanding. Each time you improve the diagram and deepen the analysis, you also increase user confidence that you will build an application that answers his or her need.

Interviews are useful for obtaining both qualitative and quantitative information (see Table 4-2). The types of qualitative information are opinions, beliefs, attitudes, policies, and narrative descriptions. The types of quantitative information include frequencies, numbers, and quantities of items to be tracked or used in the application. 

Interviews, and other forms of data collection, can give you misleading, inaccurate, politically motivated, or irrelevant information (see Table 4-2). You need to learn to read the person's body language and behavior to decide on further needs for the same information. Table 4-5 lists respondent behaviors you might see in an interview and the actions you might take in dealing with the behaviors.

For instance, if you suspect the interviewee of lying or 'selectively remembering' information, try to cross-check the answers with other, more reliable sources. If the interview information is found to be false, ask the interviewee to please explain the differences between his or her answers and the other information. The session does not need to be a confrontation, rather, it is a simple request for explanation. Be careful not to accuse or condemn, simply try to get the correct information.

Persistence and triangulation are key to getting complete, accurate information. You are not required to become 'friends' with the application users, but interviews are smoother, yield more information for the time spent, and usually have less' game-playing' if you are 'friendly' than if you are viewed as distant, overly-objective, or noninterested.

Source: Sue Conger,
Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 License.