Data Collection Techniques

Single Interview

There are seven techniques we use for data gathering during application development. They are interviews, group meetings, observation, temporary job assignment, questionnaires, review of internal and outside documents, and review of software. Each has a use for which it is best served, and each has limitations to the amount and type of information that can be got from the technique. The technique strengths and weaknesses are summarized in Table 4-2, which is referenced throughout this section. 

In general, you always want to validate the information received from any source through triangulation. Triangulation is obtaining the same information from multiple sources. You might ask the same question in several interviews, compare questionnaire responses to each item, or check in-house and external documents for similar information. When a discrepancy is found, you reverify it with the original and triangulated sources as much as possible. If the information is critical to the application being correctly developed, put the definitions, explanations, or other information in writing and have it approved by the users separately from the other documentation. Next, we discuss each data collection technique.

TABLE 4-2 Summary of Data Collection Techniques

TABLE 4-2 Summary of Data Collection Techniques (Continued)

Interviews use two types of questions: open-ended and closed-ended. An open-ended question is one that asks for a multi-sentence response. Open-ended questions are good for eliciting descriptions of current and proposed application functions, and for identifying feelings, opinions, and expectations about a proposed application. They can also be used to obtain any lengthy or explanatory answers. An example of open-ended question openings are: "Can you tell me about ... " or "What do you think about ... " or "Can you describe how you use ... ".

A closed-ended question is one which asks for a yes/no or specific answer. Closed-ended questions are good for eliciting factual information or forcing people to take a position on a sensitive issue. An example of a closed-ended question is: "Do you use the monthly report?" A 'yes' response might be followed by an open-ended question, "Can you explain how?"

The questions can be ordered in such a way that the interview might be structured or unstructured (see Table 4-4). A structured interview is one in which the interviewer has an agenda of items to cover, specific questions to ask, and specific information desired. A mix of open and closed questions is used to elicit details of interest. For instance, the interview might start with "Describe the current rental process." The respondent would describe the process, most often using general terms. The interviewer might then ask specific questions, such as, "What is the daily volume of rentals?" Each structured interview is basically the same because the same questions are asked in the same sequence. Tallying the responses is fairly easy because of the structure.

An unstructured interview is one in which the interview unfolds and is directed by responses of the interviewee. The questions tend to be mostly open-ended. There is no set agenda, so the interviewer, who knows the information desired, uses the responses from the open-ended questions to develop ever more specific questions about the topics. The same questions used above as examples for the structured interview might also be used in an unstructured interview; the difference is that above, they are determined as a 'script' in advance. In an unstructured situation, the questions flow from the conversation.

TABLE 4-4 Comparison of Structured and Unstructured Interviews

Structured interviews are most useful when you know the information desired in advance of the interview (see Table 4-4). Conversely, unstructured interviews are most useful when you cannot anticipate the topics or specific outcome. A typical series of interviews with a user client begins with unstructured interviews to give you an understanding of the problem domain. The interviews get progressively structured and focused as the information you need to complete the analysis also gets more specific.

User interview results should always be communicated back to the interviewee in a short period of time. The interviewee should be given a deadline for their review. If the person and/or information are critical to the application design being correct, you should ask for comments even after the deadline is missed. If the person is not key in the development, the deadline date signifies a period during which you will accept changes, after the date you continue work, assuming the information is correct. 

 It is good practice to develop diagram( s) as part of the interview documentation. At the beginning of the next interview session, you discuss the diagram(s) with the user and give him or her any written notes to verify at a later time. You get immediate feedback on the accuracy of the graphic and your understanding of the application. The benefits of this approach are both technical and psychological. From a technical perspective, you are constantly verifying what you have been told. By the time the analysis is complete, both you and the client have confidence that the depicted application processing is correct and complete. From a psychological perspective, you increase user confidence in your analytical ability by demonstrating your problem understanding. Each time you improve the diagram and deepen the analysis, you also increase user confidence that you will build an application that answers his or her needs.

Interviews are useful for obtaining both qualitative and quantitative information (see Table 4-2). The types of qualitative information are opinions, beliefs, attitudes, policies, and narrative descriptions. The types of quantitative information include frequencies, numbers, and quantities of items to be tracked or used in the application. 

Interviews, and other forms of data collection, can give you misleading, inaccurate, politically motivated, or irrelevant information (see Table 4-2). You need to learn to read the person's body language and behavior to decide on further needs for the same information. Table 4-5 lists respondent behaviors you might see in an interview and the actions you might take in dealing with the behaviors.

For instance, if you suspect the interviewee of lying or 'selectively remembering' information, try to cross-check the answers with other, more reliable sources. If the interview information is found to be false, ask the interviewee to please explain the differences between his or her answers and the other information. The session does not need to be a confrontation, rather, it is a simple request for explanation. Be careful not to accuse or condemn, simply try to get the correct information.

Persistence and triangulation are key to getting complete, accurate information. You are not required to become 'friends' with the application users, but interviews are smoother, yield more information for the time spent, and usually have less' game-playing' if you are 'friendly' than if you are viewed as distant, overly-objective, or uninterested.

Source: Sue Conger, https://resources.saylor.org/CS/CS302/OER/The_New_Software_Engineering.pdf
This work is licensed under a Creative Commons Attribution 3.0 License.

Meetings

Meetings are gatherings of three or more people for a fixed period to discuss a small number of topics and sometimes to reach consensus decisions. Meetings can both complement and replace interviews. They complement interviews by allowing a group verification of individual interview results. They can replace interviews by providing a forum for users to collectively work out the requirements and alternatives for an application. Thus, meetings can be useful for choosing between alternatives, verifying findings, and for soliciting application ideas and requirements.

Meetings can also be a colossal waste of time (see Table 4-2). In general, the larger the meeting, the fewer the decisions and the longer they take. Therefore, before having a meeting, a meeting plan should be developed. The agenda should be defined and circulated in advance to all participants. The number of topics should be kept to between one and five. The meeting should be for a fixed period with specific checkpoints for decisions required. In general, meetings should be no longer than two hours to maintain the attention of the participants. The agenda should be followed and the meeting moved along by the project manager or SE, whoever is running the meeting. Minutes should be generated and circulated to summarize the discussion and decisions. Any follow-up items should identify the responsible person(s) and a date by which the item should be resolved.

Meetings are useful for surfacing requirements, reaching consensus, and obtaining both detailed and summary information (see Table 4-2). If decisions are desired, it is important to ask the decision makers to attend and to tell them in advance of the goals for the meeting. If the wrong people participate, time is wasted and the decisions are not made at the meeting.

Joint application development (lAD) is a special form of meeting in which users and technicians meet continuously over several days to identify application requirements (see Figure 4-3). Before a JAD session, users are trained in the techniques used to document requirements, in particular, diagrams for data and processes are taught. Then, in preparation for the JAD session, the users document their own jobs using the techniques and collecting copies of all forms, inputs, reports, memos, faxes, and so forth used in performing their job.

 TABLE 4-5 Interviewee Behaviors and Interviewer Response

FIGURE 4-3 JAD Meeting

Observation

Observation is the manual or automated monitoring of one or more persons' work. In manual observation, a person sits with the individual(s) being observed and takes notes of the activities and steps performed during the work (see Table 4-2). In automated observation, a computer keeps track of software used, e-mail correspondence and partners, and actions performed using a computer. Computer log files are then analyzed to describe the work process based on the software and procedures used. 

Temporary Job Assignment

There is no substitute for experience. With a temporary job assignment, you get a more complete appreciation for the tasks involved and the complexity of each than you ever could by simply talking about them. Also, you learn firsthand the terminology and the context of its use (see Table 4-2). The purpose, then, of temporary job assignment is to make the assignee more knowledgeable about the problem domain. 

Questionnaire

A questionnaire is a paper-based or computer-based form of interview. Questionnaires are used to obtain information from a large number of people. The major advantage of a questionnaire is anonymity, thus leading to more honest answers than might be got through interviews. Also, standardized questions provide reliable data upon which decisions can be based.

Questionnaire items, like interviews, can be either open-ended or closed-ended. Recall that open-ended questions have no specific response intended. Open-ended questions are less reliable for obtaining complete information about factual information and are subject to recall difficulties, selective perception, and distortion by the person answering the question. Since the interviewer neither knows the specific respondent nor has contact with the respondent, open-ended questions that lead to other questions might go unanswered. An example of an open-ended question is: "List all new functions which you think the new application should do".

A closed-ended question is one which asks for a yes/no or graded specific answer. For example, "Do you agree with the need for a history file?" would obtain either a yes or no response. Questionnaire construction is a learned skill that requires consideration of the reliability and validity of the instrument. Reliability is the extent to which a questionnaire is free of measurement errors. This means that if a reliable questionnaire were given to the same group several times, the same answers would be obtained. If a questionnaire is unreliable, repeated measurement would result in different answers every time. 

Questionnaires that try to measure mood, satisfaction, and other emotional characteristics of the respondent tend to be unreliable because they are influenced by how the person feels that day. You improve reliability by testing the questionnaire. When the responses are tallied, statistical techniques are used to verify the reliability of related sets of questions.
Validity is the extent to which the questionnaire measures what you think you are measuring. For instance, assume you want to know the extent to which a CASE tool is being used in both frequency of use and number of functions used. Asking the question, "How well do you use the CASE tool?" might obtain a subjective assessment based on the individual's self-perception. If they perceive themselves as skilled, they might answer that they are extensive users. If they perceive themselves as novices, they might answer that they do not use the tool extensively. A better set of questions would be "How often do you use the CASE tool?" and "How many functions of the tool do you use? Please list the functions you use". These questions specifically ask for numbers which are objective and not tied to an individual's self-perception. The list of functions verifies the numbers and provides the most specific answer possible.

Some guidelines for developing questionnaires are summarized in Table 4-6 and discussed here. First, determine the information to be collected, what facts are required, and what feelings, lists of items, or nonfactual information is desired. Group the items by type of information obtained, type of questions to be asked, or by topic area. Choose a grouping that makes sense for the specific project. 

For each piece of information, choose the type of question that best obtains the desired response. Select open-ended questions for general, lists, and nonfactual information. Select closed-ended questions to elicit specific, factual information, or single answers.

Compose a question for each item. For a closed-ended question, develop a response scale. The five-response Likert-like scale is the most frequently used. The low and high ends of the scale indicate the poles of responses, for instance, Totally Disagree and Totally Agree. The middle response is usually neutral, for instance, Neither Agree Nor Disagree. Examine the question and ask yourself if it has any words that might not be interpreted as you mean them. What happens if the respondent does not know the answer to your question? Do you need a response that says, I Don't Know? Is a preferred response hidden in the question? Are the response choices complete and ordered properly? Does the question have the same meaning for every department and possible respondent? If the answers to any of these questions indicate a problem, reword the question to remove the problem.

If you have several questions that ask similar information, examine the possibility of eliminating one or more items. If you are doing statistical analysis of the answers, you might want similar questions to see if the responses are also similar (i.e., are correlated). If you are simply tallying the responses and acting on the information, try to use one question for each piece of information needed. The minimalist approach keeps the questionnaire shorter and easier to tally.

TABLE 4-6 Guidelines for Questionnaire Development

1. Determine what facts are desired and which people are best qualified to provide them. 
2. For each fact, select either an open-ended or close-ended question. Write several questions and choose the one or two that most clearly ask for the information. 
3. Group questions by topic area, type of question, or some context-specific criteria. 
4. Examine the questionnaire for problems: 

5. Fix any problems identified above. 

6. Test the questionnaire on a small group of people (e.g., 5-10). Ask for both comments on the questions and answers to the questions. 

7. Analyze the comments and fix wording ambiguities, biases, word problems, etc. as identified by the comments. 

8. Analyze the responses to ensure that they are the type desired. 

9. If the information is different than you expected, the questions might not be direct enough and need rewording. If you don't get useful information that you don't already know, reexamine the need for the questionnaire. 

10. Make final edits, print in easy-to-read type. Prepare a cover letter. 

11. Distribute the questionnaire, addressing the cover letter to the person by name. Include specific instructions about returning the questionnaire. Provide a self-addressed, stamped envelope if mailing is needed. 

Pretest the questionnaire on a small group of representative respondents. Ask them to give you feedback on all of the items that they don't understand, that they think are ambiguous, badly worded, or have responses that do not fit the item. Also ask them to complete the questionnaire. The answers of this group should highlight any unexpected responses that, whether the group identified a problem or not, mean that the question was not interpreted as intended. If the pretest responses do not provide you with new information needed to develop the project, the questionnaire might not be needed or might not ask the right questions. Reexamine the need for a questionnaire and revise it as needed. Finally, change the questionnaire based on the feedback from the test group. The pretest and revision activities increase the validity of the questionnaire.

Provide a cover letter for the questionnaire that briefly describes the purpose and type of information sought. Give the respondent a deadline for completing the questionnaire that is not too distant. For instance, three days is better than two weeks. The more distant the due date, the less likely the questionnaire will be completed. Include information about respondent confidentiality and voluntary questionnaire completion, if they are appropriate. Ideally, the questionnaire is anonymous and voluntary. To the extent possible, address the letter to the individual respondent. 

Give the respondent directions about returning the completed questionnaire. If mailing is required, provide a stamped, self-addressed envelope. If interoffice mail is used, provide your mail stop address. If you will pick up responses, tell the person where and when to have the questionnaire ready for pickup.

Document Review

New applications rarely spring from nothing. There is almost always a current way of doing work that is guided by policies, procedures, or application systems. Study of the documentation used to teach new employees, to guide daily work, or to use an application can provide valuable insight into what work is done.

Software Review