BUS607: Data-Driven Decision-Making

Be Data Driven

No amount of data will lead to accelerated impact if it is not used to inform decision making. When an initiative is data driven, quality information is available to the right people when they need it, and they are using those data to take action. The data produced by a digital initiative should be used for more than just outputs, such as published work or donor reporting. Examples of the types of data that can be collected to inform decision making include surveillance, research, operations, project management and data from secondary sources collected outside of the program.

Core Tenets

• Design programs so that impact can be measured continuously and incrementally, focusing on outcomes, not just outputs.
• Make use of existing data, including open data sets and data from interoperable systems.
• Use rigorous data collection methods. Consider and address potential biases and gaps in the data collected, perform data quality checks, and maintain strong documentation behind collected data.
• Close knowledge gaps by contributing data to the development community and using data and interoperability standards.
• Use quality real-time or timely data to support rapid decision making, improve programming for users and inform strategy.
• Present data in formats that are easy to interpret and act on, such as data visualizations.
• Create a data use culture by prioritizing capacity building and data use efforts across all stakeholder groups, including the groups whose data are being collected.
• Be holistic about data collection and analysis. Collect data from multiple sources, and use a mix of data collection and analysis methods. Analyze your data collaboratively with stakeholders.
• Identify and use open data and interoperability standards.
• Collect and use data responsibly according to international norms and standards.

Project Lifecycle

The following recommendations, tips and resources are drawn from the digital development community to give you options for applying this Principle during each phase of the project lifecycle. This guidance is not meant to be exhaustive, but rather suggests actions you can take to apply this Principle in your work. If you have other tips, resources or comments to add, please share them with the community at the Digital Principles Forum.

Analyze and Plan

During this phase, define your data needs and strategies for meeting these needs. Planning how you will collect, analyze and communicate data will improve outcomes, fill knowledge gaps and help others working in the field.

• Identify data needs in partnership with users and stakeholders. Users should drive the process of determining which data are needed for decision making, in what format and when collection should occur. Identifying the minimum data needed for decision making is also important as too much data can make it harder for users to distinguish what is important and make decisions quickly.
• Adopt standards to guide data use. Best practices and standards for data collection and management should be adhered to throughout the project lifecycle. For example, the United Nations published the following basic data principles in "A World That Counts: Mobilising the Data Revolution for Sustainable Development":
  • Data quality and integrity
  • Data disaggregation, ensuring that differences are understood by sex, age and other relevant characteristics
  • Data timeliness
  • Data transparency and openness
  • Data usability and curation
  • Data protection and privacy
  • Data governance
  • Data independence, ensuring collection of data by parties independent of the implementing organization
  • Data resources and capacity
  • Data (human) rights, including the right to be counted, the right to due process, the right to privacy and ownership of personal data, and a way to be removed.
• Identify available data sources that you can use. Draw from existing relevant open databases to minimize how much data your program will need to collect. These databases are license-free and include public data provided by government agencies, open source communities or other programs. Confirm that data are in a format you can work with and that they meet quality standards.
• Identify and address the risks associated with data privacy and security – especially around access to and use of personal data. Most programs will collect and manage sensitive information. Develop a plan to protect privacy, keep data secure and delete data that may no longer be necessary when the program concludes. Developing a responsible data framework can help to protect individuals from harm from data-driven activities, including discrimination from profiling or privacy risks.
• Assess data literacy and capacity for data production and use.
• Work with users and stakeholders to identify who in the ecosystem is using data and how they are using the data. Observe the role that data use plays in their daily activities.
• Identify skill gaps in data production or data use.
• Inventory the current data tools being used and identify where additional tools may be needed.
• Confirm that users can visualize data, quickly identifying what may be working or where improvements are needed. Get feedback from users on whether the data and visualizations are useful and if they have the data they need; if not, make changes to improve the visualizations.
• Develop a plan for filling data-related skill gaps, possibly through hiring or capacity-building activities.

Design and Develop

Design and develop your digital tool or platform so that it will address the data needs identified during your analysis and help build a data use culture. Throughout the design process, consider how information is being and will be used. When users see value in the data they are collecting and can use those data, it creates a positive feedback loop that leads to a more sustainable initiative.

• Build an initiative that meets data accessibility and quality needs. Design the tool or system so that it provides information when it is needed for decision making. For example, real-time data can be important for operational activities, while timely data – such as data available on a monthly basis – may be needed for strategic purposes. You can define data quality minimums to set a level of "good enough" for decision making. Answer such questions as How reliable does the data need to be to make decisions with it? and When is timeliness more important than accuracy? Ensure that key data are continuously available to the initiative team and decision makers.
• Consider data transmission and storage needs. Develop automated processes for data extraction, preparation, analysis and reporting so that information is quickly usable, meaningful and available to decision makers.
• Build data literacy and a data use culture. Work with users to integrate data use into policies, practices and procedures. Data use can also be incorporated into job descriptions, training, performance reviews and incentive structures. Set up internal mechanisms to encourage regular data reviews and reflection. Identify data use champions who lead by example and can inspire others to use data for decision making. Engage users and stakeholders so that they see value in the data that they're collecting – or the data that are being collected about them and understand how the data will be used.

Deploy and Implement

In the deploy and implement phase, work with users and stakeholders to advance a data use culture and share the data collected across the ecosystem and with the digital community. The data could include both data collected about the digital tool to inform modifications and data collected with the tool to support programming. When sharing data, continue to validate that you are safeguarding personal data through appropriate privacy and security measures.

• Implement data privacy and security procedures. While doing this, continue to monitor for vulnerabilities in data collection and sharing, especially when new data sources or technological tools are being used. Evaluate how well your privacy protections are working. Validate that individuals have confidence that their data are not being used or shared improperly or with unauthorized parties, and address any concerns expressed. For example, individuals may worry that information will be shared with family members or other community members, or that the government could intercept it.
• Continue to build a data use culture. Focus on building overall data use skills and an understanding of the importance of data use. Consider collaborating with other practitioners to hold joint data use trainings. Identify and support individuals who can become data use champions, including those at the leadership level; they can encourage a data use culture and push for organizational changes needed to support the transition.
• Contribute to the community. Continue to participate in and contribute to the digital development community. Watch for new data sets, tools, case studies or other resources that would be helpful to your initiative. Help fill gaps in public data by appropriately sharing the data that you're collecting so other community members can use and build on that data.

Cross-cutting: M&E

Throughout the project lifecycle, use data to monitor, measure and improve the impact of your initiative. Regularly consider how you can use data more effectively.

• Develop and implement a monitoring and evaluation plan based on best practices for data collection and management and inclusive of all types of data requirements. Specify how data will be collected, analyzed, distributed and used. Plan for regular follow-up data reviews to measure impact incrementally. Set milestones that are based on when data are needed to make decisions about the initiative. Identify the data required to support an assessment of data use for decision making. The World Health Organization's MAPS Toolkit includes a self-assessment that you can use to validate your evaluation plan.
• Collaboratively identify indicators and data to collect. Include measures on how the technology is being used and analytics generated by the technology itself. For example, are users always entering data on a specific day of the week? Are there trends indicating when users generate reports? Also, establish basic data quality standards and processes for review. For example, make it standard to include gender-sensitive indicators and sex-disaggregated data if women and men may be affected by the initiative differently.
• Develop a data analysis plan. Map out how the collected data will be organized, classified, compared and displayed relative to the evaluation questions, as well as how multiple data sources will be integrated. The plan should be written in conjunction with selecting data collection methods and instruments.
• Gather baseline data on your indicators, and regularly analyze data to measure impact and outcomes. Make use of existing data and data sets to assess the impact of your initiative, and also consider monitoring short-term outcomes. Review the data with users, stakeholders and the digital development community, as well as with funders, for additional insight. Consider how to share data usefully and meaningfully with partners; ask for partners' help in interpreting the data and considering how to make improvements based on that analysis.

Source: Digital Principles, https://digitalprinciples.org/principle/be-data-driven/
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.