Your evaluation plan will help you establish data collection and analysis systems, determine who will collect and use the data, and decide how you will analyze the data to provide insights into your policy or program.
A collaborative approach involving diverse parties, including groups or individuals who are disparately affected by the issue, will build ownership in the evaluation results, increase stakeholders’ evaluation skills, and increase the likelihood evaluators will be sensitive to participants. Work with key parties to:
- Clarify what data will be collected, from whom and how best to get it.
- Make sure the data you plan to collect will help you answer your evaluation questions.
- Decide what type of information is easily obtained from other sources and what new data you will need to collect.
- Consider the data you collected in the Assess Needs & Resources step. Would collecting from these same sources help answer some of your evaluation questions?
- Consider using a combination of qualitative and quantitative methods to give your team the best picture of whether your actions are effective. Assessing Community Needs & Resources in the Community Tool Box provides different methods for collecting information.
- Develop clear procedures for gathering, analyzing and interpreting data.
- What will be reviewed and how often?
- Build in considerations for different cultural perspectives of respondents, such as language and literacy needs.
- Develop data collection forms/instruments.
- Make sure your forms and instruments measure what you want to measure and that they will provide similar answers with the same population even if administered at different times or places.
- Surveys are commonly used to collect evaluation information from participants in a program or those affected by a policy.
- Develop training and technical support for those who will collect and analyze the data.
- Pilot data collection processes so that they can be improved prior to full implementation.
Deciding what data to collect for advocacy/policy will be challenging because of the dynamic nature and environment of policy work which makes it hard to measure. Consider non-conventional methods (e.g., surveys, interviews, document review, observation, polling, focus groups, case studies) in addition to traditional data collection methods. Use A Handbook of Data Collection Tools to select specific data tools that align with the outcomes you want to measure:
- Shift in social norms (Suggested tools: interviews, focus groups, meeting observation checklist, rolling sample survey)
- Strengthened organizational capacity (Suggested tools: Advocacy Capacity Assessment, spider diagram)
- Strengthened alliances (Suggested tools: Intensity of Integration Assessment)
- Strengthened base of support (Suggested tools: Increased Engagement of Champions, Checklist for Mobilization and Advocacy)
- Improved policies (Suggested tools: Legislative Process Tracking, Monitoring Policy Implementation, Changes in Physical Environments)
Additional tools for policy/advocacy evaluation include:
- Four tools specifically geared for advocacy evaluation in Unique Methods in Advocacy Evaluation:
- Bellwether Method – structured interviews with influential people
- Policymaker Ratings – assessment of policymakers’ support for issues
- Intense Period Debriefs – discussion of actions after a policy window or intense action
- System Mapping – visual map of what is expected to change in a system and how to measure it
- If you are evaluating media strategies, you may be interested in whether the media coverage of an issue increases over time. Typically, this requires using an online database like LexisNexis©, a news tracking service that offers a searchable database at the national, state, and local level. If you want to see if your media strategy has changed how media covers an issue, then you’ll need to do a content analysis of articles which requires more time and resources.