California Department of Resources Recycling and Recovery (CalRecycle) 

Beverage Container Recycling Grant Programs

Project Evaluation Tips

Terms

  • Performance measures are a tool to measure work performed and results achieved. It describes (a) what is to be measured, and (b) the methods of measurement.
  • Inputs are resources needed to complete program activities. Inputs include labor, materials, equipment and supplies.
  • Outputs are the direct products of program activities. Outputs include the number of people receiving a service or product, the number of services or products delivered. For example, an output may be the volume of containers collected. Outputs may be used to measure outcomes.
  • Outcomes are benefits or changes that result from the program activity. They are influenced by program outputs. Outcomes may be related to behavior, skills, knowledge, attitudes, values, condition, status or other attributes.
  • Outcome indicators are specific items of data that are tracked to measure how well a program is achieving an outcome.
  • Outcome targets or goals are objectives for a program's level of achievement.

Why does CalRecycle want you to measure performance?

  • It's a quantitative assessment of performance, quality or cost used to guide the decision-making process, guide staff, and define objectives.
  • To calculate a return on investment (i.e., the grant funds).
  • To demonstrate that funds provided through the city/county payment program made a difference and contributed toward an increase in the beverage container recycling rate.
  • To analyze findings and help you improve your program.

Step-by-step approach to developing a system for measuring program outcomes and using the results:

  1. Understand the difference between inputs, activities, outputs and outcomes.
  2. Develop a timeline and assign someone responsibility for the planning and implementation of outcome measurement.
  3. Choose the outcomes you want to measure. Set a reasonable target or goal. Define a set of outcomes that track the benefits participants experience as a result of the program. Some programs will have more than one outcome. Consider the following when setting a target: outcomes for the previous reporting period, positive outcomes achieved by similar programs, resources available, external influences (e.g., economy fluctuations, seasonal fluctuations, etc.) Your target should be achievable but challenging and lead to increased results from the previous year. For example, placing collection bins in convenient locations at parks will increase the volume of beverage containers collected over 12 months by 50 percent and decrease the amount of beverage containers in the trash and hauled from parks to landfills.
  4. Specify the outcome indicators you need to collect and measure. They should be observable and measurable. Using the example above, indicators may be the number of people that place empty containers into collection bins or the volume of containers collected.
  5. Establish a baseline. A baseline is the beginning point of outcome indicators to be used for evaluation purposes. A program can use its own data as a baseline against which to compare future performance. It can also use data from another program. It may be the volume collected, knowledge or skill level, current attitude, behavior, values, condition, status or other attributes.
  6. Prepare to collect data on your indicators. In this step, you should determine the data collection method and determine where to get the data you need. Make sure your data collection method is consistent by training data collectors, if you are not collecting it yourself. (You will need to identify the data you are measuring on page 2 of your funding request form and explain the method you used to collect the data.) Another suggestion is to develop written procedures to document your work. This may come in handy if a staff change should occur. If you are tracking volume and your program is certified, you must use the DR-6s (shipping report form) as your source of data to be consistent with the volume reported to the Department by your recycler/processor. Keep the following things in mind when determining your measurement system. A one-size-fits-all measurement system is not realistic. For instance, you can use:
    • Mechanical tests and measurements, such as scales.
    • Surveys
    • Trained observers who rate behavior or environments
    • Staff who interact with program participants or event organizers
    • DR-6s (the Department of Conservation shipping report form used by certified or registered program participants (e.g. a certified recycling center), recycling center receipts, call tracking reports, etc.
  7. Develop a database for your data.
  8. Try out your outcome measurement system. You should conduct a trial run or "pilot test" that includes collecting the data, analyzing and reporting it. This will help you identify overlooked outcomes, inadequately defined indicators, inconsistencies in reporting, cumbersome procedures and analyzing and reporting dilemmas.
  9. Analyze your data and report your findings. Compare actual outcomes with targets identified at step 6. This is the information you will use to describe your outcomes on your funding request form on page 2. For example, compare your baseline data with the data you periodically collect and calculate the percentage change (increase or decrease).
  10. Monitor and evaluate your system periodically and make continuous improvements. Build in a formal review of your program. Your program will be affected by changes in the environment. Your outcome measurement system needs to keep pace with these changes. Look for ways your processes have shown improvement, where it is slipping and what continues to be troublesome. Consider changes that occurred in the program and how to adjust your system to accommodate these changes. Solicit staff input and other participants for feedback.
  11. Use your findings to increase the effectiveness of your program. If you have reached your goals, establish new goals for ongoing progress. If you have not reached your goals or outcomes it may be time for CHANGE. Identify factors that influence your program's ability to reach it's goals such as:
    • Why is the existing program not working?
    • What are reasons for poor participation?
    • Are other key players affecting the outcome?
    • Are there partnerships that could be formed to help you meet your goals?
    • Is timing/season a factor?
    • Was the frequency of service appropriate?
    • Was the target audience you selected appropriate?
    • Was the environment supportive and appropriate for your program?
    • Was there enough awareness about the program?
    • Was it convenient?
    • Were there barriers to prevent you from collecting accurate data?

Change might involve creating new and innovative ways to carry out your program; such as, a new way to increase collection of beverage containers for recycling. Changes need to happen in order for your program to progress and make you successful in meeting your program goals.

Once you've determined the changes you need to make, develop and implement a strategy and timeline that will move you to where you want to be.

Not meeting your goal does not necessarily mean change. Perhaps you underestimated the length of time involved to effectively administer your program.

Check out some examples of goals, ways to evaluate your project and measurable outcomes for various beverage container activities.

Last updated: January 1, 2010
Beverage Container Recycling Grants, http://www.calrecycle.ca.gov/BevContainer/Grants/
Contacts: http://www.calrecycle.ca.gov/BevContainer/Contacts.htm