What makes it Difficult to Evaluate Projects Ex-post: Lessons from WWF

Jindra Cekan

The World Wildlife Fund (WWF) generously shared lessons from two ex-posts and in particular what they felt should not have been done in terms of organizational learning.

They originally wanted to see whether ”the work done is sustainable and so to provide lessons for the implementing office, donor offices and the wider WWF network about designing and implementing projects to deliver outcomes that are sustained beyond WWF’s involvement.”

They used six criteria for country/ case selection:

This was in line with Valuing Voices’ (Ex-post) Evaluability Checklists.

• Organizational considerations including interest in learning by WWF
• Methodological considerations regarding data access, quality etc.
• Choice of projects, sites, timing to ensure post-closure, contribution of results isolated

WWF Case 1

The first WWF case examined the importance of evaluating both Sustained and Emerging Impacts (those expected or those that emerged from local efforts or other entrants post-INGO-exit) and managing the ex-post evaluation process, as LWR and World Vision have done.

There were three types of interesting results (refer to Table 1).

• Methodological problems in comparability to final evaluation and impact data as well as choice of project: problems with site/ case and lack of water data comparability as well as the entry of a new NGO garnering benefits (so limited contribution/ isolation of results). WWF tried to mitigate it with good design guidance and ongoing review processes during the ex-post.
• The emergence of new data ex-post about the benefits of the local user association for conflict management and emerging new catchment management
• There were ex-post limitations intrinsic to the species and ecosystem resources on which they rely, specifically species and water. Other problems of not tracking impacts on animals, not learning from evaluations, and not having saved documents might benefit other projects’ quality.

Table 1. WWF Case 1

Some learning might be attributable to the difficulties of evaluating ecosystems or it might provide important cautionary lessons for others considering such ecosystem evaluations.

Then, there is limited value to be gained from doing an ex-post evaluation.

WWF Case 2

The excellent news is that ex-post demonstrated that all three kinds of impact indicators, namely species numbers, forest cover, and conservation area improved (note, data are confidential, thereby not cited below) and that the XX Conservation Area is still managed by the community (Table 2). Also, the team gathered information that living standards had improved, as had, impressively a country-level seed fund and active participation in managing and scientifically learning from their resource.

It also highlights that ecosystem ex-post evaluation is complex, as wildlife number was difficult to methodologically confirm, due to progression in monitoring technologies.  A valuable question that WWF shared was “how to assess [continued] benefits to people from natural resource management?”

Table 2: WWF Case 2

WWF found and shared important and appropriate lessons about what these ex-posts brought with them. Most fell into aforementioned categories of Choice of projects, sites, timing, and importantly, the value of considering Methodological considerations regarding data access, quality from project onset.

What was new to us at Valuing Voices was that Organizational considerations including interest in learning by WWF-UK and their partners which is something we have not seen before.  Equally, the importance of a solid rationale for asking a country office to participate in such an evaluation, and not imposing an ex-post on a less than enthusiastic program for whom sustainability may not be a priority.  This links these cases together, as given limited budgets, WWF and their partners could choose to prioritize design spending, perhaps above final evaluations, especially if local NGOs know donors will leave at project’s end, so funding to improve their internal capacity strengthening might be a larger priority.

Knowing the project has an immutable closeout can affect interest at the headquarters, as it can be quite difficult to find funding to evaluate ex-post, and more pressing current work competes. Also, implementing local NGOs/ CBOs could actually know about sustainability far more than donors, hereby eliminating their interest, as an M&E officer recounted. Both organizations could have other priorities, including new projects with new staff, who had less interest in the old project.  Staff turnover is an additional, important consideration in whether to do an ex-post.

Table 3: WWF Learning

Given how vital it is to evaluate the sustainability of our environment, including ecosystems and natural capital on which our participants and partners depend, WWF is going ahead on evaluating ecosystems, adding to the vital knowledge base. The Global Environmental Facility is also experimenting on remote-monitoring for the evaluation of ecosystems.

We have a whole world of sustainability to discover!

Jindra Cekan has 25 years of listening to participants in Africa, Latin America, the Balkans, Europe and the US