by Thomas Chianca, Ph. D.
Anytime we decide how good, valuable or important something is, we are doing evaluation. For instance, to buy a bicycle we take into account many factors before deciding which one will best fit our needs. Promised performance, cost, reliability of brand, size, strength, comfort and attractiveness are some examples. To make up our minds, we usually rely on our own knowledge or experience, personal preferences or the opinion of family or friends. This is an example of what we call informal or everyday evaluations. The decisions resulting from such evaluations will only affect us or, at the most, our families.
However, when what is at stake is something of public interest and the results of the evaluation may lead to important decisions that affect many people, then we should not rely only on intuition or personal preferences. In those cases we want to hire a professional evaluator with formal training on how to design, implement, report on and help people understand and use an evaluation. This is what I do for a living and the area in which I have obtained a doctorate from Western Michigan University.
I met Rienzzie Kern, Heifer’s senior director of Planning, Monitoring and Evaluation, during the 2004 conference of the American Evaluation Association in Atlanta. He was looking for an organization to conduct external evaluations of Heifer’s work in different countries. I was happy to hear that such an important organization was seriously committed to assessing its work and engaging in a learning process to keep improving, a sound practice not yet widely adopted by most international nongovernmental organizations.
Heifer and Western Michigan University soon formed a fruitful partnership. Since 2005, the university has conducted external evaluations of Heifer’s work in 23 countries in Africa, the Americas, Asia and the South Pacific, and Eastern Europe. Those evaluations have involved individual or group interviews with more than 5,000 people from 193 projects Heifer supports. I participated in the evaluation teams sent to 10 of those countries and have also provided management support for all rounds of external evaluations implemented so far.
Overall, the findings have been very positive. Heifer has consistently made important contributions to reducing poverty and improving living conditions in all countries evaluated. Greater impact has been noticed in projects that have followed Heifer’s core development model that includes extensive values-based training on the 12 Cornerstones—with special emphasis on livestock care, gender equity, environmental management and community empowerment—before the provision of high-quality livestock and the implementation of the Passing on the Gift system. Whenever Heifer and its local partners were able to provide consistent training and succeed in creating community spirit, the changes were substantial. There are projects that have established more than 10 generations of Passing on the Gift families. Creation of values-based groups seems to be a key factor in meeting key basic needs of participants such as nutrition, income, education and health.
In June and July 2010, I led an external evaluation of Heifer work in Zambia. This was my first time in Zambia, and I was excited to meet the people and assess the results of Heifer’s efforts. During three weeks traveling on rough dirt roads and always warmly welcomed, I visited nine project sites throughout the country in five provinces: Lusaka, Central, Eastern, Southern and Copperbelt. I talked with 449 people, including project participants, non-participants, community leaders and representatives from local nongovernmental organizations, the government and Heifer Zambia. My overall perception of the Heifer Zambia work was very positive.
I collected data on 103 key indicators related to impacts of Heifer’s work. Those indicators are connected to six main values:
- basic needs (food, water, housing, income and assets, health)
- livestock care and management
- environmental care and management
- education and training
- empowerment of families and communities
- impact on the larger community
For each project, we provide a score for each of the 103 indicators using a simple scale of six points: excellent (5 points), good (4 points), adequate (3 points), somewhat unsatisfactory (2 points), seriously inadequate (1 point), and none or unacceptable (0 points). The graph above captures the situation in Zambia.
The scores are entered in a spreadsheet that generates graphs representing the baseline situation before Heifer started its work (shown in tan), the positive changes attributed to Heifer (shown in orange), and the remaining needs (blue) that still need to be met in order to reach a satisfactory situation.
All project sites I visited had a baseline below 30 percent of total needs (summary of all six value groups) met. This indicates that Heifer Zambia is correctly and effectively targeting needy communities and families. When Heifer Zambia started working with those poor communities, the need was very large. It is no surprise that even after making important contributions, significant needs still remain.
I was able to document substantial improvements in nutrition; income and assets; knowledge on basic care and management of animals; children’s access to education; gender equity; community spirit; and adoption of Heifer values by other agencies at the regional and national levels.
Almost all Heifer project participants reported important changes in the amount and quality of food their families were able to access. The increased access to food is connected to increased crop production (including vegetable gardens), milk consumption (goat or cow), and access to food from the local markets due to greater income from project-related activities (e.g., selling milk, honey, vegetables and goat meat). There were numerous reports of families that no longer go hungry during the pre-harvest season or that now are able to provide enough food for their children. The increased production of crops, especially maize, was reported to be due to (i) use of composting or raw manure mostly from the cows (since most goats are kept free-range), (ii) use of chemical fertilizers (urea and compound D – mostly subsidized by the government [see note on environmental considerations below]), and (iii) expansion of field crops due to greater plowing capabilities in draft cow projects. Access to protein, especially meat, has improved but still remains limited. Participants indicated they eat meat (mostly chicken and small river fish) about two times per month, whereas before the project it was once every other month. It’s an important improvement but still far from the ideal.
Increase in income was substantial for most of the project participants. The main sources of income directly connected with Heifer Zambia’s efforts came from selling products generated by project activities. Families now can sell milk (cow and goat), honey, goat meat and vegetables. Milk, honey and meat are the direct results of Heifer Zambia’s animal gifts. Families also reported substantial improvements in vegetable production due to the use of compost and raw manure in their vegetable gardens using techniques learned from Heifer work-shops. In fact, a number of interviewees reported they have started vegetable production and distribution businesses. Families are also investing part of their income from Heifer Zambia projects to purchase herbicides (especially for tomatoes), which has increased their yield. Unfortunately, participants demonstrated limited concern about the environmental and possible health con- sequences associated with the long-term use of chemical herbicides.
For almost all interviewees, the money from their participation in the Heifer Zambia projects is the main source of their family income. In many cases, before Heifer Zambia, those families had marginal, if any, income. Often they would exchange work in other people’s fields for food or a small payment. Participants reported that the money they’ve made as the result of the project has enabled them to afford more food and improve their crops. In addition, the funds have been used to pay for school fees, uniforms and books; purchase bicycles for transportation; buy blankets, mattresses, sofas and other items for the home; make home improvements; and purchase more livestock.
Heifer Zambia has produced impressive results in the area of livestock care and management. Most participants did not have any animals before joining a Heifer project; this has made a major difference in the participants’ lives. With few exceptions, the animals observed by the evaluator were in good condition. Participants reported to have gained considerable knowledge from the Heifer Zambia training on how to manage and care for their animals. Heifer Zambia’s training of community animal health workers in all project sites was highly appreciated by participants and non-participants alike.
The area of environmental care and management presents the greatest challenge for Heifer Zambia. The only major positive impact noticed was the increased use of organic fertilizer (raw manure and compost) by project participants as a result of having more and larger livestock from Heifer Zambia. This impact, however, was limited by the practice of free grazing in most goat projects that has made it difficult for participants to collect the manure. The high dependency on firewood and charcoal for cooking and the extensive use of chemical fertilizers and pesticides subsidized by the government are major challenges Heifer Zambia needs to face in order to produce better results in this area.
Participants in all sites I visited mentioned increased ability to send and keep their children in school as one of the most common project benefits. Parents reported being able to cover direct school expenses, including annual fees, uniforms, books, transportation, food and board. A large number of participants also indicated that the income from Heifer Zambia project activities made it possible for them to keep their children (both boys and girls) in school longer. In at least two sites, project participants mobilized funds and manpower to make improvements to local schools.
Participants reported they learned a great deal from the training provided by Heifer Zambia, especially in the area of animal care and management. Several non-participants also indicated that they were able to participate in the trainings and benefit from them. The training of community animal health workers was also a high point for Heifer Zambia. Alumni were unanimous in commending the quality of the training sessions. Also, project participants and many non-participants indicated that they rely mostly on the support of those members to resolve any health situations with their livestock. However, several community animal health workers requested additional training from Heifer Zambia. They want to learn more about how to deal with complex health conditions because access to professional vet support is very limited. Empowerment of Family and Community was another area in which Heifer Zambia’s work shines. The community groups seemed to be consistent in ensuring diversity in leadership and membership, as well as in adopting democratic principles to rule their internal procedures.
Heifer Zambia has played an important role in promoting and ensuring gender balance at both the community leadership and family levels. In almost all projects, leaders indicated that Heifer Zambia influenced the way they structured their leadership group to ensure an equal balance in the participation of men and women. There were several reports of changes in gender equity at the family level, including division of household chores and family decisions. Fostering the participation of both husband and wife in the trainings was a key factor in conveying the message about the importance of gender equity.?
At all project sites there were many reports of how the Heifer Zambia projects bring community members closer together. Furthermore, in all project sites, participants shared with the evaluator their perception of increased happiness and self-esteem (feeling prouder of themselves and more respected by other community members).
The evaluation also revealed that the work from Heifer Zambia is positively affecting the lives of people beyond the families that benefit directly. Many non-participants have access to trainings on livestock care and management, and also to fresh milk, vegetables, protein, honey and draft cow plowing power. A number of local organizations have reported adopting some of the values connected with the Heifer Zambia model, especially Passing on the Gift.
Besides maintaining current efforts that are producing good results, my main recommendations for Heifer Zambia staff include the following:
- Make environmental care an essential requirement in all Heifer Zambia projects.
- Develop project participants’ capacity to manage solid waste.
- Develop strategies to support downstream Passing on the Gift generations.
- Improve preparation of members for changes that come with receiving large livestock.
- Support the building of improved kitchens.
- Provide training and support for participants to start collective businesses.
- Extend the idea of livestock insurance funds to all projects.
- Provide advanced training to community animal health workers.
- Foster local production of animal feed. Improve animal containment practices.
The big question I keep asking myself in any evaluation I do is what difference can my work make to the people involved or affected by the program I am evaluating. I like to think that a thorough and independent assessment of its work and thoughtful recommendations will provide Heifer with support to improve the good work it is doing.
I also hope that the evaluation will help you and other potential donors realize how seriously Heifer staff and leadership take the work they do and how eager they are to learn from their experiences so that they can invest wisely and achieve excellence.