AGRA

Since I started working for a non-profit organization, one of the most frequently asked questions I get is: “so, if you’re giving out free money, how do you measure your success and failure?”

This, of course, is a fair and relevant question.

Most foundations and NGOs operate as “grant-makers.” A grant is defined as a transfer of cash, goods or services for which no payback is required. In simple terms, that means the money is effectively free, well, sort of. The expected payout is not a “financial return,” but rather a social impact, such as improved health, better nutrition, increased agriculture yield and so on.

However, but if the money doesn’t have to be repaid, then how do foundations and NGOs make sure that those funds are not being misappropriated or wasted? How do they measure whether a grant is utilized properly and effectively to achieve the envisioned impact?

Or, in developmental speak, how do we carry out “Monitor and Evaluation (M&E)?”

M&E is a sophisticated subject and development organizations usually have specialized M&E teams. Monitor and evaluation can be implemented at an institutional level, for example, to examine whether the overall strategy is effective.  It can also be applied to a specific project where detailed where the goals and the process are determined in advance.

General M&E Challenges in the Developmental Sector

Needless to say, M&E is important for every developmental institution. Nevertheless, it’s extremely complex and sometimes even controversial. Some of the more common M&E challenges include:

• Insufficient funding: A comprehensive M&E study throughout a 3 year project requires a baseline survey, intermediary reporting and regular field visits, workshops, and stakeholder meetings, and, of course, a final report. As a rule of thumb, those M&E costs should be somewhere between 5% to 10% of the total project cost, but even that amount is sometimes seen as an expensive luxury.
• Inadequate planning: A good result framework, or project objective, starts with a well-knit “Theory of Change”(TOC). Theory of Change is a methodology that defines long-term goals and then maps backward to identify necessary preconditions and assumptions. However, not every project starts with a clear Theory of Change and the definition of performance indicators and process milestones. This is where that old saying “well begun is half done” really comes to mind.
• Causation or causality? Another key dispute when evaluating a project’s success is the difference between attribution and contribution. Attribution, or causation, is the idea that a change is solely due to your intervention, whereas contribution, or causality, is the notion that your influence is really just one of many factors that contribute to a change. It is almost impossible to attribute any outcome achievement in development to a single intervention, project or institution. For example, crop yield can be improved by using good seed varieties accompanied by appropriate field management, fertilization, and irrigation. However, natural conditions such as sunshine, temperature, and pests also affect the agricultural yield. Therefore, it is difficult to attribute “crop yield increase” to a single intervention such as improving accessibility of seeds to farmers.

While there are many general challenges associated with M&E in the development work, I have learned that M&E requires extra attention and efforts in the China-Africa development cooperation realm. The challenges that I have encountered in the project are on a micro-level, but these micro level M&E issues shed light on the reasons for skepticism, myths, and accusation on China-Africa Development Cooperation on a macro level.

M&E Requires Extra Attention and Effort in China-Africa Development Space

Earlier, I wrote about a China-Africa agricultural partnership in Mozambique that initially encountered quite a few difficulties that were eventually resolved. But among the challenges on this project was the difficulty in measuring success and evaluating the project’s overall impact.

There were two obstacles, in particular, that stood out in that project and are considering within the context of better M&E best practices, especially for Chinese stakeholders who are relatively new to these kinds processes in Africa:

Activity-oriented v.s. Impact-oriented:

The “Theory of Change” framework starts with the headline goals and then cascades the goals into objectives, outcomes, outputs, and activities. These higher-level goals define the overall impact intended for the project, for example, smallholder farmers’ livelihood improves; the mal-nutrition rate decreases; national food security improves. The lower level outputs are more tangible, measurable and actionable.

For example, if the goal of the project is to “improve smallholder farmer’s livelihood,” this is what would need to be done:

• The outcome would be to increase rice yield from 1 ton per hectare to 6 tons per hectare.
• The output would increase the adoption of the right type of fertilizer.
• The activity would be to conduct 4 farmer trainings to increase awareness per year.

And here’s what the cascade would look like:

Admittedly, as we discussed the difference between attribution and contribution (causation or causality), it’s impossible to attribute the high level impact to any single activity. So that’s why it’s important to have certain benchmarks throughout the course of the project, or activity-level goals, that serve as a form of oversight to make sure that funds are not misappropriated in any way.

And this is one area where international project management standards differ from those employed by Chinese development organizations.

The majority of Chinese development projects seem to only report on activities. If you read a brochure of seemingly any China-Africa project, it normally reads something like this:

“Organized most than 100 high-level agricultural demonstration and learning activities.” “dispatched 450 experts to African countries.”

“successfully experimentally raised 1000 broiler chickens.”

What’s lacking here is any reference as to what was actually accomplished that impact-level evaluation. The activity results are actually the easy part, so long as the project has enough funding and capacity.

But the bigger question, that often goes unanswered, is whether the farmers really learned how to use the methods and technologies that were demonstrated as part of the Chinese partnership?

Furthermore, even if they did learn about what to do and would like to adopt the new methods and technologies, would these subsistence farmers in a place like Mozambique even be able to buy the tools, fertilizers and seeds, and, if so, could they afford it?

By only focusing on the results at the activity level, the project runs the risk of overlooking eco-system factors and sustainability. So, whether these activities then translate into any kind of meaningful impact then remains an open question.

Lack of Private Sector Awareness

One of the biggest challenges in a Public-Private-Partnership (PPP), is that private sector players often lack awareness about the importance of M&E. It is not to say that a Chinese company wouldn’t have their own KPIs. The problem is that man of the Chinese companies KPIs focus primarily on revenue and cost, information that will be used in their income statement and balance sheet. But development projects don’t usually align with a standard profit and loss model. An entirely different set of data is needed instead, such as a farmer’s database or a training attendance sheet in order to measure whether a project meets its objectives.

For example, when I visited a Chinese company in the rice production and processing business, I asked them: “do you know whether the smallholder farmers that work with you have improved their livelihood?” One of the members of the staff proudly replied: “Totally! For example, there is one old lady. She lived in a small hut with thatching roof four years ago. Now she’s already building her first brick and mortar room!” Impressed by the story, I said: “That sounds fantastic! Do you have any photos or stories documented? Even better, do you have an overall record for income increase for all the 500 farmers who work with you?” He seems surprised: “Of course we don’t have that! I know it because the changes are happening before my own eyes!”

For a private company, it might seem unnecessary to note the age, gender, land area and income increase of each farmer but in the development sector, this kind of data collection is absolutely essential. Therefore we learned an important lesson about establishing M&E expectations and guidelines at the very early stages of any Chinese public-private partnership in Africa and also to set aside sufficient budget so as to train all of the relevant private sector stakeholders in how to properly conduct monitoring and evaluation throughout the life of the project.  

Telling the China-Africa Story Better Starts With M&E

Good M&E goes far beyond just implementing effective project management. In fact, there’s actually a lot at stake. China’s aid and development agenda on the African continent is under careful scrutiny and there is often a lack of objective and nuanced analysis. In order to paint a more accurate picture, we need evidenced-based knowledge generation and learning, and we need to start with better M&E. 

For many traditional donor countries such as OECD countries, development cooperation has existed for decades and evolved into an industry with a set of internationally recognized principles. In recent years, China has emerged as a “non-traditional” development partner with its own philosophies and methodologies. For example, the Eight Guiding Principle of China’s economic aid and technical assistance that was put forward in the early 1960s, are different and unfamiliar to the western donors. China also uses a different definition rather than official development assistance (ODA) which has been used by traditional donors for decades. Naturally, these differences result in skepticism and mistrust on China’s development intention and results. In order to build more trust and alignment in the multilateral system, China-Africa projects need to be self-critical and implement proper M&E to generate evidence in a transparent way.  

China also needs to systematically learn from past development efforts, to double down on our strengths and avoid known mistakes. Just two years ago, in April 2018, China formally established its first independent foreign aid agency: the China International Development Cooperation Agency (CIDCA). Compared to the long history of USAID (founded in 1961), DFID (founded in 1997), Japan’s International Cooperation Agency (founded in 1974), China still has a long way to go to build its institutional and technical capacity. Nevertheless, there are already plenty of lessons to be synthesized and learned. Since as early as 1950s, China has been sending out medical teams and providing product donations such as drugs, equipment and vaccines to most African countries. Chinese sponsored Agricultural Technology Demonstration Centers (ATDC) are located across the continent, the earliest ones built almost 12 years ago. There have been a lot of notable achievements, pockets of successes and also valuable lessons learned from failures that need to be documented and studied. A self-critical and objective evaluation framework is in urgent demand at this infant stage of building capacity for China International Development Cooperation Agency (CIDCA).

Lu Xinqing is a Kenya-based Associate Program Officer at AGRA who focuses on developing and implementing strategic partnerships with China and strengthening African countries’ capacity to effectively leverage Chinese aid and investment to support agricultural transformation.

Originally published