Johann Mouton is Director of the Centre for Research on Science and Technology at the University of Stellenbosch in South Africa. A professor of sociology, Mouton has advanced the field of evaluation in Africa and, with Corporation support, is building a graduate program that will produce experienced evaluators for the continent. Mouton has evaluated many national programs in South Africa and worked internationally on research, ethics and academic committees. Since 2004, Mouton has reviewed and assessed eight African university grantees for the Corporation to measure the impact and inform decisions related to Corporation grantmaking. Susan King, Vice President, External Affairs interviewed Mouton in the fall of 2007 about what evaluations can bring to foundation work.

Susan King: There has been so much talk about evaluation. In the context of grantmaking, what is the power of evaluation? Understanding whether someone is doing what they said they were doing? Or is it understanding the “so what,” the impact of a grant?

Johann Mouton: Well, of course it’s both, and it’s also more, because the reason there is an increase in interest in evaluation is because various forces drive it, including societal forces, which are sometimes dependent on social and political events in a country. If we can take an example from South Africa, when countries go through major social and political changes this usually also involves regime change, and the new administration comes up with major new policies, new programs and interventions. After the initial three or four years that new programs have been in place, someone says, How are we doing? And then, of course, the moment that question gets asked, questions about accountability and lessons learned also arise.

In some sense, all of these kinds of changes demand that we know not only what we are doing but also what we are achieving because one of the huge shifts in evaluation philosophy over the last three or four years has been that we’ve moved away from just accountability and impact issues to learning issues. In other words, if we do not learn from the evaluations—and use what we’ve learned—we’re just getting half the picture.

The whole domain of evaluation has become multifaceted, because people realize that it’s more than just looking at success and failure. In my view, if all evaluators do is analyze impact, that’s probably not worth the effort at all.

SK: So learning from and through evaluations is critical?

JM: Absolutely.

SK: Let’s say I’m a venture capitalist. I’ve been extremely successful, I’ve made billions, and I am impatient: now, I want real change, I want to fix something. I want malaria to be ended, I want schools to work—I want something big, and to make something substantial happen, I’m giving a series of grants. As an evaluator, what do you say to that venture capitalist? Can change happen in, for example, a three-year cycle with a number of grants? What can the evaluator share with someone who wants to spend millions or billions to promote change?

JM: That’s a very good question. It means we have to go back one step and think a little bit about what we mean about impact in the social domain, in society. Although I wouldn’t say it’s a mistake, there is an assumption people sometimes make when they think in terms of social impact or transformation, or culture change, or whatever, and they rely on analogies from other fields where, very often, impact is immediately visible and they assume that you’re working with the same phenomenon. For instance, if someone is cold and I give them clothes, there’s an immediate impact. If someone is sick and I give them antibiotics that work, or I give them a polio vaccine and they don’t get polio, you can see that impact quite quickly and visibly and you can measure it.

On the other hand, issues such as religious tolerance, political tolerance, multi-nationalism, multiculturalism, transformation, learning—these are tough issues, and they are tough for various reasons. For instance, if your goal is to help make a school function much better than it used to, or change an institution to become more gender sensitive, you cannot just focus on one aspect of that institution because the nature of the phenomenon and the conditions surrounding it are multifaceted. Therefore, it’s the complexity of social interventions that we have to understand first of all. Not to evade the question, but you cannot have a simplistic view about social interventions in education or in almost any other area.

SK: Venture capitalists are rarely patient. Does that mean big issues are not worth the risk?

JM: Some benefits and impacts emerge over time. To the venture capitalist looking for quick impact, I would say that what you want to do is support those interventions that are most likely to yield the success you’re looking for on the timetable you’re comfortable with. To put that another way: no one can guarantee—and if they do, they are being misleading—that if you support this or that reform it will absolutely produce the effects you would like. However, if a project is carried out properly—and “properly” means well resourced, well designed, and well implemented—we have enough evidence from other cases to improve the chances of achieving the desired results. And that’s pretty much the best that anyone can guarantee. So you have to make a decision: are you willing to take that risk? Because there is always some risk in any kind of big investment decision. A venture capitalist should know that. But we can reduce risk if we do certain things correctly.

SK: At Carnegie Corporation, we turn to you for evaluation of some African higher education grants because we want to know if we should continue to fund something or not. You help us feel more comfortable about making that decision. How do you feel about us relying on you in that way?

JM: I think one has to be comfortable in that role, which I am. Evaluation has become a professional activity. We don’t have enough professional evaluators in developing countries, but if you are an evaluator who does things according to established criteria, you know what is good practice. You know what it means to produce evidence-based reports. You know what it is to do rigorous data collection. Of course, you still might make mistakes; no one would ever deny that even experienced evaluators sometimes make poor judgments.

Because this is high-stakes research it’s different from, say, just carrying out another public opinion poll or doing another case study at a business school. There are consequences inherent in this type of work. If you carry out an evaluation study, and you recommend to Carnegie Corporation or any other institution that they should not continue funding at a particular university, for example, your recommendation has consequences. So of course it places a huge moral and professional responsibility on the evaluator. And that means you are—or at least should be—more sensitive and more careful about what you write, how you write it, and what claims you make. But I would rather that entities such as foundations come to people like myself who are sensitive to these things rather than make judgments either in terms of anecdotal evidence or just purely on the basis of internal reports or other forms of nonprofessional research.

SK: Do you think it makes a difference that you’re an African making evaluations about our work in Africa rather than being an evaluator from somewhere else?

JM: It certainly makes a difference in evaluating the African universities, although one cannot generalize: South Africa, Uganda, Tanzania and Nigeria are each very different, and even within those countries there are huge political, religious and cultural differences, though there are also a lot of similarities. And there is also a certain sense of understanding what the key issues are. In addition, institutions in these countries, especially universities, share many similar challenges. In my own case, I’ve been in higher education for thirty years. Over all that time, you learn a little bit about how universities work, so it’s not just about being an African, it’s also being involved in higher education. After all, you do need to know the area that you’re evaluating, so it’s familiarity with the social, geographical and political cultural context and the professional setting that gives you that added insight, I would hope.

SK: I’ve heard people say that in the context of Carnegie Corporation supporting higher education, twenty years ago we threw millions of dollars into African universities; it went into a black hole then and now “you’re doing it again.”

JM: It’s not just Carnegie Corporation; other organizations have been in a similar situation. But I think there are two responses. One is that twenty years ago, the whole situation relating to higher education in Africa was different—of course, if you throw money into a situation that is not receptive to that investment, you shouldn’t really expect success. We all know that in the mid- and late-1980s, even up to the early 1990s, higher education was the stepchild of education in Africa. That had to do with lack of government support for those institutions, and with the World Bank at that point taking the position that all support should go into primary schooling. They have since reversed that view, to their credit. So, those decades were not receptive times for social and education interventions, which, like any other reforms, require receptive conditions to even have a chance of success. You could even turn the question around and ask, why would you have expected success if you didn’t do your homework? The second response is that I’m not so sure that people are doing exactly the same thing now that they did twenty years ago. Most grantmaking institutions do learn from the success or failure of previous investments and they learn from other grantmaking institutions.

The Corporation’s work on higher education in Africa is now targeted, it’s better designed, it’s more focused, and there is more emphasis on monitoring and evaluation than there was twenty years ago. My sense is that, in the past, most grantmaking organizations did not use monitoring and evaluation as tools to help them reduce their risk.

Of course there will still be some money that is wasted, or not used as successfully as one would hope, but my sense, from what we have learned from our evaluations over the last three or four years is that there are clear successes—not, perhaps, in all cases to the degree that you expect, but successes nonetheless. Both the setting of the work and the approach to it have changed, and this new paradigm is showing results.

SK: We made a strategic decision at the Corporation to work with African universities for ten years. In America, ten years is a long time to be devoted to a project. As an evaluator, do you think ten years is a long time?

JM: No, absolutely not. These are possibly life-changing interventions: why would you expect them to be effective in a short period of time? The interventions and programs that the Corporation supports in terms of higher education in Africa are aimed at institutions, schools, universities—lifelong institutions. Changing them for the better will, of necessity, take time, as will worthwhile changes as opposed to those that might be just superficial or cosmetic.

I understand why people have a short-term horizon, and it’s because we sometimes bring too much of a business-type logic to bear. Businesses want to make profits, sooner rather than later, but you cannot think in terms of profits and losses with interventions—you have to think in terms of gains and benefits, which are short term, medium term and long term. In that context, with proper implementation, results will come. I don’t believe you can be a victim of a short-termist kind of approach when you are a grantmaker at the scale, for example, at which the Corporation is working in Africa. Here, I think, is an apt analogy: scientists, sociologists and even the public are not taking a “short-term” line on HIV/AIDS, are we? It’s now been a very long time that we have been trying to get a vaccine but that doesn’t mean that anyone has given up. So why would you want to give up on something that you say you can’t see results from within a year or two, especially in the critical area of education? Maybe we’ll see results in four or five years, but definitely in ten years. So we have to be realistic about our expectations, and success will be achieved.

Copyright information | Masthead | Carnegie Corporation of New York web site