With rigorous economic research and practical policy solutions, we focus on the issues and institutions that are critical to global development. Explore our core themes and topics to learn more about our work.
In timely and incisive analysis, our experts parse the latest development news and devise practical solutions to new and emerging challenges. Our events convene the top thinkers and doers in global development.
Legal empowerment of the poor, education, Africa, evaluating aid effectiveness
Justin Sandefur is a senior fellow at the Center for Global Development. Prior to joining CGD, he spent two years as an adviser to Tanzania's national statistics office and worked as a research officer at Oxford University's Centre for the Study of African Economies. His research focuses on a wide range of topics, including education, poverty reduction, legal reform, and democratic governance.
A commission led by the UN's special envoy for education, Gordon Brown, is calling for a doubling of global aid for education, without any clear reform agenda to raise learning levels in the world's failing school systems. That might be ok: bad schools in poor countries still seem to produce big benefits.
Last week the Center for Global Development hosted a panel with three heads of state vying to raise money for three rival education funds. Former UK prime minister Gordon Brown and former Tanzanian President Jakaya Kikwete were there to pitch a new entity they want to set up called the International Finance Facility for Education, aka IFFed, which they're billing as an analog to the widely celebrated Global Fund in health.
Awkwardly, something "like the Global Fund but for education" already exists, and its board chair was also on stage in the person of Julia Gillard, former Prime Minister of Australia, representing the Global Partnership for Education (GPE), which is also looking for money to replenish its coffers. Luckily, they seem to have agreed on a way to carve up the education landscape:
Brown's proposed fund, the IFFed, would serve lower middle-income countries, allowing them to borrow for education at concessional rates.
Gillard and GPE will continue to focus on low income countries with grant money. (Though, awkwardly again, the IFFed's list of "pioneer countries" also includes a number of low-income countries from the GPE's list, like Chad, Congo, Ethiopia, Malawi, Mozambique, and Tanzania. Why do we need a new organization again?)
And lastly, UNICEF's new "Education Cannot Wait" fund—represented on the panel last week by Tony Lake, UNICEF's Executive Director—will serve the true bottom of the pyramid, so to speak: refugees and displaced people.
Now, if only there was any money to divide. Aid budgets are tight these days. All the panelists were members of the U.N. Commission that Brown chaired on financing the U.N.'s Sustainable Development Goals for education. And the one thing all the panelists agreed on was that much more money was needed, both in aid and in domestic spending in developing countries.
Money probably won't improve school quality
Nobody on the panel had any appetite to talk about politically difficult education reforms that might turn money into learning.
Indeed, rather than focusing on the UN's new global learning goals, Gordon Brown and colleagues have shifted their focus to global spending goals. That's a very different conversation: in the unlikely event large amounts of new aid money are forthcoming in today's political climate, they're far from guaranteed to produce learning outcomes.
This is something of a departure from the text of Brown's Education Commission report, which dives deep into data and current literature on how to achieve learning for all. It bemoans the 'learning crisis' in many developing countries, reviews effective interventions to overcome the crisis, the need for more and better learning assessment, and notes that more spending often isn't the answer (see Figure 1 which shows Vietnam's high performance and low spending, and the inverse spending-performance relationship in a sample of Pakistani districts). Even as it turns to the question of finance, the report focuses on innovative mechanisms to incentivize reform and a stronger focus on learning.
That was the bait, now comes the switch. In the very last chapter the report turns to its "ask", tallying up why the world needs to raise education spending by $200 billion a year. And in the public discussion of the Commission's work, it's those spending goals—not all the careful analysis of reforming school systems—which now dominate the conversation.
Throwing money at failing school systems is welcome, but unlikely to work. A vast body of research questions the link between education spending and learning outcomes, and some of the best micro research on interventions to improve learning focuses on things that cost zero money, or even reduce education budgets.
In a review of programs to raise learning in the developing world in the journal Science, Michael Kremer, Conner Brannen, and Rachel Glennerster reach a conclusion broadly at odds with the push for money first:
[A]mong those in school, test scores are remarkably low and unresponsive to more-of-the-same inputs, such as hiring additional teachers, buying more textbooks, or providing flexible grants. In contrast, pedagogical reforms that match teaching to students’ learning levels are highly cost effective at increasing learning, as are reforms that improve accountability and incentives, such as local hiring of teachers on short-term contracts.
The things that Kremer et al say don't work are precisely the things the Education Commission is budgeting an extra $200 billion in global spending to expand. In contrast, the promising reforms that the authors describe here tend to have two things in common: first, they're often quite cheap, or even free; and second they're politically difficult.
But bad schools still seem to produce good outcomes
While the research literature shows a weak link from money to learning gains, it also shows that education in the developing world has amazingly high returns. This is paradoxical. We know that learning levels in poor countries are abysmally low. In an earlier post, I showed that in half of the fifty or so developing countries where we have data, fewer than 50 percent of women who left school after fifth grade could read a single sentence.
Sending kids to school has huge social returns, particularly for girls. Not only are the wage returns to a year of schooling generally estimated at around 10 percent per annum, but more educated women have fewer children and their children are less likely to die. This is somewhat puzzling if school isn't even teaching them to read.
In a background paper for the Education Commission, my colleagues Mari Oye, Lant Pritchett and I show the relationship between girls' schooling and outcomes like women's fertility and their children's survival is significantly higher where schools produce more learning. The correlation between a year of schooling and child survival is roughly two-thirds higher in countries with the highest versus lowest level of school quality.
Looking at the individual level data, women who completed six years of primary schooling had roughly 0.6 fewer children than women with no schooling, and those children were about 5 percent more likely to be alive. But if you focus only on women who went to school and didn't even learn to read—i.e., women seemingly failed by bad schools—they still had about 0.25 fewer children and their kids were still about 2 percent more likely to be alive.
To be clear, these are nothing more than correlations. You can think of lots of reasons more schooling might be correlated with lower child mortality absent any causal mechanism. But there's growing evidence to suggest the return to schooling is indeed causal, based on natural experiments and policy reforms in places where enrollment has expanded in systems that produce fairly dire learning outcomes.
So maybe pouring more money into business-as-usual and low quality education isn't such a bad investment after all.
Counterfactuals: if we spend more on education, what do we give up?
Fine, perhaps the world needs more bad schools. The world needs lots of things. Is bad education a better bet than, say, HIV drugs or famine relief?
The most intriguing part of Gordon Brown and co.'s pitch is the claim that they can create "new money." The plan is a bit of financial engineering to stretch aid dollars further:
Start with a couple billion dollars of good old-fashioned aid money
Add a couple billion more dollars of loan guarantees from rich countries to Brown's new IFFed fund.
Use that money as collateral to borrow on international capital markets at low rates, thanks to the backing from major donor countries.
Re-lend to poor countries so long as they promise to spend on education.
The basic idea, which is a good one, is to as rich countries unwilling to part with much cash during an age of austerity, to lend their credit rating instead. The problem is, it's not an idea unique to education advocates.
Recently, my colleagues Nancy Birdsall and Ngozi Okonjo-Iweala proposed a new, "Big Bond for Africa" in which rich countries would use their aid budgets to borrow at low rates and lend onward to African countries on concessional terms to pursue whatever development needs they want to prioritize—education or other. And my colleague Michele de Nevers and colleagues, including the former treasurer of the World Bank, Kenneth Lay, have proposed a Tropical Forest Finance Facility (TFFF) that would borrow from rich countries to create essentially a large endowment whose returns would fund forest conservation. There's no escaping the fact these are rival initiatives.
As that old bumper sticker used to say: "It'll be a great day when the schools have all the money they need, and the Air Force has to hold a bake sale to buy a new bomber." If the Trump administration is seriously weighing a new MOAB bomb versus a global fund for education, I'm all for the latter. Sadly, that's not the actual trade-off faced in foreign aid spending conversations in the era of Trump and Brexit.
When aid budgets are tight, advocating for one specific sector like education isn't a question of generosity or a moral crusade, it's essentially just a zero-sum game of earmarking a fixed budget. If that's the game we're playing, education advocates can't duck the conversation about how to generate the biggest learning gains at the lowest cost.
But they will still try. As moderator, my colleague Amanda Glassman opened last week's event with a brief statement about the need for improved efficiency and quality in global education. The panel was unimpressed. When her turn came to comment, Julia Gillard of GPE confessed, "while you were giving your speech about efficiency, Gordon leaned over to me and whispered, 'Yes, but we need more money too.'"
This blog post has been updated to include a correction. The International Commission on Financing Education is led by the UN’s special envoy for education, but it is not a UN commission, as originally stated.
The architect of India’s annual Economic Survey, Chief Economic Adviser Arvind Subramanian - a CGD senior fellow currently on leave – returns to the Center to discuss the big policy ideas making a dramatic mark on India’s economy, and what they could mean for the world.
According to The Economist, India's proposal to give every citizen a cash transfer using the digital platform Aadhar could reduce absolute poverty from 22 percent to 0.5 percent. For a country that is home to a third of the world's poor, could Universal Basic Income (UBI) fundamentally change the picture of poverty, health and well-being in India and the world?
Using new data on roads and cities spanning over 50 years in 39 African countries, Remi Jedwab and colleagues document the effect of road construction on city population growth, through the channel of increasing market access. They estimate a 30-year elasticity of city population with respect to market access that is smaller than found in other regions. But they also find heterogeneity in returns within Africa that provides important insights about the potential return to specific transport investments, depending on context.
A UBI is an expensive way to reach the poor, but a new report from India suggests that by cutting out the bureaucratic and political middlemen, it may be cheaper than the status quo.
Last weekend the New York Times Magazine published a feature story by Annie Lowrey looking at GiveDirectly's UBI experiment in Kenya. Reporting from a Kenyan village where eating in public is considered an ostentatious display of wealth, Lowrey does a good job of conveying just how radical this experiment is. While aid projects have come and gone, suddenly every man, woman, and child will start receiving a monthly transfer of about $22 per month—roughly equivalent to the median Kenyan household’s per capita income—for the next twelve years, no strings attached.
Over the past week, the NYT piece gave rise to some sensible concerns (see, e.g., Chris Blattman's skeptical take):
Can Kenya, or any poor country, really afford to provide a UBI?
Why not target the poor, instead of giving everyone money?
As fascinating as it is, the GiveDirectly pilot can elide some of these questions. After all, it's a relatively small-scale, privately-funded research experiment. Silicon Valley philanthropists can throw money at a few dozen Kenyan villages that might not be sustainable at a larger level.
That's why I think the truly exciting action on this topic is in India, where (as my colleague Todd Moss explained in an earlier post) the Ministry of Finance is seriously weighing the idea of a universal basic income that could transform the idea of the welfare state in a country of over 1.2 billion people, home to over a third of the global poor. So how does India’s Chief Economic Adviser, Arvind Subramanian (a CGD colleague, currently on leave) deal with these looming questions of affordability and targeting?
Can poor countries afford a UBI?
The short answer from Subramanian and team for India is that realistic magnitudes of cash can’t wipe out poverty, but a move to universal cash programs might actually be cheaper—per unit of poverty reduction—than current social programs.
It’s easy to oversell this point. As Berk Ozler notes over on the World Bank's Development Impact blog, the NYT Kenya piece cites a simple back-of-the-envelope calculation by Christine Zhang, Laurence Chandy, and Lorenz Noe at Brookings, who estimates that the size of the global poverty gap—i.e., how much extra money poor people would need to bring them up to the World Bank's $1.90-per-day absolute poverty line—is about half of what rich nations spend each year on foreign aid. In other words, more or less, if you gave half of the foreign aid budget directly to poor people as cash, presto: no more poverty in the world.
Figure 1. Original title: "Official foreign aid now exceeds the annual cost of closing the poverty gap"—reproduced from Zhang, Chandy, and Noe (2016)
This viral Brookings graph is a fascinating framing of the cash-transfer debate, but sadly, it's not a remotely practical policy proposal. My colleague Maya Forstater (with a helpful pointer from Chandy) calls this the wishful thinking of redistribution in the developing world. Ozler's post explains the arithmetic of global poverty and cash transfers in detail. He notes that governments and aid donors have a very imperfect idea of who is and isn't poor. So any attempt to end poverty through transfers would end up benefitting lots of non-poor people, driving up the costs. Furthermore, most real-world transfers in poor countries are a flat amount, as it’s impossible to know exactly how far below the poverty line a given household is.
The combined result is that eradicating poverty through cash transfers would be much, much more expensive than the Brookings calculation suggests.
In contrast, the India proposal from the Chief Economic Advisor Arvind Subramanian is designed not only to be universal, but to be budget neutral—meaning it would only roll out as other social programs were rolled back. The cost of universality is that the magnitude of any feasible transfer would likely be a small fraction of the $22 per month from the Kenyan experiment. They estimate a quite small transfer of roughly $4 per person per month would reduce poverty relative to India’s national line from about 22 percent down to about 7 percent, at a cost of just over 2 percent of GDP—roughly equal to the total cost of current fertilizer, fuel, and food subsidies (see Figure 10 and Table 2).
Some of those gains are illusory. Poverty is typically defined as failing to meet a certain consumption level, measured in monetary value. Food subsidies allow you to eat more even if you ‘consume’ less in monetary terms, whereas cash boosts monetary consumption directly.
As Jean Dreze notes, other recent UBI proposals for India have been considerably more generous than the Ministry of Finance’s trial balloon, with Pranab Bardhan proposing a UBI that would add up to 10 percent of GDP, and Vijay Joshi proposing an alternative at 3.5 percent of GDP.
By displacing other government programs, UBI would trigger powerful political opposition. Dreze’s piece worries about UBI being in tension with universal education or work guarantees. Those concerns are echoed (or pre-empted) by the report itself, which notes “while a UBI may certainly be the shortest path to eliminating poverty, it should not become the Trojan horse that usurps the fiscal space for a well-functioning state” (p. 189).
Why not target the poor?
The bold claim buried within Subramanian's proposal, is that a universal income would actually be more pro-poor than existing, targeted anti-poverty programs. The idea is that current “targeted” programs are so leaky and misallocated that they’re not targeted at all, and a well-run UBI would be better than the status quo.
The solution to this apparent paradox is disintermediation. By leveraging India's combination of an advanced electronic payments infrastructure and the Aadhar biometric identification program with over a 1.1 billion enlistees, UBI could allow the Indian central government to tackle poverty directly, cutting out layers of bureaucracy and political intermediaries who siphon off budgets or allocate programs to their cronies and favored constituents.
The obstacles to accurate targeting mentioned above in the World Bank research are mostly technical issues, preventing altruistic philosopher kings from achieving perfect poverty reduction. Real governments aren't so perfect. Misallocation doesn't happen just by accident. Politicians steer programs to their districts, poor or not; district officials siphon off money for pet projects or election campaigns; and local officials ask for kickbacks from beneficiaries.
The result is that targeting using politicized, inefficient, and oftentimes corrupt government systems could, in theory, be worse than not targeting at all.
Consider the actual counterfactual to UBI: the Indian government's current massive, public distribution system (PDS) of subsidized food for poor households. This would be a likely target for countervailing cuts if UBI ever materialized. Subramanian and team calculate that 36 percent of the PDS subsidy never makes it to any household, and (coincidentally) another 36 percent accrues to non-poor households. What's left is just 28 percent of the subsidy value for the poorest 40 percent of Indians. In short, the PDS which allegedly targets the poor is less well targeted than a (theoretical) universal program.
Figure 2. Targeting of India's Public Distribution System—reproduced from India Economic Survey, chapter 9, Figure A5
This is very similar to an argument made by Martin Ravallion in a CGD blog post a couple years ago. Ravallion noted that his own research with World Bank colleagues in Bihar found that "once one accounts for all the costs involved in India’s National Rural Employment Guarantee Scheme [NREGS], including the forgone earnings of participants, a [basic income guarantee] with the same budgetary cost would have greater impact on poverty than the labor earnings from the existing scheme."
Could an electronic UBI really do better? There’s some evidence it could. In a large-scale RCT in Andra Pradesh, economists Karthik Muralidharan, Paul Niehaus, and Sandip Sukhtankar show that shifting NREGS implementation to biometrically-authenticated “smartcards” led to a significant reduction in leakage from the program.
Curiously, though, technology hasn't solved the targeting problem: even with a biometric database of a billion people, the state still doesn't know who's poor. Poor people don't have bank accounts or pay stubs. For now, targeting the poor means allowing discretion by local politicians and bureaucrats, posing a trade-off between targeting and leakage.
Redistribution without (much of) the state
UBI embodies two radical departures from most existing anti-poverty programs. The one we usually talk about is anti-paternalism. Cash gives beneficiaries choices. Since technocrats often distrust the choices poor people make, economists have spent much of their research effort on cash transfers to showing that poor people won't choose to drink away their money or stop working.
But in many developing countries, the second distinguishing feature of cash transfer programs—disintermediation—may prove more radical in the end. By relying on electronic payment mechanisms and ubiquitous mobile phones, programs like GiveDirectly in Kenya can "zap" money straight from offices in Nairobi or even North America, directly to beneficiaries in Busia. In theory, the Indian central government could do the same across several hundred districts.
So yes, the new UBI pilot in Western Kenya may be unsustainably expensive. And the rhetoric about the ‘end of work’ and the age of robots is probably overwrought in places like Kenya or India. But as the Indian government has discovered, a modest version of UBI could potentially save money and shift expenditure in a progressive, pro-poor direction.
Thanks to Maryam Akmal and Dev Patel for help with this post.
The city of Bogotá set out to reduce crime and increase state legitimacy by raising state presence on city streets: either increasing police time by two thirds, or delivering clean up and lighting services. In their new paper, Christopher Blattman and his co-authors find that these large and sustained increases in state presence have relatively modest effects on crime, violence, and state legitimacy. They conclude that there may be returns to a more focused approach: increasing and concentrating efforts on the places with the greatest need and least prior state presence.
We respond to critics of our evaluation of Liberia’s “partnership” school program, distinguishing legitimate concerns about the charter-style program itself—which can be turned into testable hypotheses—from methodological limitations to what an impact evaluation can show.
The three of us are running a randomized control trial of the "Partnership Schools for Liberia" (PSL) program, which is President Ellen Johnson Sirleaf's effort to introduce something akin to American-style charter schools or the UK’s academies to Liberia's underperforming education system. Charter schools are controversial almost everywhere, and in some circles, so are randomized trials.
Recently, two advocacy groups—Action Aid and Education International—circulated a call for proposals offering researchers €30,000 to "conduct an in-depth qualitative investigation" of the partnership school program. The document explicitly seeks to dismiss the RCT results before the study has been concluded, asserting that "there are serious concerns over whether the evaluation of [the program] can be truly objective or generate any useful learning."
Of course, we object to this blanket dismissal of evidence, and below we offer a point-by-point response to the criticisms. Unfortunately, Action Aid and Education International have drawn their policy conclusions before collecting data. Their call for proposals notes that the research they fund will inform a "campaign against the increasing privatisation and commercialisation of education" and that "the report will be used for advocacy work… to challenge privatisation trends." We agree that what these organizations describe is advocacy not research, as their conclusions are openly advertised in advance.
We also agree with Action Aid and Education International on more substantive matters. The partnership school initiative may fail. And many of the criticisms raised below rest on sound factual premises. The key difference is that we do not read these points as criticisms of our study. Rather, many of the points below are concerns over the program itself, many of which we share—and we believe Liberia’s Minister of Education, George K. Werner, shares them as well, which is why he commissioned a randomized evaluation. We would reframe these concerns as hypotheses that we have explicitly designed the randomized evaluation to test. In some instances, the "criticisms" restate the core motivation for the RCT.
Here are Action Aid and Education International's criticisms, verbatim and unedited (though re-ordered to group similar comments together), with our responses.
1. Partnership schools are very expensive to run
"PSL schools are receiving significantly more funding ($50 per child enrolled) than the government control schools against which they will be compared."
"PSL schools receive significantly more political and managerial attention from the Ministry."
True! Unlike normal government primary schools, the Ministry decreed that PSL schools must be free at all grade levels, including early childhood education. Providing free services costs money, as do all the books, teacher training, and other things PSL provides. Crucially, so far all of the extra money spent on the program has been paid by philanthropic donors, not the Government of Liberia.
The fact that the program costs money is not, in and of itself, a failing of the RCT. If a clinical trial of a new drug that costs $1,000 per patient reduces the incidence of heart disease by 20 percent, we wouldn't declare the study a failure. Rather, we would evaluate the drug based on its comparative cost-effectiveness. Are there cheaper ways to achieve the same 20 percent reduction in heart disease? If so, doctors should prescribe those instead.
The same logic applies to partnership schools in Liberia. Even if the RCT shows learning gains, it remains to be asked whether the gains justify the cost of the program.
Two complications arise in our case. First, very few alternative "treatments" to increase learning in Liberia have been rigorously tested. In a perfect world, we could test alternative interventions head-to-head with partnership schools as part of our evaluation. If other aid donors who are spending heavily on Liberian education would like to subject their programs to that head-to-head comparison, we would love to include them. (Seriously, USAID, EU, and GPE: call us.)
The second complication is that measuring the costs of this program is very tricky. Philanthropists have poured millions into these schools, on the (somewhat optimistic) assumption that these costs will be amortized over many years and held constant under a hypothetical expansion of the program. In economics terminology, donors agreed to pay for long-term investments that represent fixed costs for PSL regardless of its scale, and operators claim their per pupil operating (or variable) costs are quite low. As evaluators, we are reluctant to take implementers' word at face value when they report very high fixed costs and claim very low variable costs, just as we would not take their word at face value if they claimed big benefits from their program without proof. More to come on this in the coming months.
2. Private operators may (illegally) try to exclude slower students
"PSL schools have been allowed to cap class sizes at 45—so have smaller classes than control schools (and many children previously enrolled—probably those from more disadvantaged backgrounds—have thus been excluded at short notice)."
"There is some evidence of providers being selective of children: those enrolled came on a “first come first served basis” but information available to parents is asymmetrical (better off parents get there first). All children in Bridge schools were also assessed and potentially re-graded."
It is true that Bridge International Academies, which runs 24 of the 94 PSL schools, has special permission to cap class sizes at 55 (not 45) pupils, while other operators may cap classes at 65 pupils—though some allow larger class sizes in practice.. To be clear, selective admissions based on fees or academic aptitude are explicitly forbidden by the PSL rules, and enrollment should be on a first-come-first-served basis.
Are operators abiding by these rules, and are poorer kids really getting equal access? This is one of the core concerns that the RCT was designed to address; and we would argue this issue highlights a strength rather than a weakness of the evaluation.
The key to our strategy here is something called intention-to-treat analysis. The basic idea is to track pupils who were in a given school before it was known whether the school would be part of the PSL program. That means we follow pupils who stay in school as well as those who leave or get kicked out. All of those kids who were in a PSL school last year count as part of the treatment group for the evaluation. So any operator who thinks they can boost their evaluation performance by rejecting weak students is woefully mistaken. It also means that we can track exactly who gets included and who gets excluded, and test whether poorer or slower pupils lose out, as alleged.
What we know so far is that despite the enrollment caps in PSL schools, they have boosted enrollment significantly relative to regular government schools. Free tuition seems to be popular, and there was apparently some excess capacity in government schools that could be filled once PSL started.
It remains to be seen in ongoing analysis of our data whether the new students flocking to PSL schools are (a) richer or better prepared, and (b) whether they were previously unenrolled, or are transferring from other public schools.
3. This model can't scale
"Some providers have been allowed to set conditions on which schools they would take over (e.g. Bridge insisted on schools on accessible roads, in clustered locations with electricity and good internet connectivity. Such conditions are highly atypical)"
”There is also evidence of selectivity of teachers and principals (with providers being allowed to remove those they do not consider good enough (who are transferred to other public schools)"
This is the first we've heard about demanding electricity in advance, but the other points are correct—particularly the demand for 2G internet connectivity in Bridge schools (as well as Omega schools) and reassignment of teachers. All of the private “partners” in the Liberia program had some say about where they would operate, and all had some ability to re-assign teachers who were unable to pass a Ministry test. Note that all partnership schools in Liberia take on the unionized, civil service teachers already working in the school, and those teachers cannot be fired.
The RCT is internally valid in that it will provide a reliable estimate of the effect on learning outcomes for a given population of students of converting a normal public school into a partnership school during the 2016/17 pilot. This in and of itself is policy-relevant information. The pilot is small (less than 3 percent of public schools), yet over 18 percent of the population is within 5 KM of a PSL school. But the bigger policy question in Liberia is whether converting even more regular government schools into partnership schools would raise scores.
The schools in the pilot have had a leg up because they could re-assign underperforming teachers and recruit new teachers from among recent training college graduates. We will continue to track teacher re-assignments through the evaluation, but data from the baseline shows that on average only one teacher per school from the 2015/16 teacher roster has been re-assigned to another location, and this number is the same in both treatment and control schools. Bridge (representing just over a quarter of total treatment schools) is the exception here, having on average 3 re-assignments per school.
Much like the issue of their funding advantage, the question is whether this privilege could be extended to many more schools if the program was expanded. If this is a zero-sum game of shuffling under-qualified teachers around the country, that's cause for concern—an issue we have planned to measure in the evaluation.
Similarly, if partnership schools simply picked the low-hanging fruit this year by working in easier environments, that also bodes ill for any future expansion. In reality, this is unclear. Some operators, like Bridge and to a lesser extent Omega, strongly insisted on working in a narrow range of schools with high infrastructure demands. But other operators volunteered to work in more remote areas with fewer resources at baseline (In all cases, treatment schools were randomized among a final list of eligible schools). The map below shows the distribution of partnership (i.e., treatment) and traditional public (control) schools across the country.
Overall, Liberia's partnership schools are not a random sample of the country, and they took over public schools which were somewhat larger and had better infrastructure than the national average. However, they are spread across 13 of Liberia's 15 counties, and are disproportionately concentrated outside Monrovia in poorer rural counties, so the pilot evaluation should give a good idea of whether this model can succeed in difficult environments.
4. Partnership schools will teach to the test
“The indicators used in the evaluation (likely to be variations on the EGRA / EGMA - Early Grade Reading / Maths Assessments) to measure literacy and numeracy are likely to be predictable to the private providers but less so to government schools (so there is a risk of distortion through teaching to the test)"
Teaching to the test is a valid concern in general, but not a serious concern for the evaluation for two reasons. First, we disagree that teaching to the test is a risk that is unique to partnership schools. There is little reason why private providers would be more likely than public school principals to conclude that the tests they'll be given for the midterm evaluation in 2017 might look similar to the tests they were given during the baseline survey in 2016. That said, we have explicitly reserved the right to change the learning assessment before then, or add entirely new modules, just to keep everyone guessing.
To sum up, we share many of the questions that have been raised by critics of Liberia’s partnership schools initiative. Indeed, the core concerns raised by advocacy organizations are a rephrasing of the central questions for the evaluation: is this model cost-effective relative to other possible education interventions, will these schools favor richer or better-prepared students, and how much value do these schools really add once we account for any differences in student composition? Our first round of follow-up survey data will be collected in June, and we can begin to answer those questions.
In the meantime, President Sirleaf and Minister Werner deserve credit for committing publicly to delaying any major expansion of the program until the evaluation is completed. Minister Werner said in a recent op-ed “…these are early days for PSL. While I believe it holds great potential, my team and I are clear that the program will not be scaled significantly until the data shows it works and we have the capacity within government to manage it effectively.”
We are not naïve enough to think that any kind of data will placate all critics of charter-style initiatives like this, but we do look forward to an ongoing debate that is increasingly disciplined by facts as data comes in from these pilot schools.
It’s no surprise that rich countries outperform poor countries on standardized tests. But if you compare kids with similar household wealth across countries, that gap disappears.
Every three years, the Program for International Student Assessment, or PISA, releases a new batch of standardized test scores comparing 15-year-olds around the world on reading, math, and science. In countries that underperform expectations, such as the United States, the rankings inevitably provoke calls for education reform to imitate the school systems of high performers. (“We should be more like Finland!” “No, Singapore!”) Jon Stewart even mocked this ritual on the Daily Show:
I always feel bad for whatever country is just above America on these lists, because invariably that country is used as a standard for just how far we have fallen as a people. Thirty-sixth, beneath the Slovak Republic. I mean, those f***ing people eat their own vomit.
Setting aside the Slovakian-American math comparison, richer countries generally do better on PISA. A lot better. Indonesia was the poorest country to administer the test in 2012, and Norway was the richest. Their performance gap is huge: Indonesian students who score extremely well, with math scores at the 90th percentile locally, would still be in the bottom half of Norwegian students. Zero Indonesian students scored high enough on reading to clear the 90th percentile in Norway.
So does that mean poor countries such as Indonesia, Peru, or Vietnam should be looking to rich countries like Norway and the United States for education policy advice? The answer depends, in part, on how much of the test-score gap between rich and poor countries you think reflects superior school systems in rich countries and how much is due to the simple fact that American and Norwegian kids are a lot richer and have innumerable advantages both in and out of school due to that wealth.
What if you compared Indonesian kids with similar wealth levels? Would Indonesia still underperform? This is the question my CGD colleagues Amanda Beatty, Lant Pritchett, and I are currently exploring in a new paper. We started off by computing the relationship between wealth and test scores within each country, as seen in the graph below. 
There’s a lot of heterogeneity across countries. In the United States, the wealth gradient in test performance is quite steep, while in Norway it is very shallow (in line with your preconceptions of America and Scandinavia). And some countries, like Vietnam, are simply off the charts with spectacular performance given its relative poverty. But overall, it turns out that on average rich and poor countries are roughly on the same upward sloping line relating household wealth to PISA scores. What does this mean? Crudely put, Indonesian and Peruvian students score about where you’d predict them to score in the United States given their household wealth. Poor countries don’t do any worse on PISA than most OECD countries once you adjust for their socio-economic demographics, and some poor countries like Vietnam do considerably better.
(Note that we’re careful not to give any causal interpretation to the relationship between wealth and scores within countries here. Our argument doesn’t require that. It’s entirely possible — indeed very likely — household wealth proxies for lots of other factors: including household factors like nutrition and parental involvement, as well as within-country variation in school quality. The point is simply to ask how a child with a given wealth level is expected to score in country X, given all the advantages and disadvantages they’re likely to have at home and at school.)
To make this a little clearer, we decided to rank countries not by their average score, but by the predicted score of kids with the same wealth level in each country. (Basically, draw a vertical line through Figure 1 at a wealth score of 50, and see where countries intersect that line.) Comparing apples and apples, which countries do best?
Here the results are even more striking: If you compare students at the global median of household wealth, the test-score gap between rich and poor countries essentially disappears. There’s no correlation between a country’s average wealth and the test performance of students in that country who are at the global median.
There’s an interesting analog here to work on global income inequality. In both cases, after controlling for everything we can control for, the gaps between countries are still huge, as Michael Clemens, Claudio Montenegro, and Lant Pritchett documented in an earlier CGD paper on earnings. Likewise, Branko Milanovic has demonstrated that the vast bulk of global inequality is between countries — it’s where you’re born that largely determines your income. Similarly here, the gaps between countries remain huge even when comparing children with similar socio-economic backgrounds.
The enormous difference is that in contrast to income, you wouldn’t necessarily maximize your test scores by picking to go to school in the richest countries. A student with global median wealth in Turkey performs much better on reading tests than an equivalent student in Norway. And Vietnam beats out the best performers in the OECD such as Japan and Canada.
There is one tempting interpretation of the first graph above that we would caution against. If Indonesia, Peru, and the United States are all on the same line relating wealth and test scores, you could argue that means economic growth is secret to better education in poor countries. As they move up the wealth gradient, scores will rise. As Ludger Woessman and coauthors have shown for OECD countries, wealth has been associated with higher test scores within countries for a long time. But as OECD countries have gotten richer, scores haven’t gone up. We should be careful when converting cross-sectional correlations into time-series forecasts.
So why then do some systems deliver so much more learning than others? We have very little idea. At CGD, we’re getting ready to launch a new research program on what makes for an effective education system, and how to reform ineffective ones. Most of the countries we’ll be focused on — low- and lower-middle-income countries in Africa and South Asia — don’t appear in the PISA sample at all. As we start this research program, it’s daunting to realize how little we know not only about how to make reform happen, but even which systems perform well and which ones don’t. One thing is clear though: we shouldn’t assume rich countries hold all the answers.
 Technical footnote: The biggest challenge in this exercise was the potential for a high degree in measurement error when asking schoolchildren about their household wealth. This turns out to be crucial. Measurement error will tend to underestimate the slope of the relationship between wealth and test scores and, by failing to fully account for the wealth-score gradient within countries, exaggerate the cross-country relationship between wealth and scores. The regressions underlying all the results shown here an instrument variables approach to minimize this problem. More details forthcoming soon in the full paper. If you’re curious in the meantime, see the Stata code posted here.
Stephen Taylor (Dept of Basic Education, South Africa), Dave Evans (World Bank), Vicky Colbert (Fundacion Escuela Nueva), and Jacobus Cilliers (Oxford University) discuss approaches to research system-wide reform in education.
A couple weeks ago we got to spend two days listening to an all-star line-up of education researchers present the current state of the art in “Research on Improving Systems of Education,” aka RISE. Here’s what we learned.
School systems are failing, and we don’t know what to do about it — but that doesn’t make it inevitable.
Lant Pritchett opened with an anecdote (gasp!) about his visit with students of failing, violent, abusive schools in a Cairo slum, his meeting the next day with the Prime Minister, and how glad he was the PM didn’t ask him how to fix the schools — because the academic literature has no convincing answers.
It’s not innate, it’’s not just poverty — cross-country learning gaps are big and they emerge due to differential school productivity, Abhijeet Singh argued using comparable longitudinal data from five developing countries from the Young Lives project.
On the upside, good schools have both social externalities and long-run impacts on society, as Leonard Wantchekon demonstrated with his research on modern effects of missionary schools under colonialism in Benin.
Leonard Wantchekon: I grew up in Zagnanado, a village of 300 people in rural Benin, with 50 kids. Today 11 of us kids have PhDs #RISElaunch
Karthik Muralidharan (University of California, San Diego) argues that randomized control trials can answer big, systemic questions.
Beyond RCTs, or bigger, better RCTs?
So what would a research program on education systems look like? Several presenters argued that randomized controlled trials — RCTs — are not restricted to narrow impact evaluations, but can answer big questions about systemic reform.
If you give me enough money and a willing government, I’ll randomize whole systems, said Karthik Muralidharan, essentially meaning that if we think school governance at, say, the district level matters more than individual school interventions, then we should randomize at the district level, sample size permitting.
Annie Duflo’s RCT of a teaching assistants program run by IPA and the Ghanaian Ministry of Education illustrated another approach to thinking about scalability, i.e., evaluating interventions embedded in dysfunctional government bureaucracies (which we know can produce very different results than small-scale NGO pilots).
Or RCTs can use interactions between different policy elements to understand the conditions necessary to make X work — as Karthik and Isaac Mbiti did in Tanzania, showing extra money only raised school performance when coupled with management training (and vice versa).
Relatedly, in the Gambia even delivering school grants and management training together had no effect on average, but results hinged on context . The program produced some impact in localities with more educated parents.
Kara Hanson (London School of Tropical Hygiene and Medicine) discusses how methods from health systems research might inform education research.
Others disagreed, arguing that we need methods to evaluate reforms where n=1.
Researching system reform often presents a small-N problem, said Kara Hanson, who illustrated ways of dealing with this in health research.
Luis Crouch fleshed out a practical approach to evaluating systemwide reforms and making (non-statistical) causal inference when n=1.
Alec Gershberg pushed for n=5, pointing to the potential for comparative case studies across countries.
But enough about methods, what kinds of reforms offer the most promise for dramatically raising student learning?
Jishnu Das (World Bank) discusses the rise of private schooling in Pakistan, and the long historical tradition of assuming parents don’t know what’s best for their children.
Public private partnerships: educational success and political failure?
The solution many poor households choose when faced with unaccountable public schools for is to exit:
Jishnu Das summarized a decade of data collection from Pakistan on the rise of low-cost private schools and experimental work to highlight and overcome the market failures that prevent them from working better.
Das’s presentation hinted strongly at the potential for voucher programs to combine private efficiency with public finance to ensure greater equity. Other presenters were armed with evidence on precisely such programs.
The results are positive…
Vouchers for poor students in Colombia to attend private vocational schools not only helped students, leading to higher graduation rates and higher earnings at age 30, as Michael Kremer showed; they also generated enough extra tax revenue to pay for the program.
According to Felipe Barrera-Osorio, building publicly funded private primary schools in Sindh Province, Pakistan, generated big gains in enrollment raised and test scores by a whopping 0.67 standard deviations.
Similarly, public funding for private schools in Seoul, South Korea, produced fewer violent incidents, higher college attendance, and better test scores — all for the same cost.
…but the politics don’t always work.
“If a Minister of Education is asking for advice on education reform, the first question you need to ask is ‘What kind of unions do you have?’” said Ben Schneider, noting that even the successful Colombia voucher program had been killed.
Illustrating these political challenges, James Habyarimana and colleagues found that PPP schools in Uganda attracted a large number of students and produced better test scores, but President Museveni still wants to scale back the program because it’s not as politically visible as the higher cost strategy of building new public schools.
.@PaulineMRose says: Public-private debate avoids the problem... how to get good teachers, how to reform teacher ed, etc. #RISElaunch
If public-private partnerships weren’t controversial enough, there was also an extended discussion of high-stakes testing and teacher incentives. We know that teacher quality matters HUGELY for student outcomes.
Remember that experimental study that made headlines showing a good kindergarten teacher in the U.S. is worth about $320,000 a year for his or her students’ future earnings?
Norbert Schady and co have done something similar for Ecuador, and found that good teachers are worth even more in Ecuador, relative to local GDP.
Sadly, teaching quality is often abysmal: Deon Filmer presented results showing that the average teacher in Mozambique, Uganda, and Tanzania is absent from their class more than half the time, and fewer than 3 percent of teachers in Mozambique, Nigeria, and Togo can competently grade homework based on the curriculum they teach.
Teacher quality is not an innate, immutable trait, though; incentives matter.
Chinese school teachers are effectively in a tournament for promotions, according to Albert Park, meaning they work hard to get more wages, but stop trying once their chance for promotion passes by.
Derek Neal listed some pitfalls of bad incentive design, and described what the ideal teacher incentive system would look like in theory.
Prashant Loyalka has actually tested Neal’s theory in China, and found bigger impacts on learning than with simper pay-for-perofrmance schemes.
Kehinde Ajayi (Boston University), who presented her research on school choice in Ghana, talks to Isaac Mbiti (University of Virginia) who presented results from a school funding and management training program in Tanzania.
Alas, high-stakes testing brings risks.
If you’re not careful, using standardized test scores as signals of school quality may encourage schools to manipulate which kids come to school as happened in Chile says Felipe Gonzalez.
And using tests to allocate kids to schools can end up being incredibly inefficient, as Kehinde Ajayi showed for Ghana.
On the plus side, getting the exam system right can encourage better assessment and better teaching lower down in the system, as Newman Burdett argued based on evidence from Pakistan.
A summary of summaries
The RISE program is just getting ready to launch six years of field research in several developing countries. So what do we take away from this discussion? Methodologically, the answer will probably be “all of the above.” The power of RCTs to provide compelling answers to policy questions is undeniable. But we’re interested in studying big, ambitious reforms that may already be ongoing and not subject to experimental study. We’ll need a diverse toolkit.
Substantively, the papers at the RISE conference suggested big impacts from focusing on systems for motivating teachers to perform and recruiting more high-quality teachers, as well as partnerships with the private sector. But caution is in order. Big open questions remain about the equity implications of market-oriented reforms, not to mention their political sustainability. As Dave Evans noted in the panel discussion, if you read five summaries of the evidence you get five different answers for “what works.” The right answers probably depends on myriad factors we don’t yet understand, from context to state capacity to political resistance to reform — all good topics for RISE research going forward.
Thanks to our CGD colleague Mari Oye for helping to organize the conference (and run the twitter account quoted above) and to our co-organizers, Clare Leaver and Jacobus Cilliers from Oxford University’s Blavatnik School of Government. The RISE program is supported by the UK’s Department for International Development.
While Modi has celebrated India’s rapid rise in the Doing Business rankings, the World Bank’s Chief Economist recently resigned amid controversy over methodological changes. Without those changes, India’s “jump” in the rankings looks much more modest.
Speaking to the assembled billionaires at Davos last month, India’s Prime Minister Narendra Modi announced triumphantly that “the largest democracy on earth is also the fastest growing major economy.” Modi used the platform to advertise that “India is open for business” and tout his accomplishments as an economic reformer—and that claim seems to have the backing of international financial institutions.
Last October, the World Bank announced that “India Jumps Doing Business Rankings with Sustained Reform Focus” moving from 130th to 100th overall. (Note the focus on the jump, or change.) The Indian press and pundits declared that the Bank had “endorsed Modi’s reform credentials.” Modi himself turned the change in Doing Business rankings into a talking political sales pitch.
Historic jump in ‘Ease of Doing Business’ rankings is the outcome of the all-round & multi-sectoral reform push of Team India. pic.twitter.com/DhrEcuurgi
On January 12, the World Bank’s chief economist at the time, Paul Romer, told the Wall Street Journal he had lost faith in the integrity of the Doing Business index, suggesting it was being politically manipulated—particularly to embarrass Chile’s socialist president Michelle Bachelet. After a public rebuke from the World Bank CEO, Romer retracted the remarks, and then on Wednesday announced his resignation.
But as we showed in our “chart of the week”a few weeks back, the pattern Romer had uncovered speaks for itself. And Chile wasn’t the only country affected. The methodology changes pushed dozens of countries up and down as well—which bring us back to India’s recent performance on the index.
How did India perform if we ignore the questionable methodological changes?
Somewhat awkwardly for the World Bank, India’s celebrated rise in the Doing Business rankings turns out to be mostly an artefact of methodological changes.
A few weeks ago we presented new data and analysis that reproduced the Doing Business rankings, dropping all of the new indicators that had been added over time, and only reporting changes for a fixed sample of countries. In an index with a more consistent methodology, Chile’s rise and fall disappears, and—if you scroll down to the bottom of last week’s post—so does the “Modi bounce.”
One comment we heard was that our “corrected” index was too narrow, ignoring the new and sometimes valuable features Doing Business has added. So to overcome that, we’ve gone back and tried a second approach, with much the same results for India: no recent spike in the rankings.
This time we use all the indicators available for a given year, relying on the World Bank’s own scores for each country and each component. Instead of dropping new indicators, we splice the series together whenever there is a methodological innovation. So for instance, when the World Bank changed the underlying indicators in the “ease of getting credit” index in 2014, they kindly reported the score using both methodologies for that year. India scored 81.25 on the old methodology in 2014 and 65 using the new method. Our approach is simply to multiply all of India’s credit scores by 1.25 (=81.25/65) for every year after 2014, so there is no artificial jump in India’s credit score.
The chart above has been updated to correct a mistake spotted by a reader, Vinayak Garg, who helpfully downloaded our data and code to reproduce the graph. The text of the post remains unchanged by this correction. Click here to see a side-by-side comparison of the corrected version and the original graph, which inadvertently omitted some sub-components from the index. The main point stands: the official Doing Business rankings show India rising 30 places in 2018, where as we previously found India rose only 7 places. After correcting our coding error we find India rose only 5 places, but from a higher starting point under both Congress and BJP governments.
Whichever method we use, once you iron out the methodological changes, India’s recent jump in the Doing Business rankings looks much more modest. To its credit, the World Bank office in Delhi cautioned against focusing on changes in the rankings when this issue reached public prominence in India. But those cautions were probably outweighed by the World Bank’s very own headlines, encouraging a misleading focus on India’s spurious “jump.”
A very misleading headline from the World Bank…
Which methodological changes explain this spurious jump? If we break the index down into its pieces (see below), we find that there was no single piece of the index that made the difference. Even the aggregate scores are not very different when comparing the official Doing Business scores and our revised series that irons out methodological changes.
The big change comes from ranking, not scoring. Because many countries perform similarly on the index, even small discrepancies in the score can produce wild swings in the rankings, which appears to have happened in India’s case—a point Martin Ravallion, former manager of the World Bank’s research department, has pointed out recently as well.
It seems that India’s rank moved up and down more due to the addition of new countries to the Doing Business sample than changes in methodology.
Even if you love the Doing Business index, you should acknowledge changes in ranking mean very little
Reasonable people disagree about the usefulness of the index. Critics point to serious conceptual flaws, like measuring the costs of regulation while ignoring the benefits, and how poorly the index predicts actual conditions as reported in business surveys. Defenders note that the index has spurred many countries to accelerate regulatory reforms which, they contend, has boosted growth. This debate won’t be settled anytime soon.
Reasonable people should all agree, however, with one simple fact: changes over time in the Doing Business rankings are not particularly meaningful. They largely reflect changes in methodology and sample—which the World Bank makes every year, without correcting earlier numbers—not changes in reality on the ground.
It is probably coincidence, rather than a nefarious neoliberal plot hatched in Washington, that these changes helped a right-leaning government in India and hurt a left-leaning government in Chile. But you don’t have to be a conspiracy theorist to think the World Bank has been irresponsible in posting headlines about “jumps” in the Doing Business rankings, when it knew these numbers weren’t comparable over time. If it isn’t scrapped altogether, serious reform of the methodology and governance of the Doing Business index are needed.
Debt relief wiped away much of Africa's sovereign debt, but after a decade of growth, debt stocks are rising again. Here's a look at the numbers, and how we got here again.
Remember debt relief?
Twenty years ago, Bill Clinton was president, Bono was still a rock star, and the jubilee movement to forgive poor countries' debts was perhaps the central preoccupation of development policy debates. The IMF and World Bank's Heavily Indebted Poor Countries initiative (aka HIPC) got the ball rolling in 1996, but it took another two decades of Bob Geldof concerts and negotiations between the Bretton Woods institutions, Paris Club creditors, and the debtors for the process to culminate with (hypothetically) 100 percent debt relief for 36 countries—30 of which were in Africa.
Part of the reason you don't hear so much about debt relief any more is that it worked, at least for a while. Countries like Ghana saw its debt fall from 120 percent of GDP in 2000 to just 12 percent in 2006. Mozambique's fell from over 200 percent down to the mid-20s over a similar time period. While we should be careful about assigning a causal role to debt relief, economic growth rates across Africa were strong over the next decade, and the "Africa Rising" narrative took hold.
This time is different: private creditors, not the Paris Club, hold much of Africa's debt now
Fast forward to 2018, and some of those same countries are gradually accumulating fairly significant sovereign debts again, which have led to talk of a new African debt crisis.
The graphs above are based on the World Bank's International Debt Statistics, where we've pulled out numbers for the eight low- or lower-middle income African countries with the highest debt-to-GDP ratios today (ignoring a couple of small countries with total GDP below 10 billion dollars).
A few simple facts stand out:
Sudden fall: Several countries saw a dramatic decline in debt stocks after they benefited from the Multilateral Debt Relief Initiative in 2005.
Gradual rise: The countries shown were chosen precisely because they have run up debts recently. In raw numbers, the figures look more extreme—Ghana's total sovereign debt is more than double today than it was at its peak before debt relief—but Ghana's economy has grown at a steady clip, so as a proportion of GDP the figure has risen only gradually.
Commercialization: In the 2000s, most of the debt was owed to multilateral institutions like the World Bank and IMF and bilateral creditors who formed the Paris Club. Today, a much larger share of African debt is held by private banks and bondholders, so the dynamics of any hypothetical workout would be considerably different.
Did the "Africa Rising" narrative feed premature enthusiasm for commercial bond offerings?
After stepping down as director of the IMF's Africa Department last year, my colleague Antoinette Sayeh wrote an essay for CGD reflecting on what could have been done differently:
In this my first post-IMF piece, I focus on whether the volume of Fund financing for SSA frontier markets… should have been greater, and whether such financing could have helped contain the indebtedness of those countries, many of which issued sovereign bonds… Has the Fund actually adapted to the need for [non-concessional loans]to these countries as they climb up the income ladder?
Poor countries need development finance. The question is where they're going to get it.
Economic growth converted a number of major African economies from low- into lower-middle income countries, and as Antoinette notes, in the process they seem to have fallen into an awkward middle ground: not facing any imminent crisis, they were too successful for most kinds of concessional finance from the multilateral financial institutions, but still fragile enough that the cost of finance from commercial creditors was high.
One consequence is that the cost of debt service (i.e., repayment flows not debt stocks) have bounced back more quickly than debt stocks
Washington systematically underestimated African economies’ credit demand
Our analysis suggests the IMF has had a perennial bias toward optimism—and hence inaction—in the face of Africa's recent debt accumulation. Swearing "never again" after the last bout of debt relief, the World Bank and IMF agreed to periodically monitor borrowers and issue joint Debt Sustainability Analyses, which are basically forecasts of a country's economic growth, exports, government revenue, and debt levels. We downloaded the archived PDFs of these reports for the two most-heavily indebted African countries, Ghana and Mozambique, and decided to look at how well the forecasts panned out.
On economic growth, the Bank and Fund performed pretty well (though for a slightly earlier period, my former colleague Ben Leo found some evidence of over-optimism in IMF growth projections for Africa as well). Forecasts were rosy, and reality mostly lived up to expectations. Contrast this with recent European crises. There the IMF was faulted for its wildly optimistic forecasts of growth rates in Greece, Spain, Italy, and Portugal (see Figures 2-6). No debt workout was needed, was the implicit message, because Greece could tighten its belt and grow its way out of distress. Each year, Greek GDP declined, and each year the IMF said they were about to turn a corner.
While the Bank-Fund growth forecasts were much better for Ghana and Mozambique, their forecasts of the countries' debt levels have been comically wrong, year after year. As debt levels rose, the official Debt Sustainability Analyses predicted they would flatten out and soon fall. But they didn't. And this was not a one-off mistake. As the saying goes, "fool the IMF once, shame on Ghana; fool the IMF every year while your debt levels soar, maybe shame on the IMF too."
To be fair to the Fund, Ghana and Mozambique are special cases. From 2011 to 2015 the Ghanaian government, feeling flush with potential oil revenues, basically told the IMF to take a hike. And the Mozambican government engaged in secretive borrowing while keeping both its creditors and citizens in the dark.
Nevertheless, reading the Debt Sustainability Analyses and looking at these graphs, the impression is that the World Bank and IMF treated their advice as forecasts. They repeatedly cautioned both countries to restrain their borrowing and spending, and forecast that they would comply. In hindsight, neither country had any intention of doing so.
Are countries borrowing too much, or are multilateral institutions lending too little, forcing poor countries toward more expensive commercial loans? In either case, several African economies appear to be on course for a new period of debt distress and pressure for fiscal austerity. More, not less, multilateral lending in Africa might be a partial solution. Otherwise, it may soon be time for Bono and Bob Geldof to dust off their guitars.
Thanks to numerous colleagues for comments. This post emerged from discussions with Masood Ahmed, Nancy Birdsall, Alan Gelb, John Hurley, Charles Kenny, Todd Moss, Mark Plant, Vij Ramachandran, and others, but the views expressed are ours alone.
I'm a little late to this, but recently Chris Blattman set off an interesting debate by criticizing Bill Gates' recent interest in the quality of GDP statistics in Africa. Chris worries that Gates is falling into the trap of "seeing like a state" -- i.e., from the top down, obsessing over national statistics -- rather than a bottom-up entrepreneur who, presumably, couldn't care less about aggregate GDP numbers.
I don’t mean to pick on Bill Gates. Most other donors do likewise. What they promote is not seeing like a state – i.e., collecting data to answer policymakers questions – but rather "seeing like a donor". Countries are the unit of analysis and need to be lined up into comparable data points. And African governments are paid to collect statistics whose main raison d’etre is populating a World Bank or UN database.
Consider education in Kenya. Donors need to allocate aid across countries, so they end up looking at maps like the one on the left below, which shows net primary enrollment rates for 52 African countries from the World Bank's 2010 World Development Indicators. This is a great map for a donor sitting in Washington, because it allows them to evaluate Kenyan and Tanzania performance in primary schooling (by one very crude metric) on a comparable basis.
But the map on the left is pretty useless for Kenyan policymakers. Why should they care how they compare to Egypt? They need to know which schools need new textbooks and which ones need new teachers. They get that from the country's Education Management Information System (EMIS), three times a year, for all twenty-thousand government schools in the country.
Ken Opalo trolled Chris’s original blog post quite beautifully with a counter-post titled, “Does Chris Blattman hate state capacity?” (The answer was “no”.) But Ken made a strong case that governing a place like Kenya would require both reliable data, and more disaggregated data – possibly linked to the recent devolution of power to counties.
International comparability be damned. Governments need disaggregated, high frequency data linked to sub-national units of administrative accountability.
Back in here in donor-land (DC), the mood is very different. There are currently a lot of proposals floating around to collect more internationally-comparable household surveys – for instance, as part of the post-2015 MDG agenda. The Gates Foundation is funding tailored surveys on agriculture and family planning across the region. USAID runs a different set of agriculture surveys for its "Feed the Future" initiative. The World Bank and various UN agencies have their own competing and often duplicative surveys, all of which get foisted on national statistics bureaus. But these small-sample one-off surveys answer very few of the questions that domestic policymakers are asking -- largely because they're too aggregated, too infrequent, and not linked to sub-national units of administrative accountability.
Lest you think this all sounds very top down and Soviet, it turns out seeing like a state is not so different from seeing like a citizen. What is it that citizens are demanding in the movement to open data platforms? People want to know how their school (or clinic, or local council) compares to the one down the road. They want the map on the right – which is in fact now publicly available at kenyaopendata.go.ke.
So the lesson here is that donors need to back off, abandon household surveys, and focus on strengthening administrative data systems. Well, not quite.
Big, Dumb Data?
There's just one problem: the real-time, local-level information coming out of the Open Data Portal appears like it might be, sort of, well, how should I put this… systematically wrong.
Marc Bellemare recently posted a nice critique of the recent enthusiasm for “big data” from a social science point of view. I want to take issue from a policy perspective.
In the Ministry of Education’s administrative data, primary enrollment rates appear to be steadily rising, with a big jump upward in 2003 when Kenya abolished all school fees in government primary schools. But survey data paints a very different picture. Independent figures from the Kenyan National Bureau of Statistics (KNBS) and the Demographic and Health Survey (DHS) show enrollment rates that are completely flat over time. There’s no sign of any increase whatsoever, even when school fees were abolished. (See the graph below. The fact that the level of the blue line is higher than the green line probably tells us more about differences in the two surveys, which use slightly different methodologies. Comparing apples and apples shows no rise in either case.)
So why does administrative data reported by schools differ so much from survey data reported by parents?
Amanda Glassman and I have a working paper forthcoming on this question, looking at these discrepancies between administrative and survey data in education and health across Africa. Here's a spoiler: when the Ministry of Education abolished primary school fees, it radically changed the incentives for truthful reporting by head teachers. There are some safeguards in place to try to avoid outright lying, but at the end of the day, schools get allocated more teachers and more funding if they report more pupils. So schools have an incentive to exaggerate their numbers. Parents, in contrast, don't. Incentives, incentives, incentives. The same point applies, Amanda and I hope to show, when you ask agricultural extension workers to report how much maize was grown in their village, or nurses how many patients they saw. Incentives to misreport plague administrative data systems.
This, rather than the need for more duplicative household surveys, is the big challenge facing African statistics. Right now governments face a trade-off between high quality survey data of limited relevance, and low quality administrative data that actually fits their needs. It doesn’t have to be this way. But to overcome the trade-offs donors are going to have to back off with their pet survey projects, and stats bureaus across Africa will need to exert some renewed independence, and stop serving as research consultancies for donors.
Thanks to Amanda Glassman for helpful comments on this post.
Internationally comparable test scores play a central role in both research and policy debates on education. However, the main international testing regimes, such as PISA, TIMSS, or PIRLS, include very few low-income countries. For instance, most countries in Southern and Eastern Africa have opted instead for a regional assessment known as SACMEQ. This paper exploits an overlap between the SACMEQ and TIMSS tests—in both country coverage, and questions asked—to assess the feasibility of constructing global learning metrics by equating regional and international scales. I ﬁnd that learning levels in this sample of African countries are consistently (a) low in absolute terms; (b) signiﬁcantly lower than predicted by African per capita GDP levels; and (c) converging slowly, if at all, to the rest of the world during the 2000s. Creating test scores which are truly internationally comparable would be a global public good, requiring more concerted effort at the design stage.
If data wants to be free, then PovcalNet, the world’s leading dataset on global poverty, is happier today because it was recently made available for download in bulk by my guests on this week’s Wonkcast, CGD research fellow Justin Sandefur and research assistant Sarah Dykstra. Scraping the data was no easy task: it required devising code that queried the database for one answer at a time, 23 million times, over nine weeks, then reassembling the 8 million resulting data points answers into a single dataset. They then posted the dataset and a related paper online for the use of researchers around the world.
Justin and Sarah tell me that they were motivated to scrape the PovcalNet website in part because they needed the full dataset for their own research, and in part because they knew other researchers had a similar need. Lacking the full dataset, they and others previously had no option but to spend hours pointing and clicking, one number at a time, to get the specific information they needed. (The code needed to run the queries was beyond what we could manage here at CGD, so the pair turned to Sarah’s brother, independent programmer Benjamin Dykstra.)
Since individual data points were already online—albeit not in a readily accessible format—the project involved no “hacking.” I ask whether they tried first just asking the World Bank for the dataset. Justin explains that, "...the underlying raw data isn’t even available to many researchers within the Bank.”
“There’s a lively internal debate in the World Bank about whether or not this data should be public,” Justin tells me. “But not all data that the World Bank has are covered by the open data policy…it was pointed out to us that PovcalNet is not.”
Justin says that the entire process illustrates the importance of making research data publicly available.
“We’re living in a new era where there are a lot of people participating in this analysis and this conversation, and a million eyeballs can find lots of mistakes.” Justin says. “So let’s put all the data and the code in the public domain and open up that conversation.”
So, what exactly was the World Bank’s response to their efforts and the resulting new poverty estimates?
“Annoyance is probably the right word,” Justin says. “The stance of the research department now seems to be, reading between the lines, that ‘we don’t really trust these [new PPP] numbers, and we’ll reserve judgment on whether we should use them yet.’”
It’s an exciting story, with some unexpected twists and turns. To hear it, and learn what Justin and Sarah have planned next, tune in to the full Wonkcast.
As African leaders meet in Washington this week, one issue is not on the agenda: the poor quality of basic economic and social data in the region. Maybe this year’s GDP re-base in Nigeria, which resulted in an 89 percent increase, was a tip-off? While inconvenient to the #AfricaAscending narrative around town, our recent work suggests that many basic data are in fact systematically distorted.
In our paper, we find that misrepresentation of national statistics in education and health does not occur merely by accident or because of a lack of analytical capacity — at least not always — but rather that systematic bias in administrative data systems stems from incentives of data producers to overstate development progress.
Administrative and Survey Data Don’t Match
Comparing administrative and survey data across 46 surveys in 21 African countries, we find a bias toward overreporting school enrollment growth in administrative data. The average change in enrollment is roughly one-third higher (3.1 percentage points) in administrative than survey data (an optimistic bias that is completely absent in data outside Africa. Delving into the data from two of the worst offenders, Kenya and Rwanda, shows that the divergence of administrative and survey data series was concomitant with the shift from bottom-up finance of education via user fees to top-down finance through per-pupil central government grants. This highlights the interdependence of public finance systems and the integrity of administrative data systems. Difference-in-differences regressions on the full sample confirm that the gap between administrative and survey of just 2.4 percentage points before countries abolished user fees grew significantly by roughly 10 percentage points afterward.
Donors also play a role. In 2000, GAVI Alliance offered eligible African countries a fixed payment per additional child immunized against diphtheria-tetanus-pertussis (DTP3), based on reports from national administrative data systems. Building on earlier analysis by Lim et al. (2008), we show evidence that this policy induced upward bias in the reported level of DTP3 coverage amounting to a 5 percent overestimate of coverage rates across 41 African countries.
It’s Not Just Education and Health
Other work by Justin suggests that official estimates of consumer price indices have been inaccurate, and — once correcting for these accuracies — rates of growth and poverty reduction in Africa are modestly slower on average than published estimates based on official data.
Inaccuracies in basic data are due in part to perverse incentives created by connecting data to financial or reputational rewards without checks and balances. But the problem of inaccuracy is also related to political interference and statistical agencies that have been inadequately and inconsistently funded over the years. Together, these factors make up a political economy of bad data.
To get to a political economy of good data, our joint working group report with the African Population and Health Research Centre lays out some ideas: (i) fund more and differently; (ii) build institutions that can produce accurate, unbiased data; and (iii) prioritize the accuracy, timeliness and availability of the basic data on births and deaths; growth and poverty; sickness, safety and schooling; and land and environment, that policymakers and citizens can use to generate real progress in development.