Earlier this month, the Global Fund’s Office of the Inspector General (OIG) released a new audit report on Wambo.org, its online procurement platform for drugs and other health commodities, which is intended to streamline global health procurement and generate savings of $246 million by 2019. The headline: despite high marks from its users, Wambo.org is not yet on track to deliver the projected savings—and, in fact, it has not even recouped its far more modest start-up costs of about $10 million. But more than the headline, a close read of the report narrative helps us understand why reality does not yet reflect the Global Fund’s optimistic assumptions—and, reading between the lines, suggests three important lessons for the Global Fund and other international funders.

1. Calculating savings requires a meaningful counterfactual

The headline of the audit report suggests that Wambo.org is not on track to achieve its projected $246 million in savings by 2019—the “realistic” scenario set out in its original business case. But savings can only be calculated relative to a meaningful baseline or counterfactual, and the report does not inspire confidence that the methodology is sufficiently responsive to these other factors. Based on past trends, it’s likely that prices for health commodities will decline by 2019—but could that be attributed to Wambo? What about the new deal, spearheaded by the Bill & Melinda Gates Foundation, to provide best-in-class HIV treatment for just $75 per patient-year? Or the efforts of the Global Fund’s Pooled Procurement Mechanism more generally, which overlaps in scope with Wambo.org and has already claimed $650 million in savings over four years? Moving forward, an honest assessment of Wambo-related savings must account for these other factors by establishing a clear counterfactual.

2. But savings ≠ value

An ad for a cheap laptop. Used to illustrate the point that savings do not necessarily equal value.

A $240 laptop is not necessarily good value for money.

True story: last fall I needed to buy myself a new laptop, but didn’t want to spend a lot of money. So I was delighted to open Amazon and find a shockingly cheap option for $240. It seemed fully operational, came loaded with a year-long subscription to Windows 10, and otherwise checked all the boxes—plus, I was “saving” hundreds of dollars! The computer arrived, turned on, and took 5 minutes to open an internet browser window—before promptly switching itself off. It continued to spontaneously shut down at random intervals, rendering it more or less unusable. And while $240 may be pretty cheap for a laptop, it’s pretty expensive for a two-pound block of gray plastic. (Thankfully, Amazon refunded the broken laptop after hearing my sob story.)

Clearly, value for money means more than a cheap sticker price; it means selecting products that deliver the most value for a given resource envelope. (Relatedly, my colleagues Kalipso Chalkidou and Janeen Madan Keller recently offered a compelling argument for why the Global Fund should adopt health technology assessment more broadly within its programs and operations.) The audit report does not directly comment on this issue with respect to Wambo, but hints at a continued lack of clarity. For example, the OIG discusses Wambo’s “catalogue function,” a promising feature that alerts purchasers when a less expensive option is available. But according to the report, the prompt appears to be generated on the basis of nominal price alone—not on cost-effectiveness. In theory, this could be pushing purchasers to choose less cost-effective options, even as they appear to generate “savings” based on sticker price alone. Perversely, this could even be counted as “savings” relative to a baseline in which more cost-effective products were purchased.

More generally, the Global Fund (and the Wambo.org project team specifically) needs to rethink its focus on achieving “savings’” without clear reference to the effectiveness of products being purchased. And going forward, the business case for Wambo.org should focus less on savings and more on delivering value: the most cost-effective health products, at sustainable prices, delivered when and where they’re needed.

3. Project planning requires assumptions—but measuring impact requires rigorous and transparent evaluation

A frame from the webcomic XKCD. Comic 1339 - When You Assume. Creative Commons.

Image by XKCD/Randall Munroe, Creative Commons.

While Global Fund documents projected that Wambo.org would save $246 million by 2019, the audit report suggests those original estimates were well off-base, and spends a substantial amount of time walking through the points where reality diverged from expectations. The OIG reports that the projections were developed by an external consultant using a proprietary model—and, somewhat alarmingly, the OIG was unable to access all the methods and data upon which those projections were based.

Reading through, it’s clear that the original estimate derived from a number of somewhat heroic assumptions, some of which are not directly tied to the functionality of Wambo.org per se. For example:

  • Assumption: 50 percent of the Global Fund’s overall procurement volume would be channeled through Wambo.org by 2019, which would in turn drive substantially lower prices for health commodities, accounting for $150 million in savings by 2019.

  • Reality: Onboarding has proceeded at a far slower pace, existing long-term agreements with suppliers remain in place (no change in prices), and there’s no evidence yet that greater volume will necessarily lead to additional savings beyond the negotiated prices already in place. Prices may already be at their lowest sustainable level due to aggressive negotiation in the previous period.

  • Assumption: The Global Fund would be able to renegotiate its logistics agreement, saving $27.8 million by 2019.

  • Reality: No change.

Of course, we all make assumptions; indeed, clearly articulated assumptions are an essential input to any sort of planning document or projection. And, as always, the Global Fund deserves enormous credit for its robust and transparent investigatory arm, which is what allows us to interrogate these methods in the first place.

Nonetheless, the large gap between expectations and reality underlines that while projections can be based on assumptions, impact must be transparently, rigorously, and directly measured based on actual data. In this audit report, we have a partial example of such a process, where the OIG is questioning/attempting to assess each of the projected savings levers. But the OIG’s audit-oriented process should not substitute for an independent, methodologically rigorous, and transparent impact evaluation, measured against a clearly articulated counterfactual.

Conclusion

Wambo.org seems to be a useful tool for helping to manage the Global Fund’s existing procurement arrangements, and high user satisfaction offers a great entry point for further expansion. But the clear divergence between the Global Fund’s lofty expectations and the relatively marginal impact observed thus far underlines the importance of understanding “value,” interrogating assumptions, and directly measuring impact against a meaningful counterfactual.