Conferencia IEA: "Evaluación de impacto y su relación con temas éticos"

Introduction to Ethical Impact Evaluation

Opening Remarks

• The speaker expresses gratitude for the invitation and acknowledges the interdisciplinary nature of the discussion, highlighting the challenges in addressing diverse disciplines without causing offense.

• The speaker identifies themselves as a professor at a Catholic university and part of J-PAL, emphasizing the importance of ethical considerations in their work.

Key Themes

• The presentation will explore two main topics: the role of impact evaluation in public policy and the ethical issues that arise from conducting such evaluations.

• The speaker believes that social science research has significantly influenced public policy through program evaluations, which can both inform policy development and raise ethical questions.

Understanding Impact Evaluation

Definition and Purpose

• Clarification on what constitutes an impact evaluation versus process monitoring; impact evaluations assess whether a program achieves its intended outcomes.

• Process evaluations focus on implementation details, such as whether resources (e.g., books for children) are delivered as planned.

Methodological Approaches

• Emphasis on descriptive evidence gathered during process evaluations, including metrics like satisfaction levels among participants.

• Impact evaluations aim to determine causality—whether observed changes are due to the program itself rather than external factors.

Challenges in Causal Inference

Philosophical Underpinnings

• Discussion on the historical context of impact evaluation rooted in causal inference principles; understanding "what would have happened" without intervention is crucial.

• The analogy of controlled laboratory experiments is used to illustrate challenges faced in social sciences where controlled conditions are often absent.

Counterfactual Considerations

• The speaker shares a personal anecdote about creating a false belief in their child regarding light switches to explain how counterfactual scenarios can be misleading if not properly constructed.

• Highlighting that constructing counterfactual scenarios is essential yet challenging since they cannot be directly observed; researchers must estimate these scenarios instead.

Methods for Evaluating Impact

Types of Methods

• Overview of three categories for evaluating impacts: non-experimental methods (less reliable), quasi-experimental methods (more robust), and experimental methods (most rigorous).

• Focus on experimental methods promoted by J-PAL, which involve random assignment to ensure comparability between treatment and control groups.

This structured approach provides clarity on key concepts discussed throughout the transcript while allowing easy navigation through timestamps for further exploration.

Impact Evaluation in Public Policy

Importance of Measuring Impact

• The credibility of impact measurement methods is discussed, emphasizing the need for fewer assumptions to accurately assess public policy effectiveness.

• Key objectives for measuring impact include cost-effectiveness, identifying program components with varying impacts, testing new ideas, and uncovering unplanned benefits or adverse effects.

Evolution of Evidence-Based Policy

• The transition from evidence-based to evidence-inspired public policy reflects lessons learned over time about the complexities involved in policymaking.

• While impact should not be the sole criterion for determining good public policy, it remains a relevant input for decision-making regarding resource allocation.

Ethical Considerations in Program Implementation

• Ethical reasons may prevent the implementation of programs that show positive impacts; an example includes educational policies that could lead to segregation based on learning levels.

• The importance of considering ethical implications highlights that even effective programs can raise moral concerns about their execution and outcomes.

Role of Impact Evaluation in Decision-Making

• Impact evaluation serves as a significant input but should not dominate the decision-making process; other political and ethical considerations must also be factored in.

• Evaluating whether resources are generating impact is crucial; if not, alternative uses for those resources should be explored.

Challenges with Randomized Control Trials (RCT)

• Ethical objections often arise when discussing randomized control trials (RCT), which are seen as a gold standard for measuring impact but face scrutiny from policymakers.

• Many criticisms directed at RCT methodologies also apply to other econometric and statistical methods used in evaluations.

Methodology of Randomized Evaluations

• A typical approach involves selecting a target population and using random sampling to create representative samples for evaluation purposes.

• Random assignment ensures comparability between treatment and control groups, allowing researchers to attribute differences directly to the program being evaluated.

Addressing Concerns About Randomization

• The complexity of implementing random assignments raises concerns among organizations accustomed to more selective criteria for participant inclusion.

• Despite external factors like natural disasters or economic crises affecting both groups equally, randomization aims to create conditions similar to controlled laboratory settings.

Impact Evaluation and Ethical Considerations in Program Implementation

Demand for Programs and Resource Allocation

• The demand for a program is assessed, and if met, the program typically sends feedback. An evaluation of impact is justified when resources are scarce to serve the population.

• When resources are limited, random selection methods like lotteries can determine who receives benefits without denying access to those who should receive them.

Ethical Justifications in Randomized Experiments

• Ethical justification for using random experiments arises when all individuals have an equal chance of accessing a program, rather than outright denying someone assistance.

• A historical example from Africa illustrates the ethical dilemma faced during a randomized trial on circumcision's effectiveness in reducing HIV transmission among high-risk populations.

Case Study: Circumcision Experiment

• Researchers aimed to demonstrate that circumcision could lower sexually transmitted disease rates among high-risk groups but faced overwhelming interest beyond available resources.

• After six months, clear evidence showed significant health benefits from circumcision, prompting researchers to halt the experiment and advocate for immediate government action to provide this service universally.

Limitations of Randomized Trials

• The ethical implications arise when researchers know a treatment works yet continue withholding it from participants; this raises questions about the limits of randomized trials.

• Alternative approaches include phased interventions where control groups receive benefits later instead of being completely excluded from receiving help.

Evaluating Impact and Addressing Negative Outcomes

• There’s an assumption that well-designed programs will yield positive effects; however, negative outcomes can occur, raising concerns about resource allocation towards ineffective initiatives.

• The ethics of learning in public policy come into play when evaluating whether continued investment in certain programs is justified based on their actual impact versus anticipated outcomes.

Challenges in Public Policy Learning

• New policies often emerge despite existing academic research indicating potential flaws or negative consequences; this highlights a reluctance within public policy to adapt based on evidence.

• Common criticisms against randomized evaluations include ethical concerns and high costs associated with conducting thorough assessments compared to implementing programs directly.

Cost Implications of Evaluations

• Evaluating impact can consume as much funding as program implementation itself; thus, careful consideration must be given regarding resource allocation for effective measurement versus direct service delivery.

• Data collection constitutes approximately 90% of evaluation costs; post-pandemic challenges have further increased these expenses due to participant reluctance.

Impact Evaluation and Data Utilization in Chile

Importance of Transparency in Resource Allocation

• The speaker emphasizes the need for transparency regarding resource allocation, noting that it is crucial to understand the costs associated with different evaluation methods.

• They highlight that using alternative data sources can be more efficient than traditional surveys, which often face challenges.

Innovative Data Collection Methods

• The discussion includes how inflation measurements have evolved, with agencies now utilizing supermarket visits and online price checks instead of direct consumer questioning.

• The speaker mentions the feasibility of tracking individuals across various databases in Chile due to its technological infrastructure.

Government Programs and Data Access

• A government program called "RIS" allows researchers to access fertility data from civil registries rather than conducting surveys, streamlining data collection processes.

• This method reduces costs significantly while ensuring compliance with legal requirements for data access.

Cost-Efficiency in Evaluations

• The speaker argues that experimental evaluations can be conducted at a lower cost by not requiring baseline data if groups are assumed to be similar before interventions.

• They stress the importance of identifying ineffective programs early on to save government resources.

Quality vs. Quantity in Research Studies

• There is a call for prioritizing fewer high-quality studies over numerous low-cost evaluations that fail to accurately measure program impacts.

• An anecdote illustrates a governmental organization’s failure to learn from past impact evaluations, highlighting systemic issues within public sector learning.

Comparative Analysis of Educational Interventions

• The speaker discusses various educational strategies aimed at increasing school attendance among children globally, emphasizing evidence-based approaches.

• Examples include providing iron supplements or addressing parasitic infections as means to improve health and educational outcomes for children.

Education and Poverty: Insights from Impact Evaluations

The Cost of Education Programs

• The speaker discusses the financial challenges associated with educational programs in Mexico, noting that costs are significantly higher compared to other regions.

• Many families, particularly those in marginalized areas, require substantial financial incentives to send their children to school due to economic hardships and isolation.

Health as a Barrier to Education

• A key finding is that children's health directly impacts their ability to attend school; poor health diminishes energy levels necessary for education.

• Interventions aimed at improving children's health are more cost-effective than providing scholarships, which often incur additional expenses.

Ethical Considerations in Learning from Impact Evaluations

• The discussion shifts towards ethical implications of not learning from impact evaluations, drawing parallels with biomedical experimentation where knowledge can save lives.

• Questions arise about the sufficiency of examples demonstrating the benefits of impact evaluations and how frequently they yield successful outcomes.

Economic Growth vs. Public Policy in Poverty Reduction

• The speaker argues that significant reductions in global poverty since 1980 are largely attributed to economic growth rather than public policy alone.

• Approximately 50% of poverty reduction is linked to broader economic improvements rather than specific policy interventions.

Effectiveness of Conditional Cash Transfers

• Conditional cash transfers have emerged as a primary tool against poverty across developing countries, effectively increasing school attendance and reducing poverty levels.

• Evidence-based policies have led governments and international donors to allocate resources more effectively towards proven strategies for poverty alleviation.

Challenges and Future Directions

• Recent political changes in Mexico have led to the elimination of effective programs, raising concerns about negative impacts similar to those seen during implementation phases.

• There is an ongoing debate regarding whether investments should focus on public policy or fostering economic growth as a means of addressing persistent vulnerabilities among populations.

Impact of Economic Growth on Poverty and Public Policy

Overview of Global Poverty Trends

• The speaker discusses the complexity of poverty measurement, asserting that global poverty has decreased when considering broader metrics beyond monetary factors.

• A significant reduction in extreme poverty is noted, with over a third of the global population previously living under the poverty line, now reduced to less than 10%.

• Improvements in nutrition and life expectancy are highlighted as indicators supporting the argument that poverty has declined.

Measurement Challenges

• The speaker acknowledges challenges in measuring relative poverty, emphasizing that comparisons with neighbors can skew perceptions of improvement.

• There are limits to what can be achieved through economic growth alone; addressing multidimensional goals requires resources and strategic planning.

Balancing Growth and Redistribution

• The ideal approach involves simultaneous economic growth and wealth redistribution to alleviate poverty effectively.

• Economic growth does not inherently resolve inequality; ongoing public policy efforts are necessary for resource redistribution.

Evaluation of Public Policies

• Criticism is directed at higher education funding versus basic education investment, suggesting a misallocation of resources based on political pressures rather than evidence-based decisions.

• The speaker argues that lack of political will, rather than absence of evidence, hinders effective climate change policies.

Role of Academia in Policy Development

• Emphasizes the need for transparency in research findings while recognizing political pressures influencing policy decisions.

• Discusses how public support is crucial for implementing educational reforms and suggests strategies for garnering broader backing from stakeholders.

Impact Assessment Challenges

• Highlights difficulties faced by policymakers regarding impact assessments due to high costs associated with evaluations compared to program implementation.

• Stresses the importance of using evaluation data effectively to enhance learning outcomes across various programs.

Evaluating Impact: Challenges and Solutions

Importance of Training in Public Policy Evaluation

• The speaker emphasizes the need to train individuals involved in public policy, highlighting that better dissemination of impact evaluation results is crucial.

• Organizations often seek evaluations for their programs without considering existing similar evaluations; educating them on leveraging past findings is essential.

• There’s a significant gap in understanding among donors and governments regarding evidence-based practices, leading to redundant program funding.

Issues with Program Naming and Recognition

• The speaker discusses how programs frequently change names (e.g., "Progresa" to "Oportunidades"), which complicates recognition and continuity for beneficiaries.

• There's an acknowledgment that while political reasons may drive program changes, combining effective evaluation processes can lead to more efficient outcomes.

Role of Evidence in Policy Making

• The justification for impact evaluations arises when ethical considerations are met, or when there is insufficient learning from previous inquiries into similar questions.

• The importance of presenting scientific evidence over personal opinions is stressed; the speaker advocates for a unified approach across political lines based on empirical data.