Institute for Research on Poverty
University of Wisconsin–Madison
November 21–22, 1996
The Joyce Foundation and the Charles Stewart Mott Foundation provided support for this project through a grant to the Institute for Research on Poverty. Opinions are those of the authors and not necessarily those of the sponsoring institutions.
List of Contributors.......................................................
iii
Introduction: Evaluating State Welfare Reforms
Thomas Kaplan..............................................................
1
The Next Generation of Welfare Reforms: An Assessment of the Evaluation Challenge
Thomas Corbett .............................................................
7
Potentials and Problems of a Pre-Post Design for State-Based Evaluation of
National Welfare Reform
Robert Haveman ........................................................... 25
Outcomes of Interest, Evaluation Constituencies, and the Necessary Trade-Offs
Maria Cancian and Barbara Wolfe ................................. 43
Controlled Experiments in Evaluating the New Welfare Programs
Glen G. Cain .................................................................. 61
Prospects for Comparing Wisconsin Works to Welfare Reform Programs outside
the State
Irving Piliavin and Mark Courtney ................................... 73
Toward a Basic Impact Evaluation of Wisconsin Works
Thomas Kaplan and Daniel R. Meyer .............................. 83
Wisconsin Works: A View from the Ground
Thomas Corbett and Elisabeth Boehnen ........................ 117
Process Evaluation of W-2: What It Is, Why It Is Useful, and How to Do It
Karen C. Holden and Arthur Reynolds .......................... 139
A Management Information System for Wisconsin Works
Michael Wiseman ......................................................... 157
Monitoring Income for Social and Economic Development
Martin H. David ...........................................................
183
Evaluating the Impacts of W-2 on Family Structure and Maternal and Child Health:
A Brief Background Paper
Gary Sandefur and Molly Martin ................................... 201
Evaluation of Child Care Services under the Wisconsin Works Program (W-2)
Karen Fox Folk ............................................................
211
Potential Effects of Wisconsin Welfare Reform (W-2) on Child Support
Stuart Kipnis .................................................................
223
Potential Effects of Wisconsin Welfare Reform (W-2) on Families Who Have Very
Young Children with Disabilities and Special Health Care Needs
George Jesien ...............................................................
231
Welfare Reform and Education
John F. Witte ................................................................
245
Welfare Reform and Child Welfare Services: Issues of Concern and Potential
Evaluation Strategies
Mark E. Courtney ......................................................... 253
Assessing the Impact of W-2 on Homelessness
Irving Piliavin .................................................................
267
Elisabeth Boehnen is Database Administrator, Institute for Research on
Poverty, University of Wisconsin–Madison
Glen Cain is Professor Emeritus, Department of Economics, University
of Wisconsin–Madison
Maria Cancian is Assistant Professor, School of Social Work and La Follette
Institute of Public Affairs, University of Wisconsin–Madison
Thomas Corbett is Acting Director, Institute for Research on Poverty,
and Assistant Professor, School of Social Work, University of Wisconsin–Madison
Mark Courtney is Assistant Professor, School of Social Work, University
of Wisconsin–Madison
Martin David is Professor of Economics, University of Wisconsin–Madison
Karen Fox Folk is Associate Professor, School of Human Ecology and La
Follette Institute of Public Affairs, University of Wisconsin–Madison
Robert Haveman is John Bascom Professor, Department of Economics, University
of Wisconsin–Madison
Karen Holden is Associate Director, La Follette Institute of Public
Affairs, and Professor, School of Human Ecology, University of Wisconsin–Madison
George Jesien is Director, Early Intervention Program, Waisman Center
on Mental Retardation and Human Development, University of Wisconsin–Madison
Thomas Kaplan is Associate Scientist, Institute for Research on Poverty,
University of Wisconsin–Madison
Stuart Kipnis is a Ph.D. candidate, School of Social Work, University
of Wisconsin–Madison
Molly Martin is a Ph.D. candidate, Department of Sociology, University
of Wisconsin–Madison
Daniel Meyer is Associate Professor, School of Social Work, University
of Wisconsin–Madison
Irving Piliavin is Professor Emeritus, School of Social Work, University
of Wisconsin–Madison
Arthur Reynolds is Assistant Professor, School of Social Work, University
of Wisconsin–Madison
Gary Sandefur is Professor, Department of Sociology, University of Wisconsin–Madison
Michael Wiseman is Professor, Department of Regional and Urban Affairs
and La Follette Institute of Public Affairs, University of Wisconsin–Madison
John Witte is Professor, Department of Political Science, University
of Wisconsin–Madison
Barbara Wolfe is Director (on leave), Institute for Research on Poverty,
and Professor, Department of Economics and Department of Preventive Medicine
(now Population Health Sciences), University of Wisconsin–Madison
Thomas Kaplan
In August 1996, Congress and the president replaced the 60-year-old Aid to Families with Dependent Children (AFDC) program with a block grant, Temporary Assistance for Needy Families (TANF), permitting states to experiment with new forms of assistance to low-income families. States seem likely to try increasingly ambitious reforms as they gain experience under their TANF flexibility, at least if they respond as they did to federal permission to waive AFDC requirements. When President Bush first announced the new AFDC waiver options, states sought approval for only one or two proposals at a time. In the last months of AFDC, however, state waivers grew much more comprehensive. North Carolina, for example, secured a waiver that would allow the state to
Thomas Corbett’s paper on “The Next Generation of Welfare Reforms” describes the nature of these reforms and some of their implications for evaluation. The comprehensiveness of such reforms and the potential for still broader change in the future create at least two challenges to evaluation. First, because components of a broad reform could be synergistic, the impact of a comprehensive reform may surpass the sum of the impact of each individual reform within it. If so, evaluators would understate the impact of a reform component in a particular context if they evaluate only the one component. Yet assessing a comprehensive reform in its entirety, especially if its elements change over time, can be harder than assessing components of it.
Second, comprehensive reforms create special difficulties for evaluations that utilize an experimental design, a standard way of determining cause and effect. The difficulties arise because comprehensive welfare reforms often bundle new requirements for school attendance, work, or a daily activity that leads to work, along with modified policies for adjusting family income according to family size, into packages designed to generate social messages that discourage dependency on public assistance, promote work, and influence family-formation decisions. It is possible that such messages would be less effectively communicated in an environment of random assignment to control and treatment groups than in a more universal program. It is also possible that the control group would be influenced—and thus rendered invalid as a true control—by the messages conveyed to the treatment group.
Researchers have increasingly remarked on these and other problems of evaluating comprehensive welfare reform. But reaching consensus on evaluating such programs has proved elusive. In November 1996, IRP held a conference on the evaluation of the Wisconsin welfare reform program (Wisconsin Works, or W-2), in an effort to use one state welfare reform plan as a laboratory for evaluation. Although Wisconsin is not the only state with comprehensive reform, W-2 is among the most detailed and ambitious of the state plans. (Its basic features are summarized in Table 1.) IRP affiliates and staff believed that the program offered a worthy test of our ability to plan an evaluation of comprehensive welfare reform.
Intensive IRP work on the evaluation of W-2 started in the summer of 1995 and was supported by financial assistance from the Joyce Foundation and the Charles Stuart Mott Foundation. These were some of the issues that researchers confronted:
The Distinction Between Monitoring and Evaluation
Monitoring involves tracking income, earnings, family composition, and
other social and economic indicators as a new program is created. Evaluation
goes beyond monitoring, to determine if a causal link exists between the reform
and any changes observed in the indicators. Experimental evaluation designs,
controlled so that the only difference between the experimental and control
group is the new program, offer the standard method of determining how much
the changed policies altered the relevant indicators. But if experimental
design cannot be used, will any other design allow for evaluation, or is close
monitoring of the indicators (a useful task in itself) all that will be possible?
Members of the IRP working group agreed that experimental design was inappropriate for evaluation of W-2, but differed over whether reasonable causal inferences could be made in the absence of experimental design. The majority maintained some optimism that at least limited causal attributions would be possible without experimental design. Comprehensive welfare reform, after all, is not the only policy intervention for which experimental evaluation designs are unsuitable. The effects on the economy of Federal Reserve Board policies, the deterrent effects on potential aggressors of new weapons systems, and the impacts of local government efforts to encourage economic development are also unknowable through formal experimental design. Yet useful causal knowledge can accumulate concerning these and similarly complex interventions.
Nonexperimental Counterfactuals
In an experimental design, the causal link between the intervention and
observed changes in indicators arises from the experimental structure: people
or families are randomly assigned to the new treatment and to the status quo.
If we have grounds for believing (1) that the people in each group did not
differ in relevant characteristics upon their entry into the program, (2)
that the treatment and control groups did not “contaminate” each other during
the experiment, and (3) that the two groups were equally affected by other
changes occurring at the same time, then any post-treatment differences between
the two groups in the relevant indicators can be attributed to the interventions.
But if experimental designs cannot be used in the new welfare reform evaluations, a different “counterfactual” is required if we are to have any sense of what would have happened without the reforms. IRP researchers discussed seven other counterfactual possibilities, which we later determined to fall broadly into two approaches: pre-post designs, in which post-treatment indicators are compared to pretreatment indicators, and cross-state designs, in which Wisconsin indicators are compared to indicators in other states with different interventions. The advantages and disadvantages of these approaches are discussed in papers by Robert Haveman, Glen Cain, and Irving Piliavin and Mark Courtney. The paper by Thomas Kaplan and Daniel R. Meyer, “Toward a Basic Impact Evaluation of Wisconsin Works,” proposes a particular form of pre-post design.
Key Indicators
Regardless of whether or how causality is established, the populations and
outcomes to be studied must be identified—issues discussed in the paper by
Maria Cancian and Barbara Wolfe on “Outcomes of Interest, Evaluation Constituencies,
and the Necessary Trade-Offs.” All members of the working group favored a
concentration on the broad low-income population—those with incomes below
200 percent of the federal poverty line—not just on participants in the W-2
program. The reason for looking beyond program participants is the possibility
that the program will have entry and exit effects —that is, the program itself
may induce some people to enter, avoid, or drop out of it, in which case its
full impact could not be inferred from observing only program participants.
There was less agreement on the possible outcomes of interest. Some members
of our group favored concentrating on earnings, hours worked, and receipt
of welfare. Others favored attention to a broader set of concerns, including
the effects on family-formation decisions, health status, participation in
health care programs, the incidence of child abuse and neglect, and the use
of foster homes and other out-of-home placements. All of us wanted to give
attention to “process indicators,” those indicators that would help program
managers judge the quality of implementation and make timely policy adjustments
based on early program experience. We did not reach consensus on what those
process indicators might be, although the separate papers by Thomas Corbett,
Michael Wiseman, and Karen Holden and Arthur Reynolds narrowed our differences.
Other Issues
We discussed several other issues at less length during the months leading
to production of the conference papers: how long the W-2 program should operate
before it was subjected to impact evaluation; the dependability of various
data sources; evolutions in the W-2 proposal itself; and whether—especially
for pre-post analysis—W-2 changes should be deemed to include major policy
modifications that the state of Wisconsin made within AFDC during 1996, or
just the establishment in 1997 of W-2 itself. The papers by Martin David,
Gary Sandefur and Molly Martin, Karen Fox Folk, Stuart Kipnis, George Jesien,
John F. Witte, Mark E. Courtney, and Irving Piliavin discuss these and other
issues in the context of assessing the potential impact of W-2 on incomes,
family structure, child care services, child support, families containing
young children with disabilities, education, child welfare services, and homelessness.
The full W-2 program is scheduled to start on September 1, 1997, but transitional parts of the program have already begun. The two most significant early implementation phases of W-2 have been in place since March 1996:
The two transitional programs will phase into the full W-2 program, which will eliminate AFDC and replace it with cash assistance available only through work or participation in work-like activities. Under the full W-2 program, participation requirements will begin when the youngest child is 12 weeks. Families will be assigned to a Financial and Employment Planner, who places families on a level of a “self-sufficiency ladder” and helps them move up the ladder to greater independence, as indicated in the grid in Table 1. Small loans, which can be paid back in cash or community service, will be available in order to assist participants to find and keep work. Under W-2, there is no provision for subsidized formal education. Although some W-2 recipients will technically receive a grant, all cash income in the program will be disbursed on an hourly basis in return for each hour of work or program participation. Failure to participate will reduce income. Unlike the current AFDC program, the income will not depend on family size, but only on the case head’s hours of participation and level on the W-2 self-sufficiency ladder. Also unlike AFDC, W-2 is not statutorily identified as an entitlement.
W-2 participants will also receive assistance with child care costs; this assistance will require copayments from recipients varying by recipient income and the licensing level of the care they select. All working families with incomes below 165 percent of the poverty line at program entry will be eligible for child care subsidies. Wisconsin has also requested but not at this writing received a federal waiver to replace the current Medicaid program for W-2 recipients with a W-2 medical program that would provide benefits generally similar to those available under Medicaid to a somewhat different population (some expansions and some contractions in eligibility) contingent on the payment of a premium. All W-2 services will be available to both single- and two-parent families with children under 18 and with incomes within 115 percent of the poverty line. W-2 will also eliminate the current practice under which child support income beyond the first $50 in a month goes to public agencies to reimburse welfare expenditures. W-2 participants will be able to keep all the child support paid on their behalf.
| Level of W-2 | Basic Income Package | Time Required of Recipients | Program Time Limits | Est. Child Care Copays ($/mo.) | |
|---|---|---|---|---|---|
| Licensed Care | Certified Care | ||||
| Unsubsidized employment | Market wage + Food Stamps + EITC | 40 hrs/wk standard | None | $101-$134 | $71-$92 |
| Trial Job (W-2 pays maximum of $300/mo. to the employer) | At least minimum wage + Food Stamps + EITC | 40 hrs/wk standard | Per job: 3 mo. with an option for one 3-mo. extension ; total 24 mo. | $55 | $38 |
| Community Service Job (CSJ) | $673 per mo. + Food Stamps (no EITC) | 30 hrs/wk standard; and up to 10 hrs/wk in education and training Total: 24 mo. |
Per job: 6 mo. with an option for one 3-mo. extension | $38 | $25 |
| W-2 Transition (placement contingent on an assessment by the state Vocational Rehab. agency) | $628 per mo. + Food Stamps (no EITC) | 28 hrs/wk work activites standard; and up to 12 hr/wk in education and training | 24-mo. limit, but extensions permited on a case-by-case basis | $38 | $25 |
Sources: Folk (1996) and presentation materials created by the Wisconsin Department of Workforce Development.
Notes: The income package and child care copayment are based on Governor Thompson’s proposals in the 1997–99 biennial budget. Estimated child care copayments are for a three-person family with two children receiving no child support payments. Department of Workforce Development materials express child care copayments on a weekly basis; the monthly copayments shown in the table assume 4.2 weeks per month. For the purpose of estimating child care copayments, the Trial Jobs position is assumed to pay minimum wage, which, after October 1, 1997, will be $5.15 per hour, or $858 per month, and the unsubsidized-employment portion is assumed to range from $6–$7 per hour, or $1,000–$1,170 per month.
NOTE: The complete text of Special Report 69 may be either: