Choose a program area to view renewal information:  

I. Early Childhood and Child Abuse Prevention Programs 
II. After School and Mentoring Programs   

 

Early Childhood Programs and Child Abuse Prevention & Intervention Programs

Introduction

All of the Requests for Investment published by the Children's Investment Fund state that funding could be granted for a specified period of time with the possibility for renewal or extension for 12, 24 or 36 months. During the summer of 2005, 29 grantees' contracts were eligible for renewal. These 29 contracts included 18 early childhood grantees and 11 child abuse prevention and intervention grantees. The total amount of investment that was eligible for renewal was approximately $4.9 million annually. All 29 grantees' contracts were renewed after careful review by staff and by decision of the Allocation Committee. The renewal process used by the Children's Investment Fund and the major findings and results of the renewal process are explained below.

Renewal Process

The following requirements were established for current grantees seeking renewal of funding for early childhood programs and child abuse prevention and intervention programs:

1. Eligibility: Renewal was available only for current grantees seeking funding for the same or similar activities.
2. Contract Compliance: Staff was charged with assessing and reporting grantees' compliance with contract terms including serving the number of children they agreed to serve, providing the program as stated in their scope of services, and complying with all other contract requirements.
   
3. Site Visits: All grantees were required to host a site visit and staff was charged with designing a rubric to assess program practices and implementation during the site visit.
   
4. Audits: All grantees were required to submit an audit for a time period that included at least nine months during which they received Children's Investment Fund funds.
5. Renewal Application: All grantees were required to submit a renewal application that included a program description, program outcomes, program budget and budget narrative and any proposed program changes. Grantees were permitted to request annual funding at the same level as their current year two budgets with annual increases of up to 3 percent (compounded annually) to cover the costs of inflation.

Results and Findings

1. Contract Compliance: As staff has reported to the committee on a quarterly basis, all grantees are generally in compliance with contract requirements. Specifically, most programs are serving the numbers of children they are committed to serve. They have all produced files on children served (or other evidence) to demonstrate that children are receiving services. Where numbers are lower or higher than projected, programs have had reasonable explanations. In all cases, numbers served were commensurate with program spending rates. Some programs served fewer children than projected but they also spent fewer funds. In some cases programs had higher turnover among children in the program so their service numbers appear to be higher than projected. In other cases, programs had full family caseloads but families had fewer children than the program projected, so service numbers appear low.
2. Site Visit Assessment: Staff performed site visits with all 29 early childhood and child abuse prevention and intervention program grantees during April - June 2005. Site visits were approximately two and a half to three hours long and included interviews with program staff, review of program files and documents, and observations of program activities. Children's Investment Fund staff assessed program performance using a rubric defining best practice standards of program implementation in five areas:
• Client Recruitment and Outreach
• Client Intake/Enrollment
• Client Files and Client Data
• Program Effectiveness and Fidelity to Model
• Program Staffing.
Based on results of site visit, program performance was rated on a scale of poor, satisfactory, or excellent for each of the five areas and programs earned an overall assessment of failing to meet, meeting or exceeding standards. Six early childhood programs exceeded program standards, and the remaining 13 programs met standards. No programs fail to meet standards. Two child abuse prevention/intervention programs exceeded program standards, and the remaining nine programs met standards. No programs fail to meet standards.

3. Submission of Recent Audit: Staff reviewed each of the audits submitted and did not find any issues of note or concern regarding the financial health of grantee organizations.
4. Renewal Application: All grantees submitted renewal applications. Applications and renewal requests were summarized per grantee and presented to the Allocation Committee for review and approval. In cases where grantees requested funding above the ceiling established in the Renewal Application, staff recommended a funding amount that met the requirements. The Allocation Committee considered each application and renewal request individually before approving to renew.
5. Staff Recommendation: Based on the results of the analyses described above, staff recommended that the Committee renew all of its investments in Early Childhood and in Child Abuse Prevention and Intervention in the amounts stated in the Renewal Requests and Performance Assessments document presented to the Allocation Committee.

Themes and Observations

• Distinction between meeting and exceeding Program Standards comes down to good organizational systems that include evaluation and reporting on program-level outcomes. Programs that exceed standards have and use clear systems for client intake, client assessment, tracking client progress, regularly aggregating client outcomes to report and measure program effectiveness, using evaluation results to perform ongoing quality improvement of program, and maintaining competent staffing.

• Most program evaluation data is captured only in anecdotal evidence, such as case notes made by program staff. Few programs have systematic ways of aggregating and reporting child outcomes in order to evaluate their services and demonstrate their effectiveness. Most programs work with children and families on "family/child goals" and then they make case files and progress notes on individual child/families' work toward goals. Few programs talk about achievement of those goals in aggregate terms relating to the program's performance or effectiveness. New contracts will require that all grantees report aggregate data on program participation and program-level outcomes.

• Programs could use technical assistance, as well as staff, technology and time for creating and improving their program evaluation methods and systems. Although grantees have new evaluation requirements after going through renewal, they will still have many program evaluation capacity challenges. Only a handful of programs have extensive program evaluation capacity and regularly track, analyze and report on their program delivery and results. The remaining programs all have regular data that they track on a case by case basis but they nearly all lack a system for regularly aggregating, analyzing and reporting those data. As a result, most of the programs evaluate their program effectiveness based on client satisfaction surveys and some anecdotal understanding of the program. With small investments in evaluation capacity building, and small investments in staff and systems to implement evaluation methods, programs could produce more robust, interesting and helpful data to understand their program delivery better and to demonstrate their programs' effectiveness.

• Staff turnover increases hiring, training and management costs for programs. Staff turnover rates varied among all programs; some programs had no turn over while other programs experienced turnover in 50 percent or more of program staff. On average, 40 percent of programs had at least one position turnover during the past 18 months. Staff turnover rates varied among all programs, but turnover was much lower on average among child abuse prevention/intervention programs as compared to early childhood programs. There were no obvious trends as to why some programs had lower or higher staff turnover rates than others. In general, staff turnover occurred because employees took other jobs, advanced within the organization/program, returned to school, or left to care for family. When turnover occurred, programs had challenges finding staff with early childhood qualifications and experience. In addition, some programs struggled to find bilingual or culturally-specific staff with early childhood qualifications and experience.
  

• Many of the services funded by Children's Investment Fund complement each other. The early childhood and child abuse prevention and intervention programs funded by Children's Investment Fund fall into two main categories: 1) Core services, particularly in child development and early education, that are classroom-based or home visit-based; and 2) Focused programs that address particular needs such as early childhood mental health, culturally-specific services, HIV/AIDS services, general health information and referral, special needs and mitigation of abuse and neglect.
  

For example, five core programs benefit from the mental health consultants funded by Children's Investment Fund; Cascade AIDS Project Kids Connection does targeted outreach and training with the majority of core service providers. Similarly, Portland Niños has provided Latino children and families at three core programs with health information and referral resources in Spanish. In addition, the child abuse prevention and intervention programs had some relationship with the early childhood programs. Home visit early childhood programs had connections to the relief nurseries. Other connections could exist, and Children's Investment Fund staff can work in the future to help more of these programs collaborate.

Renewal of After-School and Mentoring Programs

All after-school and mentoring programs that received funding beginning in July 2004 were eligible to apply for renewal of grant funds. The information below describes the renewal process, the overall results and findings, themes and observations, and a detailed summary of past performance and renewal request for each grantee.

Renewal Process

As you may recall, the following requirements were established for current grantees seeking renewal of funding for after-school and mentoring programs:

1. Eligibility: Renewal was available only for current grantees seeking funding for the same or similar activities.

1. Contract Compliance: Staff was charged with assessing and reporting grantees' compliance with contract terms including serving the number of children they agreed to serve, providing the program as stated in their scope of services, whether they were able to collect data on outcomes and if so, whether they met outcome targets, and complying with all other contract requirements.

1. Site Visits: All grantees were required to host a site visit during which staff conducted an extensive interview to assess program practices and implementation. Staff also completed program observations for all programs except for the one-on-one mentoring programs.

1. Financial Assessment: CHIF retained McDonald Jacobs, the firm that conducts the annual audit of CHIF, to review a quarterly expense report for each grantee to assure that the grantee could produce supporting records for all expenses claimed. A copy of the report is attached to this memorandum for your information.

Grantees were also required to submit an audit. Audits were reviewed using the audit tool designed by McDonald Jacobs to get a sense of the financial status of the organization applying for renewal. Individual results of the tool application to audits are in the Performance Assessment for each grantee.

1. Renewal Application: All grantees were required to submit a renewal application that included a program description, proposed program outcomes, program budget and budget narrative and any proposed program changes. Grantees were permitted to request annual funding at the same level as their current year 2 budgets with annual increases of up to 3% (compounded annually) to cover the costs of inflation.

Results and Findings

1. Contract Compliance: All grantees are generally complying with contract requirements and delivering the programs they agreed to deliver. Many grantees did not meet service goals in Year 1 of their contracts, primarily due to start-up issues such as hiring and training staff, and recruiting participants to new programs.

2. Site Visit Assessment: Staff performed site visits with all 20 grantees during January - March 2006. Site visits were approximately 2.5 to 3 hours long and included interviews with program staff, review of program files and documents, and observations of program activities. CHIF staff assessed program performance using a rubric defining best practice standards of program implementation in five areas:

• Client Recruitment and Outreach
• Client Intake/Enrollment

• Client Files and Client Data
• Program Effectiveness and Fidelity to Model
• Program Staffing.

Based on results of site visit, program performance was rated on a scale of poor, satisfactory, or excellent for each of the five areas and programs earned an overall assessment of failing to meet, meeting or exceeding standards. Ten after-school and mentoring programs exceeded program standards in three or more of the categories listed above, and the remaining ten programs met standards. No programs fail to meet standards.

3. Submission of Recent Audit: With the exception of Bridge Builders, all grantees submitted audits. Audits were reviewed using the audit tool and individual results are listed under the review of each program. Most organizations are in acceptable financial health. The most common issues of concern raised by the tool were low reserves, high debt to asset ratios, and low operating reliance which measures the ability to pay expenses with program revenue. These concerns are common in the non-profit world where organizational budgets may fluctuate significantly from year to year as grants come and go and public funds are cut.

4. Renewal Application: All current grantees submitted renewal applications. Staff reviewed the applications and relevant information regarding the applications is included in the individual program reviews appended to this memorandum. In cases where grantees requested funding above the ceiling established in the Renewal Application, staff has recommended a funding amount that meets this requirement.

5. Staff Recommendation: Based on the results of the analyses described above, staff recommends that the Committee renew all of its investments in after-school and mentoring programs in the amounts and on the terms stated in the individual Performance Assessments accompanying this memorandum.

Themes and Observations

1. Impact of Expansion and New Program Funding

Grantees' service and budget projections for the first year of operations for new or substantially expanded programs proved ambitious in many cases. Hiring and training staff, staff turnover, and successfully recruiting students at schools where a program was new proved challenging. Fifty-five percent of programs (11/20) served at least 90% of the children they projected to serve. Fifty percent of programs spent at least 90% of their Year 1 program budget. Interestingly, programs may have spent all budgeted funds, but not served the projected number of children due to lower than expected student participation and fixed staff costs to deliver a program. In the future, CHIF staff will encourage budgeting and service projection that takes into account a ramp-up period.

2. Data Collection and Outcome Information

All after-school and mentoring program contracts included outcome targets. Most grantees agreed to track outcomes for attendance, behavior and academics. Some grantees also agreed to track other outcomes including outcomes related to developmental assets. Five grantees were unable to provide data on any of the outcome targets set in their contracts, and seven grantees were only able to obtain data on one or some of the outcomes targets set in their contracts. In some cases, this was because they could not obtain the relevant data on enough of the participants in the program to generalize results. Sometimes grantees were simply unsuccessful in getting data from the schools or from parents despite working to do so. In one case, PPS refused to release the data to the individual program because there were too few participants in the program and confidentiality could be compromised.

Our new requirement that all grantees submit participant lists to the school districts in which they are working will assure that data can be gathered and outcomes analyzed on consistent data points across programs. We may need to work to devise alternate outcomes and data collection methods for programs that do not serve enough participants annually to allow districts to release data. Finally, staff will need to assure at the front end that grantees specify measurement methods and timeframes for any outcome targets they set for which the school districts will not collect data.

3. Organizational Development Needs

While all programs at least met program standards, many of the smaller organizations we fund could use resources to develop and/or solidify organizational processes and procedures for carrying out their work. The most universal need is for data systems and staff time to take information that is gathered on an individual client basis and aggregate it so that programs can use that data to inform program decisions.

4. Site Visit Interview Tool

CHIF staff may need to revise the site visit interview tool and the program implementation standards to better suit activity-based after-school programs. The tool includes some implementation standards that are more applicable to full spectrum social services rather than more limited interventions such as after-school music or art classes.