Program Evaluation

The following is the External Evaluation of the CALIFORNIA ACADEMIC PARTNERSHIP PROGRAM, Final Evaluation 1993-96 Program Cycle Executive Summary of October, 1996 for the program cycle of 1993-96.

This external evaluation was submitted by (ETI) Evaluation and Training Institute, 12300 Wilshire Boulevard, Suite 420, Los Angeles, California 90025

I. EXECUTIVE SUMMARY

The California Academic Partnership (CAPP) was created in 1984 to develop "cooperative efforts to improve the academic quality of public secondary schools with the objective of improving the preparation of all students for college." The program places a special emphasis on assisting students from groups which have traditionally been underrepresented in higher education. In order to achieve this objective, CAPP establishes partnerships among secondary schools (grade 6-12), postsecondary institutions and business/industry. These parties work together to provide curricular and instructional reform, student support services and world-of-work experiences at selected school sites.

The Evaluation and Training Institute (ETI) has conducted the statewide evaluation of the California Academic Partnership Program (CAPP) since 1987. This report presents the outcomes of our evaluation of the CAPP projects operating during the 1993-96 funding cycle. In addition, this report includes the outcomes of a survey of a sample of CAPP projects funded prior to 1993 to assess the long-term impacts of CAPP upon schools and students.

In assessing the value of CAPP, we believe that it is important to examine the program within the total environment for educational reform. Since the inception of CAPP over 12 years ago, the push for reform in all aspects of education has mushroomed. In addition to the types of curricular and instructional reform promoted through CAPP, structural reform is being implemented on a massive scale, with local school sites and communities exercising increasing autonomy in trying to meet the unique needs of diverse student populations. On the basis of our nine year involvement with CAPP, ETI staff believe that the program mad a significant contribution to the advancement of reform within California. Additionally, CAPP's 12-year history provides a rich base of knowledge which can, and should, be used to improve future reform efforts.

Specific findings of ETI's most recent evaluation include the following:

Expansion of the Use of Educational Technology

Although the national reform effort has stimulated the development and dissemination of educational technology, many teachers have resisted implementing technological strategies in the classroom because they have not been properly trained in how to use them. In CAPP, however, we found that teachers received appropriated training and specific opportunities to integrate technology into their instruction. CAPP enabled teachers and their students to become comfortable with technology, and to identify new ways in which it could be used to enhance learning: For this reason, we believe that CAPP provides many valuable lessons for how best to integrate technology into the classroom. In particular:

• We attribute CAPP's contribution to the expansion of educational technology to the prolonged staff development which CAPP funding made possible for teachers. This extended period of support allowed teachers to become comfortable with using technology in the classroom and to identify alternative ways for using it in instruction.

• Many teachers and students were provided access to the Internet as a result of their school site's involvement in CAPP. At some sites, this access allowed students to establish relationships with underrepresented students currently enrolled in college who could provide one-on-one support and encouragement to those interested in a postsecondary education. Additionally, access to the Internet provided students in rural areas and poor neighborhoods with entrance into whole new worlds.

Advancement of Student Assessment

The lack of assessment instruments which measure the skills and knowledge which students need to enter and succeed in postsecondary programs has recently been identified as a major barrier to improving student achievement by the California Education Round Table. during the 1980's, CAPP involved both secondary and postsecondary faculty in the development of instruments which could be used to measure student's readiness for college preparatory courses at the secondary level. The wide use of the instruments developed through the Mathematics Diagnostic Testing Project (MDTP) substantiate the value of these. ETI found that:

• During the 1995-96 academic year, MDTP tests were used by a total of 4,140 math teachers, approximately 36 percent of all secondary math teachers in the state. These teachers taught in 911 secondary schools, or approximately 47 percent of all secondary schools in California. a total of 346,400 MDTP tests were submitted from these sites for scoring by MDTP staff. Since multiple tests may be submitted for an individual student, it is not possible to report the percentage of secondary students in California whose achievement in mathematics has been assessed through the MDTP.

• MDTP has responded to the emergence of educational technology by providing two versions of each test, one of which requires the use of calculators. Additionally, tests now include written response items that provide a more comprehensive assessment of students' mathematical understanding and communication skills, in addition to the knowledge of mathematical procedures measured by more traditional tests.

Public and Private Sector Collaboration

CAPP is predicated upon the notion that the achievement of all students can be enhanced by bringing to bear the combined resources of secondary, postsecondary and private sector organizations. ETI's evaluation has shown CAPP to be an exemplary model of collaboration between secondary and postsecondary educational institutions. The program has been less successful, however, in fostering partnerships between education and the private sector. The pattern of private sector involvement in most local CAPP projects was not found to be as extensive or prolonged as that observed for postsecondary involvement. Specific findings include the following:

• In both our evaluation of 1993-96 CAPP grantees and review of projects funded prior to 1993, ETI found that CAPP significantly expanded the involvement of postsecondary faculty in the development of secondary curriculum, as well as in the provision of staff development and teacher mentoring. Additionally, with few exceptions, the expansion of postsecondary institution's involvement in local secondary schools persisted beyond the termination of CAPP funding. In some cases, additional types of funding were found to support these collaborative efforts, but in many others the postsecondary institutions continued to contribute their resources on a pro bono basis.

• Although the pattern of private sector involvement in local projects was not as extensive in all cased as that observed for postsecondary involvement, we found two exemplary models of private sector/public sector collaboration among the six CAPP grantees receiving funding during 1993-96. At these two sites, the private industry partners made significant contributions to the implementation of reform: Radionics, Inc. played a major role in the development of a mathematics curriculum for middle school students at Washington Middle School in the Salinas Unified School District which uses the company's production process as a foundation for theme-based instruction. At Chipman Middle School in the Alameda Unified School District, Arthur Andersen staff were instrumental in bringing additional financial and technological resources to support the curriculum reform of the mathematics faculty involved in CAPP.

• ETI believes, however, that the potential contribution of the private sector partners at the other four site was limited by: I} a lack of understanding of the educational challenges presented by underrepresented groups of students on the part of professional and management employees from the private sector; and 2} a failure to fully explore the resources which the private industry partner could provide, and to reach agreement on which resources would be of most value to the project. Moreover, our survey or pre-1993 projects suggests that the collaboration of business/industry partners is also less likely than that of postsecondary participants to endure after CAPP funding ends. Only two of the 13 projects included in our survey reported a continuing private-sector partnership following the termination of the local CAPP project.

Student Achievement

The ultimate objective of CAPP is to enroll higher numbers of traditionally underrepresented students in college preparatory courses and to ensure their success in these courses so that they qualify to enter postsecondary programs. Without a doubt, during its 12 year history CAPP has succeeded in enrolling large numbers of underrepresented students in college preparatory coursed who would not otherwise have enrolled. Within the 1993-96 program cycle alone we estimate that 8,400 students undertook such more challenging study. to our knowledge, no other program with California has accomplished this goal on such a large scale.

Beyond the impact of CAPP upon students' enrollment in college preparatory courses, ETI found that an assessment of the impact of local CAPP projects upon student achievement was limited by two major factors: 1} the lack of standardized instruments geared toward the curricular reforms which local CAPP projects had implemented; and 2} the inability of local project sites to provide adequate data on a consistent basis. We were able to obtain two years of data on the grades and standardized test scores of CAPP students and specific control groups from four of the six 1993-96 project sites. Additionally, in our survey of projects funded prior to 1993, we asked several questions related to the impact of the program upon student achievement and the availability of data to make such an assessment. Analyses of these data revealed:

• For the 1993-96 program cycle, ETI's analyses of the data we were able to obtain from local project sites revealed no statistically significant differences in grades of standardized test scores between CAPP students and their respective control groups in the first two years of the funding period.

• No statistically significant differences were observed between CAPP students and their respective control groups in their desire to attend college; awareness of the importance of math; enjoyment of math; parental expectations of academic achievement; students' perceptions of their own abilities.

• During our survey of 13 CAPP projects sites funded prior to 1993, three sites reported that student achievement had increased, two sites reported no change in achievement, and representatives of the remaining eight sites indicated that no data were available to make such a determination.

Recommendations for the Program Improvement

On the basis of our nine years of involvement in CAPP, ETI makes the following recommendations:

• The term of local curriculum projects should be increased from three to five years. We strongly believe that this longer timeline will enhance the institutionalization of local reforms and provide for a better assessment of program impacts.

• The true impact of CAPP funded curriculum reforms upon student achievement cannot be assessed without standardized instruments which reflect the skills and knowledge which students require to succeed in college preparatory courses and postsecondary programs. We strongly recommend that CAPP actively support the efforts of the California Education Round Table to develop a statewide assessment system which could provide such instruments.

• ETI views CAPP as an institutional reform affecting the school site, and not a support program targeting individual students. Given the lack of appropriate assessment interments, we believe that the limited amount of resources available for program evaluation could most effectively be used to examine program impact on such readily available school level measures as drop-out rates, the number of students enrolling in college preparatory programs, the number of students completing college entrance requirements and standardized test scores. These indicators should be monitored for a five year period to provide a more realistic assessment of program impact.

• If CAPP elects to persist in assessing the impact of curricula reforms upon individual students, we recommend that a written commitment to provide any and all data specified by the external evaluator signed by the district superintendent and the director of the district's data processing operations be a requirement of project funding. If appropriate data are not provided in a complete and timely fashion then CAPP should withhold project funding until such data are made available to the CAPP evaluator. CAPP must also be prepared to provide additional funding to local project sites to offset the costs districts incur in providing the necessary data. ETI's experiences indicate that these costs will vary greatly be site.

• Implementation of any reform is dependent upon the participation and involvement of the majority of teachers and administrators at the school site. Without broad based involvement and commitment, reform efforts falter of have limited results. We recommend that local project sites be required to provide written commitments signed by the majority of faculty and administrators at the site which indicate the staff's commitment to cooperation in the implementation of reform.

• High rates of student transiency severely impact both the implementation and impacts of local reforms. Unfortunately, many schools with high proportions of underrepresented students experience high rates of transiency. Given that CAPP targets this group, ETI does not believe it would be appropriate to avoid funding projects at such sites. We recommend, however, that expectations for program outcomes at schools characterized by high student turn-over be appropriately adjusted.

• Opportunities for private sector involvement must be more carefully explored and defined at the local CAPP project level. Innovative forms of involvement should be recognized and promoted by CAPP management. We believe that the use of technology within local projects would provide many opportunities for such business/industry involvement.