|
This document summarizes the Florida Tech Prep Evaluation process, including goals and objectives, characteristics, the continuous quality improvement model, and more. The document is below. Contact us to obtain a print copy.
|
Beginning January 1, 1993, Florida International University was granted a project entitled Performance- Based Project for the Development of a Florida Statewide Plan for Evaluating Tech Prep Programs. This project's activities included the planning, development, and implementation of a statewide plan to evaluate the activities of Florida's 28 Tech Prep consortia. Since the project's inception, major strides have been achieved in the design, development, and implementation of this plan. The key features of the plan are:
The State of Florida is in the process of a major educational reform effort using a plan called Blueprint 2000. Blueprint 2000 offers many education models to implement the process of reform, including magnet schools, centers of academic emphasis, academies (schools within schools), and the Tech Prep initiative. To comply with the requirements of its guiding legislation (the Carl D. Perkins Vocational and Applied Technology Education Act of 1990) and to be an effective agent of change, each of Florida's Tech Prep consortia must exhibit common philosophic components.
The cornerstone of this educational reform effort is the modification and implementation of an integrated and articulated curriculum in each consortium. An integrated curriculum infuses theoretical and applied concepts in math, science, communications, and technology with vocational/occupational course work, providing Tech Prep students with a relevant, concrete learning experience. The curriculum is then delivered through a rigorous, applied, relevant-student-manipulated and student-centered-teaching methodology.
The integrated curriculum must be articulated with an accredited postsecondary educational institution, apprenticeship training program, and/or work-based certificate granting agency. Additionally, it must contain the course work necessary to complete two years of education beyond high school. The curriculum for each vocational/occupation program and/or cluster offered by the Tech Prep consortia must also be defined in a written format called a "program of study." These programs of study serve as a guide for students, parents, counselors, business and industry partners, labor representatives, teachers, and administrators to explain the minimum course work required to complete a vocational or occupational specialty.
The Tech Prep educational reform effort also requires that partnerships be developed and strengthened between the consortium and business, industry, labor, educational agencies, parents, civic organizations, and any other group necessary for the support and promotion of the initiative within the consortium's locality. This includes solicitation of these partners for input on the Tech Prep curriculum and programs of study to insure that Tech Prep students are prepared to be competitive for not only today's careers but the careers of the future.
Tech Prep promotes accountability in schools. Each consortium must express, by means of marketing techniques, brochures, news releases, or oral presentations to various organizations, the objectives and expected outcomes of Tech Prep. This obligates the consortium to achieve the outcomes. This obligation is complemented by the development of an internal evaluation plan for each consortium that assists in the documentation and assessment of the quality, effectiveness, and achievement of its local Tech Prep program.
Tech Prep provides a number of advantages to students not offered in most education programs. These students are provided an avenue for developing lifelong learning skills to be competitive in the future workforce. Each Tech Prep graduate will have an edge in obtaining employment and beginning careers, due to the program's rigorous and relevant curriculum and involvement with business and industry. Business and industry participation in developing the curriculum and programs of study ensure that Tech Prep students are being prepared for the high-tech workforce of the future.
The overall goal of the Tech Prep evaluation process is to obtain and maintain the highest possible quality, effectiveness, and achievement of goals among the Tech Prep consortia in Florida. Figure 1 provides a graphic representation of the evaluation plan. In the center of the illustration is the Tech Prep student-the customer, ultimate product, and focus of all of Florida's Tech Prep activities. Florida's Tech Prep consortia coordinators developed the following definition of a Tech Prep student at a statewide meeting in fall 1992:
A student enrolled in an articulated, sequential program of study, *at grade level or above by grade 11 in mathematics, science, and communications, including a technical component, which leads to a minimum of a two-year postsecondary certificate or degree, and/or apprenticeship program. *Enrolled in level 2 or above courses.
This definition contains the criteria for those students involved in Florida's Tech Prep activities.
Also represented in this figure are the guidelines and activities of Blueprint 2000 that are integrated into Tech Prep. The strong internal evaluation activities within each of Florida's Tech Prep consortia to improve administrative, management, and educational operation of Tech Prep are also represented in the figure.
Finally, the Florida DOE is included in the illustration because it serves as the source of staff assistance, external evaluation, fiscal control, and data focal point for Florida Tech Prep.
The paramount aim of Florida's Tech Prep evaluation is to develop a design that determines and enhances the quality, effectiveness, and achievement of goals of each consortium and that includes strategies and action plans. An overview of the ancillary objectives of Florida's evaluation process is as follows:
The design of Florida's Tech Prep evaluation plan incorporates the following characteristics:
The intent of the Tech Prep initiative is to permanently embed Tech Prep programs into the existing educational structure. With this in mind, Tech Prep must be a total quality effort and must focus on obtaining the highest, continuous quality possible in a results-driven model. As is true for any successful business or industry, public education must operate on the premise of day-to-day as well as long-term improvement. By emulating the most successful businesses, public education programs can achieve this type of improvement through a systematic process that ensures quality. Thus, the evaluation model used for Florida Tech Prep programs is one that is well-recognized by business and industry and that has a proven track record in creating a continuum of quality.
The proposed statewide Tech Prep evaluation model incorporates a modified version of a Total Quality Management (TQM), continuous quality improvement model (Steeples, 1992). This model's format (please refer to Figure 2) is derived from a suggested example that may be utilized when applying for the Malcolm Baldridge National Quality Award. Within the next year, educational agencies (including public schools) can begin to compete for this prestigious national quality award.
The Florida Tech Prep evaluation model is a dynamic systems design consisting of three major operational component areas that stimulate the ongoing process for continuous quality improvement. The three component areas are:
Each component area shares equal importance; however, the amount of focus and energy expended on each component depends upon where each individual consortium is in its development and implementation process. A brief discussion of each component area follows.
Documentation. Documentation is the formal process of gathering vital information on the implementation and continuation of Tech Prep activities. Consortia will document (at least) the following outcomes:
1. The extent to which state and consortium proposed goals for Tech Prep activities are attained.
2. The extent of development and implementation of Tech Prep activities in the state and in individual consortia.
3. The results of an impact evaluation, including the completion of a Tech Prep self-study by each consortium, that is a comprehensive review of all the components required for the Tech Prep initiative. The self-study is followed by an on-site evaluation visit by an interdisciplinary team consisting of members from the Florida DOE. The site-visit team will examine the evidence documented in the self- study and will collectively determine:
4. An examination and assessment of the formative components of the consortium's activities including, but not limited to:
Analysis. The next component of the model consists of the analysis of the information and data collected during the documentation process. This component helps each consortium develop quantifiable goals and benchmarks for its activities, develop improvement plans and analyses of trends from the impact evaluation process; complete an in-depth analysis of each of the Tech Prep components; and identify best Tech Prep practices in the consortia to be shared, disseminated, and modeled with one another.
Improvement. The last major operational component for this model is the improvement process. This begins when the plans for targeted areas recommended for improvement are implemented and their progress is monitored to ensure positive change and impact. The last step in the improvement component is to begin the process again, building from the assessed results and data gathered from the prior evaluation process. The goal of Florida's Tech Prep evaluation process is to achieve superiority in all the Tech Prep component areas; this model provides a continuous, systematic process for determining the quality, effectiveness, and achievement of goals for Florida's statewide Tech Prep activities.
The 1990 Carl Perkins Legislation requires that states address five program performance measures. These include:
The federal legislation failed to note any standardized methodology for states to use in determining and reporting these data. To be in compliance with federal legislation, Florida began its evaluation effort by focusing on these five areas. However, provisions have been made for each Tech Prep consortium to add additional program performance measures as needs dictate. A committee composed of Tech Prep consortia coordinators developed five common data elements to aid in the documentation of the state Tech Prep movement initiative. These data elements were presented at a statewide consortia meeting on October 28, 1992, at which time all consortia members agreed to provide data for all five measures to the Florida DOE to document Tech Prep consortia activities. These identified data elements are:
Florida has continued to collect the identified benchmarks and outcomes to determine Tech Prep programs' quality, effectiveness, and achievement of goals. Data elements available from Florida's Education and Training Placement Information Program (FETPIP) and the Florida DOE's vocational database are also continually explored as part of the evaluation process.
The goal to be the best at all facets of the Tech Prep initiative has been a key to Florida's success. With over 960 Tech Prep consortia across the United States-most within their first few years of operation-the possibilities and excitement of professional, collaborative, and competitive learning experiences exist. In Florida, several Tech Prep consortia have received national exemplary awards from both the American Association of Community Colleges and the U.S. Department of Education for being the outstanding Tech Prep consortia. For a Tech Prep consortium to reach these levels, comparative techniques must be used to strengthen all consortia's operations. Sharing experiences with each other allows everyone to grow and enhance each consortium's efforts, thus all the players become winners-students, parents, businesses, industries, schools, and communities.
This project's director has been actively involved in the review, revision, and dissemination of the U.S. Department of Education's development of the national evaluation project for Tech Prep. Mathematica Policy Research, Inc., at Princeton, New Jersey, was awarded the contract for this effort. The national evaluation plan includes a local Tech Prep consortium survey, a state Tech Prep directors' survey, and in-depth studies. The local survey has been conducted each fall since 1993 and will be continued until 1997. The state directors' survey will be conducted twice: fall 1993 and fall 1996. In-depth studies of Tech Prep projects are being conducted nationally at 10 consortia, one of which is Florida's Alachua/Bradford/Santa Fe consortium. Efforts are being extended to help Florida's consortia comply with and respond to the national evaluation survey. Every possible step has been taken to eliminate any duplication of data collection between the national Tech Prep survey and Florida's statewide evaluation activities.
The site visit process began in spring 1993, and visits were conducted at each of the initial 11 Tech Prep consortia during their second year of operation. The process has evolved into three distinctive types of site visits: initial, follow-up, and continuing consortium visits. To date, a total of 38 site visits have been conducted.
Initial site visits have been conducted at each Tech Prep consortium at the end of their second year of planning and implementation. At the conclusion of the spring 1993 initial site visits, it was apparent that a majority of the consortia were having difficulty with the development and implementation of many of the components necessary for Tech Prep success. Therefore, a follow-up site visit was developed and implemented in spring 1994.
The follow-up site visit determined the extent to which each consortium had addressed the cited concerns from the initial site visit and the effectiveness of the recommendations addressing each cited concern. The continuing site visits began in spring 1995 to document the progress of those Tech Prep consortia who had not participated in a site visit within a two-year period. As Tech Prep evolves, continuing site visits will assist Florida's Tech Prep consortia in their continuous quality improvement process.
The objectives of the site visits are as follows:
1. Document the extent to which each of Florida's Tech Prep consortia is attaining its stated goals/objectives as proposed to the Florida DOE.
2. Observe and document positive examples of secondary and postsecondary Tech Prep initiative activities including, but not limited to, the following components:
3. Collect information for determining the profiles and characteristics of each of Florida's Tech Prep consortia, including the number of students served, accomplishments, results, and progress made toward achieving the required key elements as stated within federal legislation for Tech Prep.
4. Collect data to determine the impact of Tech Prep by identifying measurable benchmarks and parameters and, where possible, using existing data sources for future programmatic comparison.
5. Collect information to determine Florida's role in supporting its Tech Prep consortia and providing technical assistance needs for each.
6. Recommend actions for the improvement of the consortia's Tech Prep activities.
The external teams visiting each consortium consist of an interdisciplinary group that may include Florida's Tech Prep coordinator, Blueprint-for-Career-Preparation liaison, state Tech Prep evaluation liaison, a statewide community college representative, a school improvement facilitation representative, an academic area program director, and a statewide occupational program director. Also participating in the visits have been statewide Tech Prep advisory members and other state-level academic and vocational administrators when available.
Prior to each visit, the consortium responds to a self-report package, called a "site self-study," that asks questions concerning each of the focus areas necessary in developing and implementing Tech Prep. The self-study also asks for numbers of students, programs, and courses related to the Tech Prep initiative. Evidence of documentation and materials development to support Tech Prep are also requested. The documentation, evidence of implementation, and reports for each Tech Prep site visit conducted are developed and coordinated through the Florida DOE.
These reports consist of the findings, documentation and evidence, observations, and input from the interdisciplinary site-visitation team. They also document demonstrated best practices, strengths, areas in need of improvement, and recommendations. The completed site visit reports are disseminated to the appropriate Tech Prep consortium administration, including the school district superinten-dent(s) and community college president. All site-visit reports, documentation, and evidence collected during the site-visit process are stored as archival documents at the Florida School-to-Work Clearinghouse located at Educational Services Program, Florida State University, Tallahassee, Florida.
Research and activities continue to determine what data Florida's Tech Prep consortia should be required to collect or add to administer and manage their programs. The Florida DOE's Management Information System (MIS) Database and FETPIP are the focal points for this research. Direct data links among Tech Prep consortia and these two entities are being investigated and, where appropriate, developed. The objective of these linkages is to make timely and relevant information from the two information systems available to Tech Prep consortia for their administration and management.
To be successful, the Florida Tech Prep initiative must develop and implement a strong internal continuous quality improvement process within each consortium. A review of the literature has revealed a void in systematic and quality program evaluation efforts at the local Tech Prep consortium level throughout the nation that is also confirmed by numerous local consortia site visits and discussions with national, state, and local administrators of Tech Prep.
The consensus among those trying to address this evaluation need is that those responsible for implementing local Tech Prep program evaluation efforts lack the necessary primary skills. Key consortia members need training in evaluation design and implementation. Annual statewide evaluation workshops will continue to help consortia members establish a strong internal evaluation component and gain necessary skills. Consultants whose expertise is in both Tech Prep and evaluation will continue to deliver hands-on, relevant training experiences to Florida's Tech Prep administrators, coordinators, and stakeholders.
The major assertion of this evaluation design is that quality in public schools and educational institutions is only achievable if a formal, systematic, ongoing evaluation process is in place. This process should document those measures necessary to continuously improve and ensure programmatic outcomes and results. In Florida, the Tech Prep initiative can best be described as a "total quality effort." It has been created by many dedicated and hard-working persons who are using a continuous quality improvement process to positively impact Florida's citizenry and workforce now and into the twenty-first century.
Steeples, M. M. (1992). The corporate guide to the Malcolm Baldridge National Quality Award: Proven strategies for building quality into your organization. Milwaukee, WI: ASQC Quality Press.
Developed by:
Frank T. Hammons, Ed.D., Director/Assistant Professor
Institute for Workforce Competitiveness &
School-to-Work Evaluation Liason Office
Florida International University EAS-2614
Miami, Florida 33199
(305) 348-6529 FAX (305) 348-6524
E-Mail: hammonsf@fiu.edu
Sponsored and Published by:
Florida School-to-Work Programs
Bureau of Applied Technology Programs and Services
Division of Applied Technology and Adult Education
Florida Department of Education
Frank T. Brogan, Commissioner
Produced by:
Florida School-to-Work Clearinghouse
Educational Services Program
Florida State University
Editor, Charlene H. Cappellini
Visual Designer, Gary Carroll
Editorial Assistant, Alice Fisher
Word Processing Specialist, Arthur Copeland
Web Specialist, Tammy V. Dover
For Additional Copies, Contact
IWC or:
Florida Department of Education
Division of Applied Technology and Adult Education
Product Distribution
243-D Collins Building
Tallahassee, FL 32399-0400
(800) 342-9271
(904) 488-0400
c State of Florida, Department of State, 1995
Send comments to IWC at xiwc@fiu.edu
Copyright © 1999 by Institute for Workforce Competitiveness.
Revised: July
27, 1999.
