DOCUMENT RESUME FL 800 626 ED 355 837 AUTHOR Hemphill, David F. Workplace ESL Literacy in Diverse Small BusinesE TITLE Contexts: Final Evaluation Report on Project EXCEL. Career Resources Development Center, Inc., San INSTITUTION Francisco, CA. Office of Vocational and Adult Education (ED), SPONS AGINCY Washington, DC. National Workplace Literacy Program. 'PUB DATE 92 CONTRACT V198A10293 57p.; For related documents, see FL 800 627-631. NOTE Evaluative/Feasibility (142) PUB TYPE Reports Tests /Evaluation Instruments (160) EDRS PRICE MF01/PC03 Plus Postage. DESCRIPTORS Basic Skills; Communications; Curriculum; Employee Attitudes; Employer Attitudes; Employer Employee Relationship; *English (Second Language); Fashion Industry; Food Service; *Job Performance; Language Skills; *Limited English Speaking; *Literacy Education; Productivity; Program Evaluation; Questionnaires; *Small Businesses; Work Environment - California (San Francisco); Mail Room Occupations; IDENTIFIERS *Project EXCEL CA; *Workplace Literacy ABSTRACT Project EXCEL, a workplace literacy project involving four small business enterprises in San Francisco, is evaluated. The project focused on literacy and basic skills training for limited-English-proficient (LEP) workers. The businesses included the following: a communications and mass mailing firm; a dessert wholesale company; a Mexican restaurant and food product manufacturer; and an outerwear design and contract manufacturer. Three general areas comprised the evaluation: the extent to which the literacy abilities of the workforce participants improved; the extent to which their productivity improved; and the relationships between literacy gains, productivity gains, and worker advancement. Logistics of operating an English-as-a-Second-Language (ESL) oriented workplace literacy program in a series of diverse small business settings were also assessed. Evaluation instruments and procedures were developed and employed across the worksites. Conclusions are as follows: there was a need for this program; the program accomplished its overall goals; learners perceived substantial gains in language/literacy and some gains in productivity; employers perceived substantial gains in language/literacy and productivity, and saw a relationship between the two; and worker-learners demonstrated measurable and substantial gains in general language/literacy skills. Evaluation instruments and objectives of each curriculum module are included. (Adjunct ERIC Clearinghouse on Literacy Education) (LB) Workplace ESL Literacy in Diverse Small Business Contexts: Final Evaluation Report on PROJECT EXCEL a project funded by the U.S. Department of Education under the National Workplace Literacy Program and operated by CRDC Career Resources Development Center, Inc. 655 Geary St, San Francisco, CA 94102 Phone: (415) 775-8880 FAX: (415) 775-1748 "PERMISSION TO REPRODUCE THIS MAIFELAL HAS BEEN GRANTED BY U.S. DEPANTIVENT OF EDUCATION Office of Educations' Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER /ERIC? his document has been reproduced as received from the anum of otganathon originating it Minor changes have been made to improve reproduCtiOn Quality TO THE EDUCATIONAL RESOURCES Points of view or opinions stated in fhis docu- INFORMATION CENTER (ERIC)." ment do not necessarily represent official OERI position or policy External Evaluator Professor David F. Hemphill Center for Adult Education, San Francisco State University (N.) 0 Fall, 1992 2 Ll BEST COPY AVAILABLE TABLE OF CONTENTS Introduction 1 Evaluation Design & Procedures 2 Findings 6 Conclusions 17 Appendix (Copies of Instrumentation & Curriculum Descriptions) 3 1 I. INTRODUCTION This report documents the findings of an effort to evaluate Project Excel, a recently-completed workplace literacy project operated by CRDC (Career Resources Development Center), a community-based employment and training organization in San- Francisco, over approximately the past 15 months. Project Excel sought to establish a series of partnerships with four small business enterprises in San Francisco. The project focused on literacy and basic skills training for limited English proficient (LEP) workers, since immigrants and refugees comprise a large proportion of the entry-level workforce in the San Francisco labor market area. The goals and objectives of the project as stated in the original proposal were to: 1. Improve the productivity, efficiency, and cost effectiveness of the LEP workforce. 2. Enable LEP workers to achieve added job security, enhanced prospects of job mobility, and greater job satisfaction. 3. Enhance the ability of employers to understand and interact more effectively with their multicultural workforce. 4. Demonstrate that communication and literacy skills can be improved through training grounded in functional content that is relevant to the workplace. 5. Establish and disseminate a replicable program and curriculum model of collaborative effort in addressing the communication and literacy skill needs of a multicultural, LEP workforce. Project EXCEL is somewhat unique among the workplace literacy projects funded by the U. S. Department of education for two reasons: (1) it specifically targets limited-English-proficient (LEP) workers; and (2) it is working primarily with small businesses, one of the primary sources of jobs 4 2 in the U.S. economy. The small businesses involved in Project EXCEL included Ace Mailing (a communications and marketing firm), Just Desserts (a dessert producing plant with wholesale and retail outlets), Casa Sanchez (a Mexican food production plant), and Momentum (an outerwear design and contract manufacturer). Due to the first of these unique features (the LEP worker focus), the evaluation plan for Project EXCEL had to take into account the fact that the literacy skills to be acquired by workers, and the workforce productivity gains to be achieved as a result of the workplace literacy program, needed to be addressed in terms of second language acquisition issues and cross-cultural communication issues, as well as the more commonly-construed conceptualizations of workplace literacy for native speakers of English. Due to the second of these unique project features (the small business focus), the evaluation plan had to be to some extent worksite- specific, so that the specific workforce literacy and workforce productivity needs of each individual small business site engaged in Project EXCEL could be addressed. II. EVALUATIONDESIGN AND PROCEDURES The Project EXCEL evaluation plan initially addressed the three following general evaluation questions: 1. To what extent have the literacy abilities of the workforce participants improved? 2. To what extent has the productivity of the workforce participants improved? 3. What relationships do there appear to be between literacy gains, productivity gains, and worker advancement? It should be noted that these evaluation questions (particularly Questions One and Two) owe substantially to the useful guidance provided to workplace 5 3 literacy project directors in the recent monograph by Thomas G. Sticht ("Evaluating National Workplace Literacy Programs," April, 1991). The third question builds upon the first two in an attempt to synthesize and correlate in a meaningful way to the lives of the workers the results gathered through the investigation of the first two questions. In addition to these questions, as the final phases of the evaluation were under way, it became apparent that an evaluation question regarding the logistics of operating an ESL-oriented workplace literacy program in a series of diverse small business settings needed to be added. Therefore, data were also gathered with respect to this latter issue. The flow chart on the next page summarizes the initial concept of the evaluation plan. This graphic representation depicts the starting point of the evaluation effort. As noted above, the evaluation process subsequently underwent some modification due to the demands of the multiple-site, small-business nature of the workplace literacy program being evaluated. 6 4 Project EXCEL Evaluation Procedure s Evaluation Question 1: "To what extent Evaluation Question 2: "To what extent have the literacy abilities of the work- has the productivity of the workforce force participants Improved?" participants Improved?" AProcedures for Evaluation Question 1 Prooeckres for Evaluation Question 2 22.A 'Coker 2.1A Worker 2.3A Worker 1.1.A Criterion- 1.2.A Standardized 2.4.A Superviso Language/Literacy Referenced Productivity Interviews Produu ivity Interviews Language Skill Skill Tests for LEP Supervisor Self-Assessment PRE- PRE- PRE-TESTS learners (BEST Test Checklist Checklist TRAINING TRAINING (different forms or CASAS Adult PRE-TEST PRE-TEST for each curricu- Lifeskill Listening Test) lum module and worksite) PRE-TEST 12.B Standardized 2.2.B Worker 2.1.B Worker 1.1.B Criterion- 2.3.B Worker 2.4.B Superviso Referenced Language/Literacy Productivity Productivity Interviews Interviews Supervisor Skill Tests for LEP Language Skill Self-Assessment POST- POST- POST-TESTS learners (BEST Test Checklist Checklist TRAINING TRAINING or CASAS Listening POST-TEST POST-TEST Test) POST-TEST Evaluation Question 3: "What relationships do there appear to be between literacy gains, productivity gains, and individual worker advancement?" Procedures for Evaluation Question A Collection of individual wdrer job rank and 3.1 pay data at the start of each prow PRE-TRAINING 3.1.B Correlation of gains in jobank and pay for individual workers with indhival literacy and productivity gains measured her Questions 1 and 2 above POST-TRAINING 1. 7 5 Ultimately the following evaluation instruments and procedures (see Appendix for copies) were developed and employed across the worksites served by Project Excel: A. LEP Worker Interview Protocol: used in initial stages of project to gather data on workplace literacy and language use demands, to feed into the curriculum development process. B. Worker Productivity Self-Assessment Checklist (Form WP): a self- report device to assess individual worker perceptions of work-related literacy skills needs and development (used as pre-post in some sites, as post-only in others). C. Supervisor's Worker Productivity Assessment Checklist (Form SWP): a supervisor's report device on individual workers covering the same items as Form WP above (used as a post-device, and to enhance validity of claims of gain). D. Employer Program Evaluation Summary (Form ES): a summative individual written survey device used to gather management perceptions of overall program efficacy. E. Employer Program Evaluation Summary Interview Form (Form ESI): a summative individual interview schedule used to gather management perceptions of overall program efficacy. F. Worker Program Evaluation Summary Group Interview Form (Form WSI): a summative group interview schedule used to gather worker perceptions of overall program efficacy. G. Standardized Skill Assessment Instruments: forms of BEST test instruments and CASAS instruments were used as some sites, some as post-tests, and some as pre-post tests. 3 6 H. Teacher-Made Skill Assessment Instruments: post-tests constructed by teachers to reflect specific curriculum content developed at each worksite for each group of worker learners. I. Group and Individual Interviews with Program Staff: open-ended group and individual interviews were conducted with all program staff members. J. Attendance and Related Program Documentation: these forms of data which were routinely gathered as a part of program operation were also analyzed for evaluation purposes. III. FINDINGS This section begins to report on evaluation findings. These findings are organized according to the major data sources that were accessible during the evaluation process, with the thrust of the analysis focusing on the project's four basic evaluation question areas (noted earlier). A. Participant Profiles This section presents and interprets demographic and needs assessment information gathered regarding project participants. Figure 1 on the following page reports data on workers who were participants in classes at each of the four sites during August, 1991. These data offer an instructive snapshot of the nature of the program's participants and points out some important educational needs of the population. Close to two-thirds (65%) of the workers were women, and their ages were fairly equally distributed among the ages of 20-30 (35%), 30-40 (25%), and over 40 (32%). Mexico and Latin America (combined 51%) and China or other Chinese-speaking areas (37%) were the two main areas of cultural and linguistic background of the 9 7 Project Excel Worker Participant Profiles (Enrolled during August, 1991) Worksites Totals tlamnlun Just Desserts Item Descriptions Ace Mailing Casa Sanchez Number Per Cent NO. OF WORKERS 34 52 26 30 142 100% Men 0 2 5 49 8 35% Women 34 50 22 21 123 65% AGE Under 20 8 4 0 0 8% 12 20-30 30 7 6 49 6 35% 30-40 4 13 2 35 16 25% 24 Over 40 0 46 14 8 32% NATIONAL ORIGIN Mexican 0 6 28 52 37% 18 Other Latin American 4 4 0 12 20 14% Oinese 52 0 0 0 52 37% Filipino 0 0 16 0 16 11% Other 0 2 0 0 2 1% EDUCATION IN HOME COUNTRY None 15 0 0 0 15 11% Elementary 15 10 45 10 10 32% High School 22 20 20 73 51% 11 College 4 0 5 0 9 6% EDUCATION IN U.S. High School 0 0 2 0 2 2% Community College 6 28 47 8 5 33% 24 None 28 25 93 16 65% YEARS IN U.S. Under 5 years 34 20 16 5 53% 75 5-10 years 20 0 38 5 13 27% Over 10 years 0 12 5 12 29 20% CURRENT WAGE 0 0 Minimum Wage Not 16 16 14% Min. Wage to $5/Hr. 52 5 Available 68 61% 11 $5-$8/Hr. 0 23 5 28 25% Figure 1 project's worker participants. Just over 50% had a high school education, while 32% had an elementary education, and 6% reported no prior education. Most (65%) reported that they had no education in the United States, while about a third (33%) had received some community college training. The 10