The BAS Assessor Reliability Training provides an in-depth analysis of the items and quality indicators in the Business Administration Scale. This training is designed for individuals who want to ensure that the BAS assessments are valid, reliable, and administered consistently across programs. The training concludes with a reliability test and participants who are 85% or more reliable with the national anchors are eligible to apply for BAS Certification for an additional fee. Presented by: Robyn Kelton, Director of Research and Evaluation, Paula Steffen, Manager of Quality Supports and Evaluation, Yvonne Williams, Quality Training Specialist, and Isabel Landa, Quality Training Specialist
Business Administration Scale (BAS) Assessor Certification is valid for two years and may be renewed through a recertification process. Recertification is also valid for a two year period and can be renewed once. This registration is for your FIRST recertification (2 years after your initial certification).
The PAS 3E Assessor Reliability Training provides an in-depth analysis of the items and quality indicators in the Program Administration Scale, 3rd Edition. This training is designed for individuals who want to ensure that PAS assessments are valid, reliable, and administered consistently across programs. The training concludes with a reliability test and participants who are 85% or more reliable with the national anchors are eligible to apply for PAS-3 Certification for an additional fee. Presented by: Paula Steffen, Manager of Quality Supports and Evaluation, Yvonne Williams, Quality Training Specialist, and Isabel Landa, Quality Training Specialist
Leaders, policymakers, and systems developers seek to improve early childhood programs through data-driven decision-making. Data can be useful for informing continuous quality improvement efforts at the classroom and program level and for creating support for workforce development at the system level. Early childhood program leaders use assessments to help them understand their programs’ strengths and to draw attention to where supports are needed. Assessment data is particularly useful in understanding the complexity of organizational climate and the organizational conditions that lead to successful outcomes for children and families. Several tools are available for program leaders to assess organizational structures, processes, and workplace conditions, including: Preschool Program Quality Assessment (PQA) 1 Program Administration Scale (PAS) 2 Child Care Worker Job Stress Inventory (ECWJSI) 3 Early Childhood Job Satisfaction Survey (ECJSS) 4 Early Childhood Work Environment Survey (ECWES) 5 Supportive Environmental Quality Underlying Adult Learning (SEQUAL) 6 The Early Education Essentials is a recently developed tool to examine program conditions that affect early childhood education instructional and emotional quality. It is patterned after the Five Essentials Framework, 7 which is widely used to measure instructional supports in K-12 schools. The Early Education Essentials measures six dimensions of quality in early childhood programs: Effective instructional leaders Collaborative teachers Supportive environment Ambitious instruction Involved families Parent voice A recently published validation study for the Early Education Essentials 8 demonstrates that it is a valid and reliable instrument that can be used to assess early childhood programs to improve teaching and learning outcomes. METHODOLOGY For this validation study, two sets of surveys were administered in one Midwestern city; one for teachers/staff in early childhood settings and one for parents/guardians of preschool-aged children. A stratified random sampling method was used to select sites with an oversampling for the percentage of children who spoke Spanish. The teacher surveys included 164 items within 26 scales and were made available online for a three-month period in the public schools. In community-based sites, data collectors administered the surveys to staff. Data collectors also administered the parent surveys in all sites. The parent survey was shorter, with 54 items within nine scales. Rasch analyses was used to combine items into scales. In addition to the surveys, administrative data were analyzed regarding school attendance. Classroom observational assessments were performed to measure teacher-child interactions. The Classroom Assessment Scoring System TM (CLASS) 9 was used to assess the interactions. Early Education Essentials surveys were analyzed from 81 early childhood program sites (41 school-based programs and 40 community-based programs), serving 3- and 4-year old children. Only publicly funded programs (e.g., state-funded preschool and/or Head Start) were included in the study. The average enrollment for the programs was 109 (sd = 64); 91% of the children were from minority backgrounds; and 38% came from non-English speaking homes. Of the 746 teacher surveys collected, 451 (61%) were from school-based sites and 294 (39%) were from community-based sites. There were 2,464 parent surveys collected (59% school; 41% community). About one-third of the parent surveys were conducted in Spanish. Data were analyzed to determine reliability, internal validity, group differences, and sensitivity across sites. Child outcome results were used to examine if positive scores on the surveys were related to desirable outcomes for children (attendance and teacher-child interactions). Hierarchical linear modeling (HLM) was used to compute average site-level CLASS scores to account for the shared variance among classrooms within the same school. Exploratory factor analysis was performed to group the scales. RESULTS The surveys performed well in the measurement characteristics of scale reliability, internal validity, differential item functioning, and sensitivity across sites . Reliability was measured for 25 scales with Rasch Person Reliability scores ranging from .73 to .92; with only two scales falling below the preferred .80 threshold. The Rasch analysis also provided assessment of internal validity showing that 97% of the items fell in an acceptable range of >0.7 to <1.3 (infit mean squares). The Teacher/Staff survey could detect differences across sites, however the Parent Survey was less effective in detecting differences across sites. Differential item functioning (DIF) was used to compare if individual responses differed for school- versus community-based settings and primary language (English versus Spanish speakers). Results showed that 18 scales had no or only one large DIF on the Teacher/Staff Survey related to setting. There were no large DIFs found related to setting on the Parent Survey and only one scale that had more than one large DIF related to primary language. The authors decided to leave the large DIF items in the scale because the number of large DIFs were minimal and they fit well with the various groups. The factor analysis aligned closely with the five essentials in the K-12 model . However, researchers also identified a sixth factor—parent voice—which factored differently from involved families on the Parent Survey. Therefore, the Early Education Essentials have an additional dimension in contrast to the K-12 Five Essentials Framework. Outcomes related to CLASS scores were found for two of the six essential supports . Positive associations were found for Effective Instructional Leaders and Collaborative Teachers and all three of the CLASS domains (Emotional Support, Classroom Organization, and Instructional Support). Significant associations with CLASS scores were not found for the Supportive Environment, Involved Families, or Parent Voice essentials. Ambitious Instruction was not associated with any of the three domains of the CLASS scores. Table 1. HLM Coefficients Relating Essential Scores to CLASS Scores (Model 1) shows the results of the analysis showing these associations. Outcomes related to student attendance were found for four of the six essential supports . Effective Instructional Leaders, Collaborative Teachers, Supportive Environment, and Involved Families were positively associated with student attendance. Ambitious Instruction and Parent Voice were not found to be associated with student attendance. The authors are continuing to examine and improve the tool to better measure developmentally appropriate instruction and to adapt the Parent Survey so that it will perform across sites. There are a few limitations to this study that should be considered. Since the research is based on correlations, the direction of the relationship between factors and organizational conditions is not evident. It is unknown whether the Early Education Essentials survey is detecting factors that affect outcomes (e.g., engaged families or positive teacher-child interactions) or whether the organizational conditions predict these outcomes. This study was limited to one large city and a specific set of early childhood education settings. It has not been tested with early childhood centers that do not receive Head Start or state pre-K funding. DISCUSSION The Early Education Essentials survey expands the capacity of early childhood program leaders, policymakers, systems developers, and researchers to assess organizational conditions that specifically affect instructional quality. It is likely to be a useful tool for administrators seeking to evaluate the effects of their pedagogical leadership—one of the three domains of whole leadership. 10 When used with additional measures to assess whole leadership—administrative leadership, leadership essentials, as well as pedagogical leadership—stakeholders will be able to understand the organizational conditions and supports that positively impact child and family outcomes. Many quality initiatives focus on assessment at the classroom level, but examining quality with a wider lens at the site level expands the opportunity for sustainable change and improvement. The availability of valid and reliable instruments to assess the organizational structures, processes, and conditions within early childhood programs is necessary for data-driven improvement of programs as well as systems development and applied research. Findings from this validation study confirm that strong instructional leadership and teacher collaboration are good predictors of effective teaching and learning practices, evidenced in supportive teacher-child interactions and student attendance. 11 This evidence is an important contribution to the growing body of knowledge to inform embedded continuous quality improvement efforts. It also suggests that leadership to support teacher collaboration like professional learning communities (PLCs) and communities of practice (CoPs) may have an effect on outcomes for children. This study raises questions for future research. The addition of the “parent voice” essential support should be further explored. If parent voice is an essential support why was it not related to CLASS scores or student attendance? With the introduction of the Early Education Essentials survey to the existing battery of program assessment tools (PQA, PAS, ECWJSI, ECWES, ECJSS and SEQUAL), a concurrent validity study is needed to determine how these tools are related and how they can best be used to examine early childhood leadership from a whole leadership perspective. ENDNOTES 1 High/Scope Educational Research Foundation, 2003 2 Talan & Bloom, 2011 3 Curbow, Spratt, Ungaretti, McDonnell, & Breckler, 2000 4 Bloom, 2016 5 Bloom, 2016 6 Whitebook & Ryan, 2012 7 Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010 8 Ehrlich, Pacchiano, Stein, Wagner, Park, Frank, et al., 2018 9 Pianta, La Paro, & Hamre, 2008 10 Abel, Talan, & Masterson, 2017 11 Bloom, 2016; Lower & Cassidy, 2007 REFERENCES Abel, M. B., Talan, T. N., & Masterson, M. (2017, Jan/Feb). Whole leadership: A framework for early childhood programs. Exchange(19460406), 39(233), 22-25. Bloom, P. J. (2016). Measuring work attitudes in early childhood settings: Technical manual for the Early Childhood Job Satisfaction Survey (ECJSS) and the Early Childhood Work Environment Survey (ECWES), (3rd ed.). Lake Forest, IL: New Horizons. Bryk, A. S., Sebring, P. B., Allensworth, E., Luppescu, S., & Easton, J. Q. (2010). Organizing schools for improvement: Lessons from Chicago. Chicago, IL: The University of Chicago Press. Curbow, B., Spratt, K., Ungaretti, A., McDonnell, K., & Breckler, S. (2000). Development of the Child Care Worker Job Stress Inventory. Early Childhood Research Quarterly, 15, 515-536. DOI: 10.1016/S0885-2006(01)00068-0 Ehrlich, S. B., Pacchiano, D., Stein, A. G., Wagner, M. R., Park, S., Frank, E., et al., (in press). Early Education Essentials: Validation of a new survey tool of early education organizational conditions. Early Education and Development. High/Scope Educational Research Foundation (2003). Preschool Program Quality Assessment, 2nd Edition (PQA) administration manual. Ypsilanti, MI: High/Scope Press. Lower, J. K. & Cassidy, D. J. (2007). Child care work environments: The relationship with learning environments. Journal of Research in Childhood Education, 22(2), 189-204. DOI: 10.1080/02568540709594621 Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System (CLASS). Baltimore, MD: Paul H. Brookes Publishing Co. Talan, T. N., & Bloom, P. J. (2011). Program Administration Scale: Measuring early childhood leadership and management (2 nd ed.). New York, NY: Teachers College Press. Whitebook, M., & Ryan, S. (2012). Supportive Environmental Quality Underlying Adult Learning (SEQUAL). Berkeley, CA: Center for the Study of Child Care Employment, University of California.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
This document may be printed, photocopied, and disseminated freely with attribution. All content is the property of the McCormick Center for Early Childhood Leadership.
Early educational interventions, such as Head Start, have been widely recognized as an effective way to mitigate the negative effects of poverty on early learning and development (Camilli, Vargas, Ryan, & Barnett, 2010). In the past decade, there has been a strong expansion of early childhood programming, including Head Start and state-funded prekindergarten programs. However, the cost of the programs calls into question the extent to which this expansion can be maintained. A tension exists between serving as many children as possible and providing the most impact with limited economic resources (e.g., Barnett & Hustedt, 2011; Steuerle, Reynolds, & Carasso, 2007), making the study of program design such as length and intensity of programming critical to efforts to serve low-income or at-risk children in the most efficient fashion. The field knows little about the specific program design factors that lead to favorable program outcomes (Reynolds, 2004), and very few studies have addressed this issue. Therefore, we reviewed two frequently cited studies that looked at Head Start programs, the nation’s largest early educational intervention, and examined the impact of one program factor, intervention dosage, on children’s school readiness outcomes. In the two studies, program dosage was defined as the amount of services children received, and was measured as the duration of program enrollment (i.e., one year versus two years) and the intensity of the program (i.e., half day versus full day). ONE-YEAR VS. TWO-YEAR HEAD START Method . Using a nationally representative sample of Head Start children, Wen and her colleagues (Wen, Leow, Hahs-Vaugh, Korfmacher, & Marcus, 2012) examined school readiness outcome differences by the end of kindergarten between children who attended Head Start program for two years and those who attended for one year. This research question sounds simple, but is hard to address. It is challenging to make a causal conclusion regarding whether children and families who experience a longer duration of intervention would perform better on measured program outcomes than those who are enrolled for a relatively shorter time, because participants who experienced different amount of intervention may differ in other ways as well, including their demographic characteristics (Hill, Brooks-Gunn, & Waldfogel, 2003; Powell, 2005). Simply stratifying participants by intervention duration or estimating the impact of duration in a standard regression model will not typically yield unbiased estimates because sample selection bias might be operating. Therefore, the researchers adopted a rigorous statistical methodology, propensity score analysis, to match Head Start one-year versus two-year program children on 28 family background variables, so that the impact of demographic differences on child outcomes can be largely controlled for, and therefore, the researchers can draw a precise conclusion on how different program duration would lead to different outcomes. This methodology is innovative in addressing the causal relationship when the research design of randomizing children into programs of different durations is almost impossible. These demographics used to find similar comparison groups of children encompassed a comprehensive list of variables identified in the early development and education literature that are associated with child development and learning, including (a) child characteristics (e.g., ethnicity, health status, whether they had diagnosed disabilities, whether they were dual language learners); (b) family characteristics (e.g., parent education, employment status, family income, family size, marital status, parent age, maternal depression, welfare status, parental health status, and home language); (c) parenting styles (e.g., parental warmth) and parent involvement with child (e.g., frequency of reading, weekly and monthly activities with child); (d) child’s initial receptive language skills at the beginning of Head Start; (e) child’s prior intervention experience (i.e., Early Head Start); and (f) the amount of Head Start services the child and family received (i.e., frequency of missing Head Start and parent participation with program activities). The sample consisted of 1,778 children from 63 Head Start programs, 175 centers, and 337 classrooms. Forty-seven percent of children were three-year olds who attended the program for two years and the rest were four-year-olds who attended the program for one year, and 49% were boys. The study examined six academic and social outcomes assessed by the end of children’s kindergarten year: receptive vocabulary skills (PPVT test), emergent literacy skills (Woodcock Johnson III letter-word identification and word attack tasks), mathematic skills (Woodcock Johnson III applied problems and quantitative concept tasks), academic skills (teacher rating on 5-point scale), learning behaviors (e.g., reluctant to tackle new activity; cries when faced with difficulty), and social competence. These measures represent a broad definition of school performance that goes beyond the narrow focus of academic-related skills. Results . Before children with different lengths of program attendance were matched on their baseline characteristics, the outcome comparison yielded significant differences on only two Woodcock-Johnson subtests (literacy skill and math reasoning), favoring the two-year program children (see statistics in Table 1). However, after children were matched on their demographic characteristics, the researchers found that among the five matched comparison groups, children in two-year Head Start performed significantly better than those who attended the program only for one year on all six outcome measures, with decent effect sizes (Table 2). The findings convey a strong and clear conclusion that more, rather than fewer years of Head Start would accrue greater program outcomes. HALF-DAY VS. FULL-DAY HEAD START In a different research study, Leow & Wen (2016) examined another Head Start dosage variable, the program intensity (i.e., half-day versus full-day), and its impact on child outcomes. Similarly, the study involved a Head Start national sample and adopted the same methodology, propensity score analysis, to match children in full-day and half-day programs on various demographic backgrounds before comparing the effects of program intensity. The method would allow the researchers to draw precise causal conclusions on how program intensity predicts child outcomes by controlling for other potential factors. In reality, it is almost impossible to randomly assign children to programs with different dosage intensity to test the effects because it is unethical to deny services to eligible children, especially for public service programs. The advanced methodology of the study helped to address a critical question that has significant policy implications. Method . The sample included 2,097 children who were newly enrolled in Head Start in the fall of 2006. They were from 135 Head Start centers and 410 classrooms, of which 61% were three-year olds and the rest were four-year olds. The three-year old children were eligible to stay in the program for two years, while the four-year olds were enrolled for one year before they transitioned into kindergarten. About 51% of children were enrolled in the half-day program. This study assessed five child outcomes related to cognitive skills (PPVT and Woodcock-Johnson III letter-word identification subscale), preschool learning behaviors, and social skills. The demographic variables used to match full-day and half-day program children were even more extensive than the Wen et al. (2012) study. A total of 45 demographic variables collected from initial parent interviews were used in the propensity score matching. Results . The analyses were performed separately with two different age cohorts – the three-year olds who stayed in the program for two years and four-year olds who stayed in the program for one year. The results showed that in comparison to a demographically comparable group of children who attended the Head Start half-day program, children who experienced more intensive full-day intervention services showed no significant differences on any of the five academic and social outcome measures, and this was true whether children attended the program for one year or two years (Table 3). Discussion . Given limited resources, how should we design the most optimal Head Start and state-funded early childhood education programs that would maximize their impact on children’s school readiness? There is a recent national push to expand state-funded prekindergarten programs to enhance school-related academic skills and social-behavioral competence (Howes et al., 2008). Statistics show that these state-funded programs mainly recruit four-year old children who would be eligible to stay in the programs for only one year before transitioning into kindergarten (Barnett, Hustedt, Robin, & Schulman, 2005). However, the Head Start one-year and two-year comparison study (Wen et al., 2012) clearly suggests that that public preschool programs should target children as early as possible and keep them in the programs for a longer period of time in order to maximize the educational benefit for these vulnerable children. This study provides strong policy justifications for public funding for early education for a minimum of two years. However, the finding regarding the association between program intensity and child outcomes is contradictory to our hypothesis, and to some extent, it is surprising. Hypothetically, we would hope that full-day preschool programs offer children more opportunities for child-centered creative activities and free play, as well as more opportunities for socialization with peers. But instead of making the policy recommendation that Head Start should drop the full-day model and offer only the half-day model to serve more children, the authors think the study actually raises the question of how to promote Head Start program quality, so that the full capacity of this public early intervention program can be fulfilled. Also, instead of answering the question of whether full-day and half-day models make a difference in child outcomes, the study brings up more research questions that need to be addressed. For example, it is unclear how the combination of program intensity and duration would impact program outcomes. Would one-year, full-day programs be similar to two-year, half-day programs? Secondly, program quality needs to be taken into account. Both quality and quantity of Head Start intervention matter in shaping low-income children’s development. Future research should also address the interaction between program quality and quantity and the association with program outcomes. References Barnett, W. S., & Hustedt, J. T. (2011). Improving public financing for early learning programs. Preschool Policy Brief, 23. Retrieved from http://nieer.org/resources/policybriefs/24.pdf . Barnett, W. S., Hustedt, J. T., Robin, K. B., & Schulman, K. L. (2005). The state of preschool: 2004 preschool yearbook. New Brunswick, NJ: NIEER. Camilli, G., Vargas, S., Ryan, S., & Barnett, S. W. (2010). Meta-analysis of the effects of early education interventions on cognitive and social development. Teachers College Record, 112(3), 579-620. Hill, J. L., Brooks-Gunn, J., & Waldfogel, J. (2003). Sustained effects of high participation in an early intervention for low-birth-weight premature infants. Developmental Psychology, 39, 730-744. Howes, C., Burchinal, M., Pianta, R., Bryant, D., Early, D., Clifford, R., & Barbarin, O. (2008). Ready to learn? Children’s pre-academic achievement in pre-kindergarten programs. Early Childhood Research Quarterly, 23, 27-50. Leow, C., & Wen, X. (2016). Is full day better than half day? A propensity score analysis of the association between Head Start program intensity and children’s school performance in kindergarten. Early Education and Development, 28(2), 224-239. Powell, D. R. (2005). Searches for what works in parenting interventions. In T. Luster & L. Okagaki (Eds.), Parenting: Ecological perspectives (pp. 343-373). Mahwah, NJ: Erlbaum. Reynolds, A. J. (2004). Research on early childhood interventions in the confirmatory mode. Children and Youth Services Review, 26, 15-38. Steuerle, C. E., Reynolds, G., & Carasso, C. (2007). Investing in children. Washington, DC: The Partnership for America’s Economic Success. Wen, X., Leow, C., Hahs-Vaugh, D. L., Korfmacher, J., & Marcus, S. M. (2012). Are two years better than one year? A propensity score analysis of the impact of Head Start program duration on children’s school performance in kindergarten. Early Childhood Research Quarterly, 27, 684-694.
