A Program Evaluation of School-wide Positive Behavior Intervention & Support (PBIS)

in a Rural Middle School Setting

Program Evaluation MEDT 8480

Jamie Bennett, Valarie Gutierrez, Libra Royster-McGriff, Cati Rhodes

 Background of the Program & Evaluation

Richmond Hill Middle School is in its second year of using a PBIS program and

will be the focus of the program evaluation. The school consists of roughly 1,658

students and 127 teachers, with a demographic breakdown of 4% Asian, 14% African

American, 81% Caucasian, and 1% Latino. Since it is a framework, PBIS can be

altered to best fit each school that implements it. The program gives students specific

expected behaviors in various areas of the school even including the bus. In exchange

for students meeting the expected behaviors, they receive various types of rewards that

each school decides on. The Positive Behavior Intervention & Supports, or PBIS, is are

a systems-level framework used in schools and districts around the country to improve

school climate, reduce exclusionary discipline, and improve overall student

success.(Swain-Bradway, Pinkney, & Flannery, 2015) The programs purpose is to

attempt to find a positive and effective way to respond to students misbehaviors rather

than solely utilizing punishments as a way to correct negative behaviors. Using only

punishments such as reprimands, office referrals, and suspensions has proven to be

ineffective for the most part; therefore, PBIS focuses on teaching proper behaviors and

rewarding students when they display them as well. exchange for students meeting the

expected behaviors, they receive various types of rewards that each school decides on.

research shows that three themes which lead to the prevention of problematic behaviors

in schools. To begin with there must be a time and resources dedicated to describing

desired expectations. this must be followed with more time and resources devoted to

teaching expectations. There must also be a systematic process in which students are

recognized for the adherence to the PBIS behavior expectations. in addition to this
recognition there must also be some sort of reward for students who display the desired

behaviors as laid out by the schools PBIS program. Finally, there must be a plan in

place that will insure the expectations and rewards remain a consistent part of the

schools culture (Sprague & Horner, 2006).

Richmond Hill Middle School plans to address and strictly adhere to the 3 core

themes of a successful PBIS program as previously referenced. In order to make this

goal a reality the school will need to follow the seven key features of a successful PBIS

program, as described by the research of Horner, Todd, Lewis-Palmer, Irvin, Sugai, and

Boland in 2004. According to their findings a school must (a) describe expected

behaviors for all students; (b) make sure that students learn school wide behavior

expectations; (c) demonstrate approval for students who exhibit the appropriate

behaviors; (d) as protocol use a behavioral continuum to correct problematic behaviors;

(e) collect and store student data to use in school discipline decision making (f)

administration must create a team to carry out the objectives of the PBIS program and

make sure that they work to meet all of the demands of a successful PBIS program; (g)

finally there must be district support for the resources, and organization that will be

necessary in a PBIS program. The evaluation of Richmond Hill Middle Schools PBIS

program will focus on the schools ability to uphold the lofty standards of what makes a

successful PBIS program. To insure that Richmond Hill Middle School, the evaluation

team and all stakeholders are on the same page there will be monthly meetings

between these three groups. One of the focuses of these meetings will be to make sure

that all groups understand the values, purposes, and process of the evaluation process.

Stakeholders purpose and use of evaluation must also be identified, so that only the
most relevant information is gathered during the evaluation. These purposes need to be

established early and addressed each time the groups meet so that the evaluation can

serve the stakeholders in the most efficient way possible. More importantly each group

will gain an understanding of what the possible implications of a program evaluation

would mean for Richmond Hill Middle School and stakeholders. All possible uses of the

evaluation must be discussed at each meeting while also addressing any possible

unintentional consequences of such an evaluation plan. These meetings will also be an

opportunity to make sure that members of each group understand the process of the

evaluation and would also serve as a time to answer any questions about the progress

of the evaluation process. One final piece of information that must be addressed for all

stakeholders is the timeliness of reporting information to the stakeholders. Information

must be delivered not only incrementally but scheduled throughout the entire evaluation

process and must also be made available at the request of stakeholders. As information

is gathered stakeholders need to be able to rely on the timeliness and quality of

information that they are receiving to help them plan and make decisions that could

impact themselves, the evaluation or Richmond Hill Middle School.

Evaluation Team

The program evaluation team is made up of veteran teachers who have a combined 31

years of experience in education. Each member of the evaluation team has a masters

Degree in an educational field and is also in the process of completing their Educational

Specialist degree. Each member of the evaluation team also has varying degrees of

experience with the implementation and maintenance of a successful PBIS program at

their respective schools.

Evaluation Leader:

Jamie Bennett is a sixth grade social studies teacher in Bryan County. During his six

years of teaching he has taught language arts, social studies and is also an assistant

football coach. Jamie received his Bachelors degree in business administration from

Georgia Southern University and his Masters of Arts in Teaching from Armstrong Atlantic

University. Jamie is currently working to complete his Ed.S in instructional technology

from the University of West Georgia.

As the evaluation leader Jamies primary responsibility will be to make sure an

unbiased, data driven and collaborative program evaluation takes place.

Communicating the process and purpose of the evaluation with all staff of Richmond Hill

Middle SChool and stakeholders will also insure that there is a trusting and transparent

evaluation team and process. With additional responsibilities to include establishing

data collecting, usage and storage protocols. While also maintaining positive

relationships with all staff and stakeholders.

Evaluation Facilitator:

Cati Rhodes is a sixth grade gifted English Language Arts teacher at Lee Middle

School. In addition to her eight years of teaching experience, she is also a grade level

leader, a member of the leadership team, and is the schools event coordinator. Cati

earned her Bachelors degree in Middle Grades Education from Georgia College &

State University. Her Masters degree in Middle Grades Education from University of the

Cumberlands and is currently working to complete her Educational Specialist in

Instructional Technology from the University of West Georgia.

Evaluation Facilitator:

Valarie Gutierrez is an English Language Arts teacher at Marietta Sixth Grade

Academy. Valarie has seven years of teaching experience and has a number of extra

curricular and leadership roles at her school. Some of Valeries leadership positions

include being a team leader, Fall Festival coordinator, and member of the Literacy
Committee. Valarie also serves her school as the yearbook and step team chair. Valarie

earned her Bachelor's in Communication from Kennesaw State University, her Masters

degree in English Education from Kennesaw State University, and is currently working

to complete her Educational Specialist in Instructional Technology from the University of

West Georgia.

Evaluation Facilitator:

Libra B. Royster-McGriff is currently an assistant principal at Frederick Douglass High

School and has ten years of experience working in education. Her responsibilities as

assistant principal include overseeing Science, Math and CTAE content areas. Some of

her other leadership roles include district cluster STEM administrator, administrator of

school improvement, administrator of federal grants, lead testing & assessment

coordinator, curriculum and instruction leader as well as a member of the district

discipline team. Libra earned her Bachelor of Science Information Technology from

Georgia Southern University and her Masters in Educational Leadership & Supervision

from the University of West Georgia. Libra is currently working to complete her

Educational Specialist in Instructional Technology from the University of West Georgia.

Evaluation Purpose

According to the website of Positive Behavioral Interventions and Supports,

teaching behavioral expectations and rewarding students for following them is a much
more positive approach than waiting for misbehavior to occur before responding. The

purpose of school-wide PBIS is to establish a climate in which appropriate behavior is

the norm. Based upon the push throughout the district for PBIS to be used and effective

in and out of the classroom. The success and potential long term sustainability of the

program needs to be evaluated in order to allocate the necessary resources and make

any needed improvements. For the program evaluation to prove its legitimacy and instill

a sense of credibility among all stakeholders the evaluation group plans to conduct our

evaluation plan through the lens of Sprague and Corners 3 core themes of preventing

problematic behavior. In addition we plan to examine the programs necessity and

effectiveness even further through determining if the program utilizes the 7 key features

of a successful PBIS program which have been established and proven through

research by Horner, Todd, Lewis-Palmer, Irvin, Sugai, and Boland. The purpose of this

evaluation is to look closely at how PBIS affects the discipline, safety, and climate

throughout the school. With the graduation rate in the state of Georgia declining and 3rd

in the nation, the evaluation process will determine if PBIS offers a solution that will

support the interest of the individuals and groups.

An honest and earnest evaluation of Richmond Hill Middle Schools PBIS

program would mean that the school has identified a positive way to shift negative

behaviors to positives behaviors. An evaluation plan which could prove the

effectiveness of the PBIS program would also mean that additional resources would

deservedly be allocated to Richmond Hill Middle School for the purposes of continuing

and expanding the PBIS program. This evaluation plan would also impact the many

stakeholders who have a vested interest in the results of this evaluation plan. Students,
families, teachers, administrators, local government, local law enforcement and even

local businesses would stand to benefit from the success of this program and thus

would be highly interested in any of the program evaluations findings. These groups

could provide valuable resources or possibly restrict resources which would be devoted

to the PBIS program based on the evaluation's findings. School discipline is an issue

which can reverberate throughout an entire community, which would of course impact

all stakeholders. A successful PBIS program and an effective program evaluation would

prove the validity of the program and satisfy the desires of all stakeholders.

To ensure that the program evaluation uses meaningful activities and

assessments as well as determines the impact that the program is having on the

student body and the school, an evaluation that is responsive to the stakeholders and

their community is warranted. The goal of these activities and assessments should be

that evaluation findings would lead to a deeper understanding of the processes of the

PBIS program. Below you will find the program logic model which will be used as a

guide for the program evaluation.

 Figure 1. Program Logic Model: PBIS Evaluation Plan

The three (3) evaluation questions which will help render judgment on the

effectiveness of the program are as follows:

1. To what extent will PBIS result in fewer discipline referrals during the
school year?
2. To what extent will PBIS reduce the number of total student absences
during the school year?
3. To what extent will PBIS improve student achievement on standardized
tests?

The key PBIS stakeholders of Richmond Middle School will include representation from

the following:

Administrator

Counselor

District Social and Emotional Learning Coordinator

 Grade Level Teacher Staff

Special Education Staff

Paraprofessionals

The evaluation team consists of three (3) veteran teachers and one (1)

administrator, each of whom is in the process of completing their Education Specialist in

Instructional Technology. The evaluation teams primary goal for this program evaluation

is to provide an unbiased evaluation of the effectiveness of the PBIS program at

Richmond Hill Middle School. Answer to the above questions will provide both

quantitative and qualitative data to determine the impact that PBIS is having on the

climate and culture of this school. A balanced approach to discipline is one in which

safety is maintained and students rights to a free and appropriate public education is

maintained as well (IDEA, 2004)

Program Evaluation

Fitzpatrick, Sanders, and Worthen (2004) define evaluation as the identification,

clarification, and application of defensible criteria to determine an evaluation objects

value (worth or merit) in relation to those criteria (p. 7). The evaluator works with those

who have an investment in the program being evaluated stakeholders in this case

PBIS, to determine the criteria to which to judge the programs value. The evaluation

will uncover both the strengths and weaknesses of the program and to ultimately to

determine the effectiveness of the program's original goals. To judge the quality of an

evaluation, it is important to investigate its accuracy (the extent to which the information

obtained is an accurate reflectionwith reality), utility (the extent to which the results

serve practical information needs of intended users), feasibility (the extent to which the
evaluation is realistic, prudent, diplomatic, and frugal), and propriety (the extent to which

the evaluation is done legally and ethically) (Fitzpatrick et al., 2004, p. 7).

Our schools and school districts have so many programs and initiatives that can

be overwhelming for the administrators, staff, and students. Furthermore, with so many

programs and initiatives pushed down to schools, we often question their effectiveness.

In order to paint a complete picture of program effectiveness, evaluation of the program

is needed. Evaluations of programs in schools are important. While our focus is on

PBIS, some of the other examples of evaluations in our schools include monitoring

progress towards benchmarks, making decisions about the effectiveness of afterschool

programs, and judging the pacing and quality of a specific curriculum. While there are

several different types of research methods to measure PBIS programs, the program

evaluation provides the stakeholders with a method to acquire key information about

PBIS that is specific to the needs and goals of Richmond Middle School.

Methodology

The program participants will include middles school students between the ages

of eleven and fourteen (grades 6-8). The program is comprised of students from three

grade levels (6th -8th), referred to as teams, into which they are grouped based on their

respective grade levels. Assistant Principals, counselors, social worker, grade level

teachers, special education teachers, and teaching assistants employed at Richmond

Middle School are also participants. Those staff members and students from whom an

informed consent was obtained participated in this evaluation.

Staff Participants. The evaluators presented a consent form to each staff member,

and they then explained, read, and answered any questions that they may have had
pertaining to consent to participate. The staff was also given time to read, review, and

think about participating in the evaluation. It was explained that participation is

completely voluntary and that their decision to participate would not negatively affect

their TKES evaluation nor employment at Richmond Middle School.

Child Participants. Parents of all students enrolled were provided with a written

statement of the evaluation, and a follow-up phone call was made a week later by the

evaluators to answer any questions the parent might have. This was also an opportunity

to obtain a verbal consent if they had not returned the written consent. It was explained

that participation is completely voluntary and that their decision to allow their child(ren)

to participate would not negatively affect their students academic and behavior

performance at Richmond Middle School.

The data for the plan came from the referral logs of the 2015-2016 and the 2016-

2017 school years. Both years will be compared to see if discipline referrals, both major

and minor issues, are decreasing. Also, an attitudinal survey will be given to teachers to

determine changes and approaches to PBIS. Below you will find an example of one of

the many data gathering tools which will be used during the program evaluation.
 Figure 3. PBIS Discipline Form for Richmond Hill Middle School

Some of the data that will be gathered to determine the effectiveness of the PBIS

program will come from purposeful contrast sampling. This will allow us to compare

previous years behavioral data on approximately 1,658 students with behavioral data to

be gathered during and after a year of implementation of the PBIS program. Cluster

sampling will be used to gauge how the PBIS program has influenced the attitude and

opinions of 127 teachers. Data analysis will come from both qualitative and quantitative
sources. The behavioral data will be analyzed quantitatively to identify trends that may

prove the effectiveness of the PBIS program. While the qualitative data, which will come

from teacher surveys will be used in part to determine the effectiveness of the PBIS

program. This data will also be analyzed to find methods for the improvement of the

PBIS program. These are practical procedures which lends itself to a natural

assessment of the process.

Standard(s) for Evaluation Question(s)

We will collect and use information from the evaluation of the PBIS program for

decision-making. Our purpose of the PBIS Evaluation questions is to provide those

involved support in a) addressing evaluation questions that may be most useful, (b)

selecting evaluation measures and measurement questions that meet the needs of the

primary stakeholders and (c) using the evaluation information for active decision-making

at the school, and district. We propose the model for addressing evaluation questions

using (who, where, when, why) in which implementation of PBIS is to occur or has

occurred.

Context

1. What are/were the goals and objectives for School wide PBIS implementation?

2. Who provided support for PBIS implementation?

3. Who received support during PBIS implementation?

Input

1. What professional development was part of PBIS implementation support?

2. Who participated in the professional development?

3. What was the perceived value of the professional development?

Fidelity

1. To what extent was PBIS implemented as designed?

2. To what extent was PBIS implemented with fidelity?

Impact

1. To what extent is PBIS associated with changes in student outcomes?

2. To what extent is PBIS associated with changes in academic performance,

attendance and behavior?

These evaluation plan questions will provide information to determine the

success of PBIS implementation at Richmond Hill Middle School. They will also provide

detailed summaries of the evaluation study to include the data collection process and

key findings.

Evaluation of PBIS

Through our research we discovered that there have been several evaluations of

a school-wide PBIS program. Quite a few of the documented evaluation studies were

relevant for our evaluation plan. Warren, Edmonson, Griggs, Lassen, McCart, Turnbull,

et al. (2003) conducted an evaluation of a school-wide PBS program in an urban school

setting, and described certain considerations that needed to be taken into account when

working in a diverse, urban setting. First, establishing buy-in from administration, faculty

and students was crucial to the success of the program. Second, they utilized a four-

tier-model of PBIS, based on the particular behavior challenges they encountered.

Many of the students in the school received intervention beyond the first tier. Finally, a

positive school culture and improved social and learning outcomes for all students
derived from utilizing multi cultural social and emotional learning. We also discovered

that in a study by Lowe, Jones, Allen, Davies, James, Doyle, et al. (2007), staff training

positively impacted knowledge and perceived confidence among staff, whereas the

training had a minimal effect on staffs attributions or emotional responses. Training was

emphasized as one of the key components to conducting effective evaluations. The

study concluded, however, that for ongoing program success, systematic organizational

changes had to occur. Implementation of PBIS programs typically presents many

challenges. However, the evidence suggests that positive results are possible and

stakeholders will use our evaluation plan to identify areas of success and/or areas

needed for improvement.

 References

About Us - PBIS. (n.d.). Retrieved October 23, 2016, from https://www.pbis.org/about-us

Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2004). Program evaluation: Alternative
approaches and practical guidelines (3rd ed.). Boston: Pearson Education, Inc.
Individuals with Disabilities Education Improvement Act of 2004, P.L. 108-446, 20
U.S.C. 1400 et seq.

Lowe, K., Jones, E., Allen, D., Davies, D., James, W., Doyle, T., et al. (2007). Staff
training in positive support: Impact on attitude and knowledge. Journal of Applied
Research in Intellectual Disabilities, 20, 30-40.

Sprague, J. R., & Horner, R. H. (2006). School wide positive behavioral supports. In S.
R. Jimerson & M. J. Furlong (Eds.), Handbook of school violence and school safety:
From research to practice (pp. 413427). Mahwah, NJ: Erlbaum.

Swain-Bradway, J., Pinkney, C., & Flannery, K. B. (2015). Implementing Schoolwide

Positive Behavior Interventions and Supports in High Schools: Contextual Factors and
Stages of Implementation. TEACHING Exceptional Children, 47(5), 245-255.
doi:10.1177/0040059915580030

Warren, J.S., Edmonson, H.M., Griggs, P., Lassen, S.R., McCart, A., Turnbull, A., et al.
(2003). Urban applications of school-wide positive behavior support: Critical
issues and lessons learned. Journal of Positive Behavior Interventions, 5, 80-91.
Appendices

Figure 1. Program Logic Model: PBIS Evaluation Plan

Figure 2. Program Evaluation Contract
Figure 3. PBIS Discipline Form for Richmond Hill Middle School