Process Indicators

Abstract

This paper explores which accountability indicators are likely to reveal the distinct contexts and qualitative characteristics of school that stimulate and improve authentic pedagogy and accountability. In the era of accountability, data-driven decision making is a new research area for authentic pedagogy through monitoring student progress and improving school accountability. It is based on input-and-result oriented indicators such as school demographics, facilities, budget, standardized test scores, dropout rates. But the indicators are unlikely to capture a dynamically interactive qualitative characteristics of school organizations featuring a loosely-coupled system and difficult to be measured or assessed. Thus, process indicators need to be complementary to input-and-outcome data for a valid and graphic description, monitoring and explanation of “why” and “how” the school outcomes occur. The author concluded that the data-driven decision making (DDDM) based on process indicators strengthens reflective professionalism and provides for the educational welfare for the poor and left-behind students.

Keywords: Data-Driven Decision Making; Process Indicator; Educational Accountability; Transparency; Educational Policy

Source: Kyu Tae Kim, https://www.scirp.org/journal/PaperInforCitation.aspx?PaperID=23349
This work is licensed under a Creative Commons Attribution 4.0 License.

Introduction

In the era of accountability, data-driven decision making (DDDM) is a new research area for authentic pedagogy through monitoring student progress and improving school accountability. It is based on input-and-result oriented indicators such as school demographics, facilities, budget, standardized test scores, dropout rates. But the indicators are unlikely to capture a dynamically interactive qualitative characteristics of school organizations featuring a loosely-coupled system and difficult to be measured or assessed.

School organizations and professional performance have many invisible and qualitative characteristics that cannot be fully understood and evaluated by input-and-output indicators based on objective observation, rational and logical analysis, and operational and quantified experiment. Young and Wayman and Springfield identified, in spite of the positive effect on agenda setting for using data, that schools tend to show distinctive response to use and approach indicators in terms of their organizational contexts and cultural norms. This means that school organizations can be understood by indicators for "multi-side description" of their total qualities such as values and meaning systems formed within organization, educational experiences and lifestyle, and the complicated contexts and processes of schooling. Scholars have referred to these indicators as process indicators.

Process indicators are usually related to the quality and realities of curriculum, instruction, and interaction. They may be useful for describing equal educational opportunity, for monitoring school reform practices such as change in curriculum, change in organizational structure, change in pedagogical practice, and for explaining and diagnosing causes and results of the educational systems. Also, the indicators can be really used for measuring and evaluating authentic student progress such as higherordered thinking, problem solving, student's happiness and satisfaction, prevention of unhealthy behaviors, and social capital. Thus, process indicators need to be complementary to input-and-outcome data for a valid and graphic description, monitoring and explanation of "why" and "how" the school outcomes occur.

In this paper the author will argue that process indicators produce authentic pedagogy, school effectiveness and accountability. This paper is to address what accountability indicators are likely to reveal the distinct contexts and qualitative characteristics of schools in order to stimulate and improve authentic pedagogy and accountability and how we capture better qualitative characteristic of teaching and learning, and to draw on schools' "what's-going-on". In the following sections the author will cover what DDDM and process indicator are, why process indicators are considered in the loose-coupling school, what are the relations between DDDM and process data in the era of accountability and then will draw on the implications and suggestions.

Data-Driven Decision Making

DDDM means educators and policymakers utilize and analyze school and student data to improve school effectiveness and to recognize the value of data. The term DDDM has been generally used with databased decision making, research-based decision making, and evidence-based decision making interchangeably. According to Marsh, Pane and Hamilton, DDDM means that schools "systematically collect and analyze various types of data, including input, process, outcome and satisfaction data, to guide a range of decisions to help improve the success of students and schools". The multiple sorts of indicators include: input indicators such as school demographics of students and teachers, and expenditures; process indicators related to operation of curriculum and instruction; outcome indicators connected with dropout rates and student test scores; satisfaction data connected with opinions from teacher and students, etc.

DDDM is a sphere of currently emergent research areas for monitoring student progress and school improvement, certifying educational problems and needs, and assessing program effectiveness. DDDM is based on accountability indicators which refer to comprehensive statistical information linked to generate and utilize the accurate information of process and performance on complex school organization. The current accountability indicators are composed of quantitative inputand-result oriented indicators such as standardized test score, dropout rates, graduation rates, and so on. They, intrinsically, may be designed to promoting the equality of educational result through the advancement of student learning and the enhancement of professionalism for taking care of the poor and leftbehind students. It is evident that accountability policy can strikingly close the achievement gaps among students by paying attention to the reading and math standard and high-qualified teachers.

and high-qualified teachers. However, school organization has both a tightly-coupled and a loosely-coupled perspective. The tightlycoupled frame highlights centralized control, coordination by written rules, vertical communication, hierarchy, supervision, compliance, efficiency, and extrinsic incentive. Meanwhile, school organization is a professional organization, which provides an operational core for schooling. Also it is a loosely-coupled system in that schools can be conceptualized as "differentiated organizations", or "culturally heterogeneous organizations", which means that they have internally complex and distinctive cognitive and emotional strategies. Therefore a school is a loosely-coupled lens focused on professional organization oriented to educators' professional knowledge and judgment. In this respect, school organizations are likely to interact dynamically with a variety of individual and group-level contextual factors. As Greenfield indicated, a school's organization should be understood as "an object of understanding".

In spite of the loosely-coupled image, as Hoy and Miskel said, the demands for accountability may make school organization more formalization, more centralization, less professionalization. The tightly coupled policy has been influenced by the government's increasing involvement in schooling. Current educational policy was designed to improve education through "a tightly-coupled DDDM based on higher standards, testing, and accountability. However, teachers mostly have worked in solitary classrooms where they are not easily evaluated by colleagues or supervisors. Firestone and Herriott indicated "in schools, the actual work of bringing students in contact with what is taught is done by teachers who have considerable discretion. In effect, a major portion of the school's central purpose cannot be controlled by the administrative cadre". Put differently, teachers are educational critics that distinguish and evaluate their works and students' needs in their own way within specific contexts.

In an era of strict demand of accountability, top-down accountability policy is focused on students' academic performance. However, it may need to be balanced with teacher-centered indicators focused on the active involvement of professionals and the mutual collaboration of practitioners. O'Day indicated the importance of the rich generation, valid interpretation, and relevant circulation of information and data among accountability players because the conflicts result from the miscommunication between administrators and professionals who have different accountability views: administrators focused on students' academic performance by state's high-stakes test such as reading and math score, attendance rate, and graduate rate; however, professionals put an emphasis on educators' professional knowledge and judgment according to peer review and sanction. In this respect, she argues that well-informed data catalyze as a medium of communication between both sides.

on between both sides. As O'Day indicated, the rich generation, valid interpretation, and relevant circulation of the proper data and information related to school reality are likely to contribute to being successful for organizational capacity and improvement. The data and information should be related to generating and focusing on information relevant to teaching and learning and to changes for the continual "calibration" and "feedback". They are likely to motivate educators and others to attend to relevant information and to expand the effort necessary to augment or change strategies in response to this information. Furthermore, they are able to develop the knowledge and skills to promote valid interpretation of information and appropriate attribution of causality at both the individual and system levels. Because the teacher-based indicators are dependent on an acquired and processed indicators by teachers who in a actual context have a sense of the operational situations, problems and alternatives for enhancing school improvement and effectiveness. In this vein, it is reasonable for DDDM to be based on not input-andoutcome-based indicators but process indicators that can describe "contexts" and explain "causes", in that "a snapshot of school practice is not sufficient; assessment of change is needed".

Young and Wayman and Stringfield identified that schools show distinctive responses to use and approach data in terms of their organizational contexts and cultural norms. DDDM needs to be based on the flexible and diverse process indicators used to provide timely diagnostic information of improvement, to capture a better qualitative characteristic of teaching and learning, to explain "whys" when students and schools don't reach the standard and to provide which support to schools. In this respect, process indicators can be linked to substantive instruction support and curriculum provisions by calibrating through a productive and reflective "test talk" or "communication" with stakeholders. By using process indicators, school-level working condition or district-level agenda setting can be related to establishing collaborative works and learning norms and climate by helping to understand everyday instruction-related practices within the contexts and by helping find how to align and arrange district-driven policies with their contexts and change endeavors.

Scholars suggested four types of indicators, context, input, process, and product for the decision making in terms of accountability. In terms of context indicators, it would be the students' achievement level which needed to be improved, instructional and personal barriers to study in classroom, students' absence and dropout rate, etc. It could be referred as input indicators: school budget and resources and time invested to solve the school problem in order to achieve educational goal, and so on. It would be considered degree of the relationship and understanding between students and teachers, adequacy of time schedule, teaching activities, and school resources as process indicators. It may be thought of the output on whether learner achieved learning objects and the indicators which is related to the output such as parents and students satisfaction in terms of product indicators.

DDDM is based on contextual factors or school cultural and institutional factors: data quality, calibration, principal leadership, faculty involvement, educator collaboration, and other institutional supports. First, data quality is related to keeping highquality, accurate data, and an accurate and quality database. It provides the following information for educators and policy makers: what programs have been provided to students and how students completed the programs, how the test result has been improved, and what teachers have used an adequate teaching method for improving student achievement by a school year. Second, calibration defined as collective reflectivity or deliberation is combined with how educators define indicator use, how teaching conduct under these definitions, how they assess student learning, and how they react to results. Third, principal leadership is connected with principals' investment and support for the use of data system. The existing research recognizes a role of principal as one of the critical factors to improve school management and student achievement. Principal is also considered as an important player in the data use. However, school leadership needs to be focused on distributed leadership, being stretched over more broadly and distributed beyond individuals because the role of school principals for DDDM is limited. Fourth, faculty involvement has to do with teachers' engagement and interest to the data generation, use and application for their classroom and school program. According to Marsh, Pane and Hamilton, factors to promote the data use depend on accountability policies and intrinsic motivation: Federal, state, and local accountability policies such as incentives and pressure to use data; the internal desire of teachers to make better use of data. Fifth, collaboration is consistent with educators' data sharing and co-using. Collaboration for DDDM in the level of school is closely related with organizational culture and school leadership. Finally, institutional supports are coupled with the education authorities provide deeper and continuous professional development, establish a friendly data warehouse, and give teachers sufficient time to access and examine data.

Based on the above discussions, as Figure 1 shows, we will elaborate on how five elements are correlated with DDDM. DDDM is divided into two parts: three basic elements and two cultural catalysts. Three basic factors are "who makes use of indicators or data for what", referring to data or indicator, generating or supporting users, and calibration. Data/indicator is a connecting factor between users-generating or supporting users. Generating users are a principal, a school faculty or a school data team for exploring the problems and alternatives for school improvements and accountability in terms of school results and "what's-going-on" through calibration. Administrative staffs, who establish database system, computer software or data warehouse, support DDDM by providing professional development program in order to collect, monitor, use, and interpret indicators of schools' contexts, processes and results through calibration or collective thinking process for authentic pedagogy, accountability, and school effectiveness.

Figure 1.

The conceptual structure of DDDM.

Catalysts are embedded in the calibration meaning collectively cognitive inquiry through the interaction between users. Copland pointed out that capacity building and school improvement would bring from collective cognitive processes through organizational learning and distributed cognition. According to Spillane, distributed cognition works in a situation composed of routines, tools, structures, and institutions. A routine includes regular procedures and committee for determining activities to achieve school and team activities. A tool encompasses from the documents regarding student's achievement to protocol. A structure is related to a form of institution such as class teachers' and regular teachers' meeting, team structure within school organization, and committee and spontaneous forms regarding temporary team. An institution includes vision, goal, and regulations of school organization. These make a difference in that each school has a discrete calibration; so it has a distinctive mode and characteristic of leadership, collaboration, and involvement for DDDM.

Therefore, the catalysts are embedded in the school situation and in each situation they are emergent for DDDM. Put differently, these facilitating factors are derived from the calibration nested in interaction between users. As discussed earlier, Young and Wayman and Springfield proved that schools tend to show distinct indicator use and approach in the context of their organizational cultural features. In this respect, the cultural catalysts, referring to distributed leadership, collaboration and involvement, have a significant effect on "how or under what conditions users put to use indicators" in terms of leadership, climate and culture within a school or across schools.

Using Process Indicators for Facilitating DDDM

As far as the abovementioned information is concerned, it is appropriate to use process indicators related to describing how school has been/is going on, what factors of school process and context are effecting on "better" pedagogy, how schools respond to policy agenda. In this respect, outcome-based indicator system needs to be balanced with process indicators that can describe "contexts" and explain "causes", because "a snapshot of school practice is not sufficient; assessment of change is needed", as Porter says. Process indicators can describe, explain, and explore the school's needs and practices. The output-based data under the current accountability are not likely to reveal and measure not only the dynamic contexts and qualitative characteristics of school but also the qualitative and formative results of schooling such as higherthinking skills, quality of instruction, and student interest of reading itself.

Process indicators can stimulate data-based leadership because they give live descriptions of "what's going on" and student's real needs, and also identify barriers to use data for instructional improvement, and explain the causes of failures and draw on alternatives for improvement. Data-driven leadership may be a key medium of connection for building capacity among educators. Young argued that principals mediate actual use of data by teachers. Wayman and Stringfield asserted that professional development must equip teachers to be independent users of data in the service of instructional planning.

Process indicators can lead district and school leaders to advocate a supportive and collaborative data use culture in order to encourage their teachers and staffs to access and use data, to reflect on their instructions, and to distribute and share school leadership. According to Lachat and Smith, the school-level data use result in creating "collective leadership" and "data-based team". In this respect, data use acts as the redesign of school structure and leadership. Copland pointed out that distributed leadership based on data use contributes to sharing responsibility and collaborative work condition, drawing on each leader' own expertise and experience for enhancing school effectiveness and upgrading school organizational capacity. Distributed leadership focuses on the leader-plus through the interaction of leader and followers in the situation, the sharing of professional expertise and experience through collective leadership for organizational effectiveness and accountability.

effectiveness and accountability. Process indicators may increase reflective professionalism based on peer reviews, collaborative team activities, and shared information by fitting for educators' identity and professionalism. Schön saw professionals as "reflectors in action," emphasizing contextual and situational reflection in action when they make a decision according to continually updated contextual knowledge. Spillane found that implementers have their own interpretative frames of what they should do and their own preferences of what is the most important for their working. In this respect, process indicators are likely to combine with "databased reflectivity and deliberation" through a productive "test talk" or "communication" with teachers. Process indicators tend to lead to organizational learning through collaborative inquiry and shared expertise and experience among colleagues. This "collaborative inquiry" helps teachers deliver from teachers' individualism caused by a loosely-coupled organization and to flow relevant information into a separate room of teachers.

Process indicators can be really used for measuring and evaluating authentic student progress such as higher-ordered thinking, problem solving, student's happiness and satisfaction, prevention of unhealthy behaviors, and social capital. Process indicators are considerably consistent with micro tasks such as the information of teachers' and students' day-to-day interactions, realities and lives. The Information is to an acquired and processed data set from schools and teachers in order to facilitate data-based decision-making for enhancing authentic pedagogy and reflective professionalism for school improvement and effectiveness.

In spite of these bright sides, there are several limitations needed to be considered in introducing process indicators into classrooms and schools. The first consideration is that process indicators are oriented to formative self-evaluation focusing on identifying and treating educational progress during the student learning or the school operation process; so, it is hard to gauge a school's success or failure and to make teachers and schools districted from their attainment of standards and goals.

Second, it is indispensible for teachers and schools to make use and interpret process indicators regularly and daily and maintain the updated data warehouse frequently. It forces them to do too much additional work apart from their instruction and resource preparations. This may result in the increase and expansion of teachers' roles such as data preparation, interpretation, and reporting; so, teachers may invest their more time on data use and input more than instructional improvement and provision of resources to students.

Third, specific perils which too much focus on data generation and use can cause serious work stress and depression and lead teachers to dampen student interest and deemphasize students' authentic pedagogy and narrowed curriculum dedicating to data preparation and provision instead of substantial amounts of instructional time.

Fourth, process indicators are inefficient and infeasible because they related to a complicated and delicate cases and realities; they are required for teachers' long-term work time and effort; they cannot set up the standard indicator system in order to get the standard data from a distinctive school.

Fifth, process indicators are too subjective and individualistic to secure validity, reliability and objectivity for identifying a school's and a district's summative performance and for integrating the data derived from an individual school in the state or national level.

Sixth, it is necessary for teachers and schools to have the professional expertise and know-how about generating, using, and interpreting of process indicators within a school or across schools. However, most teachers do not understand data use and DDDM.

Implications and Conclusion

Process indicators enable schools and teachers to scientifically make decisions for fit-for-all instructional strategies and high-quality professional development, to provide differentiated instruction, to increase organizational learning;, to calibrate their "what's going on" and to stimulate collaborative or collective learning.

Process data may be required to a new principal leadership that can not only lead teachers to generate and use data and build data-use culture for their instructional improvement and school accountability. However, result-based accountability revealed the limitation in that the heroic leadership may fail to draw on the teachers' active involvement and the mutual collaboration of practitioners with school leaders because of limited information flow and sharing, one-way communication, centralization of role and responsibility to one leader. Distributed leadership puts an emphasis on the fact that there are multiple leaders, multiple followers and situations and that leadership activities are "widely shared within and between organizations". Distributed leadership is able to facilitate teacher's motivation for sharing, co-performance and collective responsibility for school improvement and accountability. If principal leadership is stretched out to teachers, teachers may play a active role in shaping the culture of their schools, improving student learning, and influencing practices among their peers by becoming a resource provider, an instructional specialist, a curriculum specialist, a learning facilitator, a mentor, a school leader, a data coach, a catalyst for change and a learner.

Accountability policies are designed to promote the equality of educational results by taking care of poor and left-behind students. However, the input-and-output based accountability has resulted in the heated discussion of equality versus excellence. Proponents of educational equality, a teacher union and liberal interest group, worried that the policies would further polarize educational opportunity along class lines and family background and that it would have a pernicious labeling effect among schools. The advocates of educational excellence, government and conservative interest groups, tried to push through the school choice policy by increasing competition among schools and by promoting test score publication. These conflicts are due to lack of the deep consideration and discourse for jumping into the perspective and interest of each stakeholder.

Put another way, the conflicts come from a lack of the databased deliberation and collective inquiry process. In this case, it is not likely to facilitate "non-self-interested motivation" for increasing self-sacrifice and public good through "deliberation democracy" based on the deliberative communication, altruism and cooperation in a public sector. Ranson indicated that it is necessary that players of school accountability recognize a conflicting plurality and contestation and reach a mutual understanding about the meanings, purposes, perspective, and practices of school organization under open discussion and discourse processes. This reflective deliberation, fundamentally, results in the stimulation of a collective learning process and the formation of a professional community. In this vein, process data can be a key medium of connecting between proponents and opponents. It is not easy to reconcile the conflicting perspectives of both sides without considering what's-going-on data.

The process data can identify how poor students are learning higher order thinking and problem solving ability when comparing with affluent family's children, and how teachers have high expectation of learning to all and how class activities enhance their emotional and social development. Also the process data can check what factors have had a significant effect on stimulating critical thinking, conceptual learning and intrinsic interest in the subject matter, and desire to pursue future education. Furthermore, the process can pay attention to how and what make low-performing schools and poverty students have been improved their progress. In this respect, process data can promote Anderson's "advocacy leadership" emphasizing students' whole-being growth and all-round education by holding the following belief and practicality:

Process data is intrinsically required to internal accountability in that the data put an emphasis on collective inquiry and collaborative responsibility. Newmann, King and Ridgon found that school performance can be improved by internal accountability rather than external accountability in that it can facilitate self-producing organizational capacity by stimulating relevant utilization of professional knowledge and skills by sharing of objectives among stakeholders, and by establishing a cooperative system. Also, Abelmann and Elmore researched how schools conducted their own accountability mechanisms: 1) Putting emphasis on individual or professional accountability rather than administrative accountability; 2) Pointing to internal accountability through collective expectation and mutual control; and 3) Focusing on the strong leadership of principals and the internalization of accountability. In this respect, process indicator use must be conducted to facilitate organizational learning through which administrators and professionals can explore and share school problems and performance together in order to overcome the teacher individualism caused by a loosely-coupled organization and to flow relevant information into a separate room of teachers. Organizational learning makes administrators enter into the loosely-coupled school; on the contrary, it makes teachers open their closed window toward the external world and its changes. Therefore, as Darling-Hammond and Ball indicated, accountability practices must point to facilitate collective learning through open and deliberate dialogues and discussions between administrators and professionals to understand mutual perspectives and realties.

In the context of accountability, DDDM is a crucial driving force for school accountability and improvement. The successful implementation of DDDM within a school and between schools and local educational agencies are dependent on what indicators are stressed on. If DDDM is linked to input-and-output indicators, it is difficult to make sense of schools' processes and realities, draw on the best practices, figure out students' actual progress, and facilitate new culture creation and collective inquiry or organization. As a result, authentic pedagogy cannot be realized because it is combined with intensifying reflective professionalism and caring for the educational welfare for the poor and left-behind students. It undoubtedly comes from process indicators.