Visualisation of judgements

A key theme that stood out for me in the block readings related to accountability and the judgements that are made, collected, and used to measure success.

For my data visualisation I therefore decided to try to collect data on judgements, or considered measurements, that I made in one day. I recorded where I completed an action that involved a degree of consideration of information presented, or where tools were used to help determine an outcome.

As the data table shows I started out with three distinct categories, for the visualisation these fitted better grouped into two main types of ‘judgement’, ie qualitative and quantitative. The definitions I used to classify my judgements:

Quantitative – a value, discovered by measuring, that corresponds to the quantity, size, shape, etc.

Qualitative – a subjective assessment, estimation of the quality (value/worth) of something or someone. 

I also chose to capture surrounding information on how the judgement was arrived at, to try to capture a bit of the consideration process to ensure the judgements were not just quick decision but were instead something I had arrived at having used tools or information to draw a conclusion from. I opted to record:

• The classification – to show whether the measurement or judgement was for example, precise or an estimation.
• Time bound – if the judgement related to a measurement of time.
• Any equipment used – either a physical measurement tool or reference materials to help inform my ‘judgement’.
• The time taken – to try to capture how quickly I arrived at the ‘judgement’.

This information was useful to help identify patterns or trends in the data collected; these are noted under ‘observations’ in my visualisation. As with previous visualisations I appreciate there could be issues with the ‘raw’ data, for example:

• the classification recorded was ‘precise’ – but this was only precise for my purposes. Had the measurement of, for example, water been required in a scientific experiment then a higher degree of accuracy may have been required.
• when considering the classification of ‘opinion’-  I noted I was giving my opinion multiple times within the context of a large document. This was however only counted as one occasion or one judgement.   

The question I kept coming back to when recording my data was, ‘would others agree with my measurements or judgements?’ For the quantitative measurements it could be argued that the use of a common agreed measurement (such as cm/inches etc) and a reliable measuring tape should result in two people agreeing the same outcome, but that is only true if there is clarity around what is being measured, what the agreed process would be, and how the measurement would be recorded.

The discourse around measuring school performance often sounds as though standard measurements are being made as standardised tests, or set criteria, are used on visiting government inspector reports etc, but how do we guarantee that these tools are being used effectively and consistently to provide accurate measurements which may then be used to inform judgements on school performance or even to inform policy on improvement programmes.

When we make inferences on schools’ performance or hold them accountable for the standard in the school can we be sure that the judgements being made are accurate? Have we got the tests right for measuring performance and are they being measured correctly? As Williamson (2017) highlights authority over education has seen a move away from centralised government and its agencies to instead include a wide range of other organisations and advisory bodies. This group of actors will likely bring with them a range of talents and knowledge; however, they also will have their own data collection methods, analysis, and interpretations and therefore potentially different ‘solutions’ to perceived issues or suggestions of future improvements (which may also happen to fit with the agencies own direction of travel). The increased prevalence of data being captured from digital education as Anagnostopoulos et al, (2013) describes has opened the opportunity for consultancies and computing firms to generate or expand their business as they assist districts or states as the reliance on data analysis and the testing process to generate the data becomes increasingly key to determining the success, or otherwise, of an institution. However, with more agencies involved is there a question over the reliability of the data collection and analysis process.  

As Fontaine (2016) highlights accountability in education often starts with schools being held responsible for learner performance which does not take account of any of the socio-economic factors that could have contributed to outcome of the learner’s performance. With so much at stake for schools is it perhaps understandable why some schools may resort to teaching to the test to try to ensure that any measurement of performance results in their school reflect positively.  As Williamson (2017) reflects this a move in educational institutions to focus on what can be ‘counted and accounted for’, which in turn prioritises activities that can be measured rather than activities which cannot be measured. The question here is what the long-term effect of that might be on the learners, and teachers, who are continually monitored on performance (counting results and outputs) rather than considering the much wider learning process which is a far more significantly complex process.

References

Anagnostopoulos, D., Rutledge, S.A. & Jacobsen, R. 2013. Introduction: Mapping the Information Infrastructure of Accountability. In, Anagnostopoulos, D., Rutledge, S.A. & Jacobsen, R. (Eds.) The Infrastructure of Accountability: Data use and the transformation of American education.

Fontaine, C. 2016. The Myth of Accountability: How Data (Mis)Use is Reinforcing the Problems of Public Education, Data and Society Working Paper 08.08.2016

Williamson, B. (2017) Digital Education Governance: political analytics, performativity and accountability. Chapter 4 in Big Data in Education: The digital future of learning, policy and practice. Sage.