Article #4: Never mind the quality (of education), feel the width!

Writing this blog today I am reminded of a UK TV programme from the 1960's entitled "Never mind the quality feel the width". It was a comedy about two tailors in the East End of London making suits from inferior quality cloth, and when anyone ever questioned them about the material they would always say "never mind the quality, feel the width!"

Schools in Nepal can be like this, "never mind the quality of the teaching, look at how many more children now enrol for school"

Training programmes can be the same, "never mind the quality of the programme, it WAS 5 days long"!

Except our programmes are not like this, and we can prove it.

It's all about evaluation, a difficult thing to do in many cases where individuals attend for training as random individuals instead of being from an organisation such as a school where all of their colleagues attend too, and this is the scenario we have in Nepal. So, how do you evaluate a training programme?

Professional trainers use a technique based on the Kirkpatrick model or framework which is organised into four levels of measurement, all of equal importance, each building on the previous level.

1. Kirkpatrick Level 1 measures the "Reactions" of delegates to the training; not just did they like it, but giving a rating to a number of important factors such as the course content, the learning achieved, and the competence of the trainer. This is usually achieved by using a pro forma with rating scales at the end of each course.
2. Kirkpatrick Level 2 measures the "Knowledge gain" by the delegates at the end of the course; simplest way to do this is with a test of the delegates knowledge at the end of the course compared with their knowledge at the beginning using questions based on the course content.
3. Kirkpatrick Level 4 measures "Behaviour change" and now takes us into the realms of post course application; this is achieved by visiting the delegates place of work and observing them at work and looking for the application of specific techniques, skills and concepts from the course. Harder to do and time consuming, but the only way to assess value for money and the real effectiveness of the training to the individual.
4. Kirkpatrick Level 4 measures "Benefits & Results" to the organisation where the delegates work; we would do this in schools by measuring exam passes and completion rates for example over an elongated period of time. Once again more difficult since time must be allowed for the training to take effect.

So, there you have it, a professional trainer's evaluation kit. Are we using it? You bet we are!

At the beginning of the year we began with Kirkpatrick Level 1 by using a Reactions Questionnaire on all of our courses, and a results summary from around 150 teachers is shown below with all ratings on a scale of 1-5.

From April we began implementing Kirkpatrick Level 2 with a Knowledge Test on our Foundation course applied pre and post course and the results are also shown below.

Average Delegate Pre-Test Foundation Score: 6%

Average Delegate Post-Test Foundation Score 56%

Next, we have just formalised Kirkpatrick Level 3 by making systematic inspections of schools once 80% of teachers at any school have attended Foundation. The inspection lasts for about 2 hours and our Consultants use a 10 point checklist observing lessons on a number of teachers, looking for physical changes in the school environment and in discussion with the Principal. We will inspect around 20 schools this month and publish the results in August.

Finally, we will begin with Level Kirkpatrick Level 4 towards the end of this year when we amalgamate the process with some management training we plan for Head Teachers and Principals at each school we have worked with.

I'm sure you will agree that this is a thoroughly professional approach to the evaluation of our work and that the results so far are outstanding. All our Consultants are trained in this approach and have contributed to the development of materials at each level.

We are indebted to Leslie Allan, Managing Director, at Business Performance for freely sharing their materials with us to help everyone in Kathmandu learn the technique.