Month: July 2014

Posted on

Secrets of metrics

Reading Time: 2 minutes

In a blog post, James Bach discusses the virtues of dumping Key Performance Indicators (KPI’s). It’s a shock to think that maybe we should get rid of quantitative metrics, however, there is a sound argument behind the idea.

If you think about it carefully enough, quantitative metrics serve two purposes:

  1. Snapshots
  2. Objectifying reality

Snapshots

As an experienced software performance optimizer, I can attest to the value of snapshots. Without objective metrics there is no way to narrow down what is consuming time in a software system. However, there are a couple of important points to remember about capturing metrics in software development — and by extrapolation the real world.

  • Capturing good metrics adds a lot of overhead, especially to repetitive operations.
  • If you’re not adding a lot of overhead, you’re not capturing good metrics.
  • Once you’ve optimized the processes, you need to get rid of the overhead that is capturing the metrics.

Objectifying reality

This is a fancy term for vanity metrics. The problem with metrics is two fold: you can only manage what you measure and when you measure, that’s what you get — whether or not you wanted it. Usually, you discover that what you are measuring isn’t what you really want. That leads to

TestsICanTakeTests
https://twitter.com/CharlesM_S/status/479722716216324097?utm_source=fb&utm_medium=fb&utm_campaign=SirKenRobinson&utm_content=479731710964142081

In Bach’s words

Gather relevant evidence through testing and other means. Then discuss that evidence.

That’s how it works for us. That’s how it works for publishers. That’s how it works for almost everything.

In conclusion

I’m not against KPI’s and metrics in and of themselves. I’m opposed to blindly relying upon metrics to replace qualitative discussions. Numbers are a hollow model of reality and are too easy to use to justify distortions. Until you really understand what the number means, it’s a mistake to put too much faith into it.

Posted on

Replacing the SOL

Reading Time: 2 minutes

512px-Inside_a_classroom_of_a_school_in_KabulThis is a thought experiment more than anything else, however, it may have practical application. In business, we’ve learned that there is one question to ask a customer which is more useful than any other

On a scale of 1 – 10, how likely would you be to recommend us to a friend? Why or why not?

On a similar vein, I propose totally changing the Standards of Learning test as a way of evaluating a teacher. The flaws in the SOL are very straightforward:

  1. It takes place too late — by the time the student fails the SOL, it’s too late to bring them back up to speed
  2. It incentivizes “teaching to the test” and no employer gives a rat’s ass if a student can pass a test.

My hallucination is that the primary goal of teaching is not to transfer knowledge per se, but instead to light the fire in the student so they not only want to pursue learning but have the tools necessary to do so. In a sense, our goal should be to make teachers obsolete — to transform every student into an autodiadact and transform the teacher into a mentor role.

My solution is to replace the SOL with a weekly two question feedback from the students. In the modern age, this can be done via computer very quickly and the results evaluated rapidly. The trick is to know what to look for. The questions are:

  1. What is one new thing you learned in class this week?
  2. On a scale of 1 – 10 (1 being “yawn” 10 being “heck yea!”), how excited are you to return to class on Monday? Why?

For the first question, we don’t actually care what the student says, we care how the student says it. Does the reply indicate they are fully engaged? Does the reply show that the student actually learned something useful? What are the changes from last week and the week before?

The second question reinforces the engagement issue and allows a bit of quantitative analysis along with evaluation of the why portion.

The feedback is more or less instantaneous. By supplying the summaries to the teacher, they can get feedback on what’s happening in the classroom. Oh, and give the teachers enough time to evaluate the information. This isn’t an assembly line. Kids can learn more in 2 hours when engaged than they can in 2 days when they aren’t.

It’s not a perfect solution, but I can make the argument that something along these lines, which can be completed in under five minutes a week, will give better feedback and course correction to the school system than the 20 some hours of annual testing which all students, teachers, and administrators view with loathing.