When I'm not writing award entries, I'm deep diving into award processes. If I'm going to invest my client's time and money into entering something, it has to mean something. Every week, I go down the rabbit hole researching awards and have noticed almost weekly, there's new award on the scene. Awards have become an industry and sadly, many of these new awards seem to be more funnel fillers.

Once you get past the free entry, you find that there's a cost to buy the trophy or if you want to be able to say you're a finalist, you have to buy the marketing package. When you dig deep enough, you can see how little thought there has been into making the process credible, robust, logical and clear.  

A new award. A new day
Today I came across another new award.  It's for coaches and I work with a lot of coaches so I checked it out. Ever curious, I read every page to see if it's worth my clients' while.

It had a few things in common with this new crop of awards: a disturbing lack of clear criteria, judging process and questions that sounded like every other award out there. 

What caught my eye was this line: free, judged by our independent AI panel. It says judging is merit over popularity.

AI and judging

I've a few bugbears with award judging, and a big one is the lack of transparency around judging criteria. This often means human judges have to lean into subjective decisions because the award program doesn't clearly map out what excellence looks like or give the judges a rubric to guide them (and no, the questions don't count). Add in unconscious bias, lots of entries to get through and actual bias, and it's not always the best business that wins.

This particular award makes a big deal of its independent judging panel (I would've thought that's a given). Each entry is scored by three AI models, Claude, GPT-4 (not even the latest version) and Gemini, with the average taken as the final score.

The problem with this is AI judging self-reported claims can't check whether any of it's true, so the person who writes the most confident entry wins.

AI does have bias. Most models are trained on polished, mostly American English so if English is your second language, or you use Aussie turns of phrase, you can be marked down for sounding like you. Research on AI judges shows it tends to favour longer, more formal sentence structure, and can lean towards writing that sounds like it; forget about writing with personality. It can also be inconsistent. Go on, run your entry through twice and I bet you'll get different feedback.

They say there's a weighted rubric, but it's nowhere to be seen. Nobody entering knows what they're being judged on or why.

To be fair, one of the questions has teeth. The rest are all about belief. How do you judge that? It's like my Year 11 English teacher asking the class to analyse a poem and then marking our opinions down. But at least the English teacher had read the poem. AI has no idea whether your impact is real, so it's marking the confidence of your opinion, not the substance of it.

I wrote about this recently in Kochie's Business Builders, looking at the impact AI will have (is having) on awards. AI is changing awards and can make judging better: applying the same criteria to every entry, flagging claims with no proof, and giving us human judges more time for the deeper assessments.

Whether it's humans or AI judging, the rubric should be public. What's being judged, how each question is weighted, and what separates a strong answer from a weak one. How are entrants meant to shoot for gold if they have to guess?

Handing the whole kit and caboodle to three chatbots, keeping the marking guide and instructions a secret and calling it independent doesn't sit well with me.

Independent of what?

 

By Annette Densham

Keywords: AI, Personal Branding, Public Relations

Follow Us On

Become a Contributor Newsletter Signup