Part of my week now goes to marking homework for language models: reading their answers and deciding whether they're any good.
It's contract work evaluating and writing training data, in English. I rate responses for safety, accuracy and fluency, write down why, and write research questions and reference answers of my own. I can't talk about specific projects. I can talk about what the work has done to the way I read and write, because that part is mine.
Fluent is not the same as right
The first lesson: the smoothest answer is often the one to check hardest.
A model can produce a paragraph with clean grammar, a confident tone and a tidy conclusion, built on a premise that's false. Fluency is the easiest quality to see and the cheapest one to fake. Grade by how an answer feels and you'll reward the wrong ones.
So you learn to keep the questions apart: is it safe, is it correct, and is it well written. An answer can pass one and fail another. Keeping them separate is harder than it sounds, because a well-written wrong answer makes you want to believe it.
I now read pull requests, documentation and other people's posts the same way. Nice prose gets no credit for being true.
A rationale keeps you honest
Every rating comes with a written rationale. "This is bad" doesn't count. You have to name the claim that's wrong, the instruction that was ignored, the part that could cause harm.
Writing the rationale often changes the score. You start with a feeling that an answer is weak, try to point at something specific, and realize the problem was style, not substance. Or an answer feels fine until you try to justify that, and the justification falls apart by the second sentence.
It's the same reason a code review comment should say what's wrong and why. A judgment you can't explain is a preference wearing a lab coat.
Good questions are hard to write
Writing research questions for models turned out to be the hardest part.
A useful question has one defensible answer, but it can't be solved by looking up a single fact. It needs enough context to be fair and no stray detail that gives the answer away. And someone other than me has to be able to check it.
Most first drafts fail at least one of those. Two different answers could both be right. Or it's really a trivia question: one lookup and you're done. Or it leans on something I know and forgot to say.
It also made me far more suspicious of my own tickets. A bug report or a feature request is also a question someone else has to answer, and it fails in exactly the same ways.
You start seeing the habits
Read enough model output and its habits start to jump out. Lists of three. Closing lines that sound wise and say little. Sections that open by restating the question. "It's not X, it's Y." A dash in every other sentence. Words like "crucial" and "comprehensive" doing no work at all.
None of these is wrong on its own. People wrote this way long before models did, which is how the models learned it. The problem is density. When every paragraph has the same shape, the reader stops trusting any of them.
Once you see the pattern, you see it everywhere, including in your own drafts. That's uncomfortable and useful. My rule now: if a sentence would fit in anyone's post, it probably doesn't belong in mine.
The work made me calmer about AI, in both directions.
Models are good at things I used to assume needed a person: summarizing, restructuring, explaining a familiar idea clearly. They are also confidently wrong in ways that only show up when someone checks claim by claim.
A lot of a model's quality depends on people doing slow, careful, slightly boring work: reading every line, writing down why, and refusing to give credit for confidence. I like that part.