Introducing OpsKarma

Most engineers have a bad interview story. You know the answer, more or less, but the clock is running, someone is watching you type, and halfway through you lose the thread. Then there's the opposite story: the person who interviewed brilliantly, because they'd drilled the same fifty problems for a month, and who then had no idea where to start the first time a real service fell over.

Both stories point at the same thing. Interviews mostly measure how well you interview. They don't tell you much about how someone works when something is actually broken, or how they think through a problem they haven't seen before.

We started OpsKarma because we wanted a place to practise the real thing, get honest feedback on it, and end up with something to show for it. This post is about what we've built so far, what it's good at, where it still falls short, and what we're working on next.

Starting with the interview

We had to start somewhere, and interviews felt like the right place. They're the part of the job search people stress about most, and they're where a lot of bad practice habits come from. So the first thing we've opened is an interview whiteboard.

It works the way a real interview does. You get a question, a blank board and a Run button.

There are 27 questions right now. Some are the classics you'd expect, like finding two numbers that add up to a target or counting islands in a grid. Others are things people don't practise enough: writing a debounce function that still behaves when it's called twice in a row, designing a rate limiter for 20,000 requests a second, or working out why a site started throwing errors right after a deploy. There are also a few behavioural questions, because those trip people up too.

Each question gives you the statement, a few worked examples, the limits you're working within, and a short list of what a good answer should cover. You write your answer however you'd write it in front of someone. Code in whatever language you like, pseudocode, a few lines of explanation, a rough diagram made of boxes and arrows. It's a whiteboard, so it's plain text, and that's on purpose.

What happens when you press Run

When you press Run, an AI judge reads your answer and scores it out of 100.

We didn't want a score that just said 72 and left you guessing. So every question has a handful of checks, which are the things a good interviewer would be looking for. For an algorithms question that's roughly: did you pick a method that fits the limits and explain why it works, does your code match that method, did you get the time and memory cost right, did you think about edge cases, and did you walk through an example. Each check is worth part of the 100.

The judge marks each one as met, partly met or not met, and writes one line about why. Then it gives you a few short points on what to work on next. The idea is that you can read the result the same way you'd read notes from a friend who just watched you answer.

Getting this to feel fair took longer than building it. Our first version was too strict with answers that were correct but written differently from what it expected, like a solution in pseudocode, or one that used a different method that also works. It was also too easy on answers that were correct but far too slow for the question's limits. So we graded a set of answers by hand, compared our scores with the judge's, and changed its rules until the two mostly agreed. It still isn't perfect, and we'd rather say that plainly. When you think it got something wrong, there's a "Report a problem" link under every result, and it comes straight to us.

The coach that won't do your homework

There's also an AI coach next to the whiteboard. You can ask it what you're missing, ask for a hint, or ask it to explain something that isn't clicking.

What it won't do is write the answer for you. We went back and forth on this. It would have been easy to let it hand out solutions, and people would probably like that in the moment. But then you'd paste in the answer, get a good score, and learn nothing. So if you ask for code, it tells you no and asks you a question to think about instead. It can be a little annoying. That's sort of the point.

What it can't do yet

A few honest limits.

The judge reads your answer, it doesn't run your code. It's good at telling whether your reasoning holds up, but it can miss a subtle bug the way a human reviewer sometimes would.

There are 27 questions. That's enough to practise properly, but it's not a huge library yet. We'd rather add questions slowly and make each one good than fill the list and fix them later.

And it's free, with a daily limit: 10 judged answers and 25 coach messages a day. The judge isn't free for us to run, so this keeps it sustainable while we're small. We'll be upfront about it before that changes.

What we're working on next

The whiteboard is the first piece. The bigger idea behind OpsKarma is practising the work itself, not just talking about it, and that's what we're building next. None of this is open yet, and we'll open each part when it actually works rather than set dates we can't keep.

Real servers. This is the part we're most excited about. You get a real Linux machine that's been broken on purpose. Maybe a service won't start, or a port isn't reachable, or a worker died and the queue behind it keeps growing. You get a terminal and you figure it out, the same way you would on call. When you think it's fixed, a check from outside the machine confirms whether it really is. If you're aiming for backend, DevOps or SRE work, this is much closer to the job than any interview question.

Projects. A real code base and a ticket, worked on in VS Code in your browser. Before you start, you choose how much AI help you want, from none at all to a full assistant, and that choice shows up on your result. Using AI well is part of the job now, and we'd rather be open about how work was done than pretend it doesn't happen.

A page that shows your work. Everything you do on OpsKarma, questions, servers you fixed, projects, can come together on one page you can share. Not a certificate. Just the work itself, and how you explained it.

Hiring. Eventually, teams will be able to use the same pieces to test candidates, see what each person actually did, step by step, and still have a person on the team make the decision. A lot of hiring today is guesswork, and we think it can be a bit less of that.

Try it

If you're preparing for interviews, pick a question at opskarma.com/learn and answer it like someone is watching. Two sum is a good warm-up if you're not sure where to start.

And if something feels off, a score you disagree with, a question that's unclear, anything at all, tell us. We read everything that comes in, and at this stage it shapes what we build next more than anything else.