Overview
AI Output Tester Jobs in Cape Town, Western Cape, South Africa at Blue Oak Consulting
Title: AI Output Tester
Company: Blue Oak Consulting
Location: Cape Town, Western Cape, South Africa
Where the role sits
The right person for this role understands that AI systems only provide value when their results are reliable in professional environments. Even if a response appears well-written, it may still fail to answer the prompt, hallucinate details, ignore specific directions, or adopt an inappropriate tone for a client workflow.
Our AI Output Tester is responsible for evaluating these results before they move further into production. You are not expected to be an engineer or build models; instead, the job requires close reading, comparing different responses, verifying instruction adherence, and identifying content that is confusing, wrong, or poorly structured.
We need someone fascinated by the potential of AI who also recognizes that success in this field relies on high levels of precision, sound judgment, and personal discipline.
Core responsibilities
- Identifying recurring flaws in model behavior to help the team refine instructions and internal processes
- Reviewing AI-generated outputs against written instructions and quality guidelines
- Checking whether a response actually answers the task it was given
- Comparing two or more outputs and explaining which one is stronger
- Flagging factual issues, unclear reasoning, missing context, weak structure, or inappropriate tone
- Testing prompts and recording how the model responds across different examples
- Writing short, clear notes explaining why an output passed, failed, or needed review
- Working through structured review tasks with consistency, even when the work is repetitive
- Escalating unclear cases rather than guessing when the correct judgment is not obvious
What we need from you
Essential
- A high level of consistency. Your decisions should remain stable across similar examples and not shift randomly during a session.
- Strong written English. You need to read carefully and explain your reasoning clearly.
- Good judgment when comparing written answers. You can tell the difference between an answer that sounds good and one that is actually useful.
- Attention to detail. You notice when an instruction has been missed, a claim is unsupported, or a response does not match the requested format.
- Comfort following detailed guidelines. You do not make up your own rules when a project has a defined review standard.
- Basic digital confidence. You should be comfortable working in online tools, spreadsheets, forms, shared documents, and simple task platforms.
- Patience with repetitive work. Some tasks will be interesting, some will be routine. Both need the same level of care.
Helpful but not required
- Experience using ChatGPT, Claude, Gemini, Perplexity, or similar tools
- Any previous work involving content review, editing, research, QA, data labeling, moderation, transcription, translation, or customer support
- Interest in prompt testing, AI evaluation, language quality, online research, or structured review work
- Basic spreadsheet skills for tracking examples, notes, or task outcomes
- Experience working remotely or independently on detailed written tasks
- A degree is not required if you can show careful thinking, clear writing, and reliable judgment
What we are not looking for
You do not need to be a machine learning engineer. You do not need to code. You do not need to have worked in AI before.
We are not looking for someone who simply likes using AI tools casually. We are looking for someone who can slow down, read the task properly, judge the output fairly, and explain what is wrong without overcomplicating it.
If your instinct is to check the instruction twice before submitting a decision, that is more useful in this role than trying to sound technical.
What you will get out of the role
- Practical exposure to how AI outputs are reviewed, tested, and improved
- A clear entry point into AI-related work without needing a technical background
- Experience with structured evaluation, prompt testing, and quality review workflows
- Feedback on how to make sharper, more consistent review decisions
- A role where careful reading, good judgment, and clear written reasoning are the skills that matter most