- Engineering / AI
- Remote
- Hourly / Part-time
About the Role
Skill Stand In is looking for experienced AI Engineers to evaluate AI-generated work involving large language models, agents, evaluation systems, and intelligent workflows.
This role is for practitioners who can judge whether AI output meets professional standards. You will not design or build products.
What You’ll Do
- Review AI-generated applications, prompts, agent workflows, and evaluation setups for technical correctness.
- Judge whether retrieval, tool use, orchestration, and monitoring choices would work in a production setting.
- Assess model outputs, traces, and scoresheets for accuracy, reliability, and failure modes.
- Identify hallucinated APIs, unsafe tool use, weak evaluation metrics, and incomplete test coverage.
- Provide structured feedback that explains defects and what a stronger engineering solution would require.
- Band AI work against practitioner standards and defend your scoring in writing.
- Work independently and asynchronously to deliver high-quality evaluations.
What We’re Looking For
- Strong software engineering experience with Python, TypeScript, JavaScript, Java, or similar languages.
- Hands-on experience with production AI/ML applications, used to judge quality and correctness.
- Familiarity with LLMs, AI agents, prompt engineering, or machine learning workflows.
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Strong understanding of software architecture, APIs, databases, and distributed systems.
- Ability to explain technical tradeoffs clearly in writing.
Helpful Technical Backgrounds
Examples include:
- Python
- TypeScript
- LLMs
- AI Agents
- AWS
- GCP
- Azure
- APIs
- Evaluation
Experience with every technology is not required.
Apply as Senior AI Engineer
Opens the application form in a new tab.