Remote Model Evaluation Developer Jobs

Requirements

• Define and own evaluation frameworks for GenAI outputs
• Design curate and govern test datasets and benchmarks
• Support development of automated evaluation pipelines
• Manage delivery through well-defined work packages
• Engage senior stakeholders to understand strategic priorities
• Work in the public interest protecting 40 million UK consumers

Benefits

• Flexible Hours
• Gym Membership
• Health Insurance
• Home Office Budget
• Learning Budget
• Performance Bonus
• Remote Work

Skills

Requirements

• Manage individual work streams to support the successful delivery and implementation of bespoke AI solutions
• Work side-by-side with key stakeholders to understand critical mission challenges and user needs
• Conduct open-source analysis to support the development of cutting-edge AI tools
• Translate complex operational requirements into clear user stories for delivery and engineering teams
• Design and test workflows that support user-centric decision-making in high-stakes environments
• Collaborate closely with technical teams to ensure projects run smoothly and solutions meet customer needs
• Contribute to internal thought leadership, strategy, and capability development

Benefits

• 401k Matching
• Certification Support
• Flexible Hours
• Health Insurance
• Home Office Budget
• Learning Budget
• Paid Time Off
• Remote Work

Skills

Requirements

• Proficiency in frameworks for auditing models
• Conduct audits on the model lifecycle from training through deployment
• Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks
• Recommend remediation strategies
• Develop and maintain collaborative working relationships with stakeholders

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Remote Work

Skills

Requirements

• - Proficiency in frameworks for auditing models, including criteria like robustness, fairness, interpretability, alignment, and compliance.
• - Model Evaluation & Audit Frameworks: conduct audits on the model lifecycle from training through deployment and monitoring, ensuring compliance with quality, performance, fairness, and risk-management standards.
• - Risk Identification & Mitigation: Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks, and recommend remediation strategies.
• - Measurement Metrics & Statistical Validation: Define and assess model performance metrics (accuracy, precision/recall, F1, calibration, robustness, fairness metrics), measurement of hallucination rates in LLMs, bias/fairness quantification, confidence scoring, and stability analyses.
• - Communication & Collaboration: Develop and maintain collaborative working relationships with stakeholders, including data partners and owners across different business verticals.

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Remote Work

Skills

Requirements

• Proficiency in frameworks for auditing models
• Conduct audits on the model lifecycle from training through deployment
• Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks
• Recommend remediation strategies
• Develop and maintain collaborative working relationships with stakeholders

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Remote Work
• Stock Options

Skills

Requirements

• - Proficiency in frameworks for auditing models, including criteria like robustness, fairness, interpretability, alignment, and compliance.
• - Model Evaluation & Audit Frameworks: conduct audits on the model lifecycle from training through deployment and monitoring, ensuring compliance with quality, performance, fairness, and risk-management standards.
• - Risk Identification & Mitigation: Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks, and recommend remediation strategies.
• - Measurement Metrics & Statistical Validation: Define and assess model performance metrics (accuracy, precision/recall, F1, calibration, robustness, fairness metrics), measurement of hallucination rates in LLMs, bias/fairness quantification, confidence scoring, and stability analyses.
• - Communication & Collaboration: Develop and maintain collaborative working relationships with stakeholders, including data partners and owners across different business verticals.

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Performance Bonus
• Remote Work
• Stock Options

Skills

Requirements

• - Proficiency in frameworks for auditing models, including criteria like robustness, fairness, interpretability, alignment, and compliance.
• - Model Evaluation & Audit Frameworks: conduct audits on the model lifecycle from training through deployment and monitoring, ensuring compliance with quality, performance, fairness, and risk-management standards.
• - Risk Identification & Mitigation: Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks, and recommend remediation strategies.
• - Measurement Metrics & Statistical Validation: Define and assess model performance metrics (accuracy, precision/recall, F1, calibration, robustness, fairness metrics), measurement of hallucination rates in LLMs, bias/fairness quantification, confidence scoring, and stability analyses.
• - Communication & Collaboration: Develop and maintain collaborative working relationships with stakeholders, including data partners and owners across different business verticals.

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Performance Bonus
• Remote Work

Skills

Requirements

• Proficiency in frameworks for auditing models, including criteria like robustness, fairness, interpretability, alignment, and compliance.
• Familiarity with emerging LLM auditing methodologies such as LLMAuditor (probe generation/answering cycles, human-in-the-loop assessments).
• Conduct audits on the model lifecycle from training through deployment and monitoring, ensuring compliance with quality, performance, fairness, and risk-management standards.
• Identify model vulnerabilities including bias, fairness violations, harmful hallucinations, security risks, and recommend remediation strategies.
• Define and assess model performance metrics (accuracy, precision/recall, F1, calibration, robustness, fairness metrics), measurement of hallucination rates in LLMs, bias/fairness quantification, confidence scoring, and stability analyses.

Benefits

• 401k Matching
• Health Insurance
• Paid Time Off
• Performance Bonus
• Remote Work
• Stock Options

Skills

Requirements

• Bachelor’s degree or equivalent practical experience
• 8 years of experience in software development
• 7 years of experience leading technical project strategy, ML design, and working with ML infrastructure
• 5 years of experience with one or more of the following: Speech/audio, reinforcement learning, ML infrastructure, or specialization in another ML field
• 5 years of experience with design and architecture; and testing/launching software products

Benefits

• 401k Matching
• Certification Support
• Flexible Hours
• Gym Membership
• Health Insurance
• Home Office Budget
• Learning Budget
• Paid Time Off

Skills