AI Evaluation Lead

Job Title: AI Evaluation Lead Division: Data, Technology and Innovation Department: AI Product Delivery Salary: National (Edinburgh and Leeds) ranging from £72,100 to £100,000 and London from £79,300 to £110,000 per annum (salary offered will be based on skills and experience) This role is graded as: Technical Specialist – Regulatory Your external recruitment contact is Benjamin via benjamin.paulon@fca.org.uk. Your internal recruitment contact is Lauren via Lauren.Pyrah@fca.org.uk Applications must be submitted through our online portal. Applications sent via social media or email will not be accepted. About the FCA and team We regulate financial services firms in the UK, to keep financial markets fair, thriving and effective. By joining us, you’ll play a key part in protecting consumers, driving economic growth, and shaping the future of UK finance services. The Data, Technology and Innovation (DTI) division enables the FCA to be a digital-first, data-led smart regulator by delivering a secure, agile, and cost-effective technology and data ecosystem that drives better decisions, transparency, and operational efficiency. Working alongside the wider AI Programme (which will continue to oversee/coordinate AI activity across the FCA), the department will partner with business leads to shape and deliver work in priority areas — Authorisations, SPC, EMO and Anti‑Money Laundering. Role responsibilities Define and own evaluation frameworks for GenAI outputs covering quality measures such as accuracy relevance robustness and hallucination rates Design curate and govern test datasets and benchmarks to ensure consistent model and solution performance assessment over time Support development of automated evaluation pipelines and operational reporting to embed assurance into delivery Identify assess and mitigate risks in model behaviour for example bias errors safety concerns and edge cases with clear escalation and control recommendations Manage delivery through well-defined work packages setting priorities operating standards and performance objectives including line management of Business Analysts Engage senior stakeholders to understand strategic priorities build a pipeline of scoped and prioritised projects and translate needs into analytics led solutions with clear business value Work in the public interest protecting 40 million UK consumers who rely on financial services and supporting long term economic growth from an industry contributing 12% of UK economic output Manage digital and data-led change by encouraging innovative experiments and working with senior stakeholders while empowering a diverse team to collaborate openly Skills required Minimum: Experience delivering analytics, data science and AI/ML initiatives, including defining success measures, evaluating model/product performance, and applying innovative problem-solving approaches Demonstrated experience leading people (line management, coaching, and performance reviews), capable of overseeing a portfolio of projects and adjusting delivery as priorities shift Effective stakeholder management skills, including working with senior colleagues to translate priorities into well-scoped, prioritised work with clear outcomes and measurable value Essential: Demonstrable experience designing and applying evaluation frameworks for GenAI ML solutions such as accuracy relevance robustness consistency and hallucination or error rates including defining clear acceptance thresholds Experience curating documenting and governing test datasets and benchmarks including version control to enable repeatable assessment and comparability over time Ability to identify assess and mitigate model risks including bias safety concerns data leakage harmful outputs and edge cases and to recommend appropriate controls and escalation routes Experience building or specifying automated evaluation and monitoring approaches such as scripted test runs scoring and dashboards or management information to embed assurance into delivery and ongoing operations Demonstrated analytical skills with the ability to interpret evaluation results communicate uncertainty and limitations translate complex technical concepts for senior stakeholders and make evidence-based recommendations to improve solution performance Consistent delivery discipline including planning and managing evaluation work packages prioritising across competing demands and ensuring outputs meet agreed standards and timelines Clear written documentation skills producing evaluation reports test plans and assurance artefacts that are audit ready and suitable for governance forums Benefits 28 days annual leave plus bank holidays Hybrid model where employees work a minimum of 40% in the office each month (expectation of 50% for senior leaders). Changing from September to a minimum of 50% in the office each month (expectation of 60% for Directors and Executive Directors) Non-contributory pension (8–12% depending on age) and life assurance at eight times your salary Private healthcare with Bupa, income protection, and 24/7 Employee Assistance 35 hours of paid volunteering annually A flexible benefits scheme designed around your lifestyle For a full list of our benefits, and our recruitment process as a whole visit our benefits page. Our values & culture Our colleagues are the key to our success as a regulator. We are committed to fostering a diverse and inclusive culture: one that’s free from discrimination and bias, celebrates difference, and supports colleagues to deliver at their best. We believe that our differences and similarities enable us to be a better organisation – one that makes better decisions, drives innovation, and delivers better regulation. If you require any adjustments due to a disability or condition, your recruiter is here to help - reach out for tailored support. We welcome diverse working styles and aim to find flexible solutions that suit both the role and individual needs, including options like part-time and job sharing where applicable. Disability Confident: our hiring approach We’re proud to be a Disability Confident Employer, and therefore, people or individuals with disabilities and long-term conditions who best meet the minimum criteria for a role will go through to the next stage of the recruitment process. In cases of high application volumes, we may progress applicants whose experience most closely matches the role’s key requirements. Useful information and timeline Advert Closing: 19th May CV Review/Shortlist: 22nd May First Stage Interviews W/C: 1st June Second Stage Interviews W/C: 8th June Your Recruiter will discuss the process in detail with you during screening for the role, therefore, please make them aware if you are going to be unavailable for any date during this time. At the FCA, we’re creating a fair and more resilient financial system. We’re establishing more transparent relationships between financial services and their customers, building trust in financial markets and protecting vulnerable consumers. Click here to learn more about the FCA.