Lead AI & ML Engineer
Accountable for how machine learning models and AI systems are released, evaluated and monitored across several teams.
3tasks in this job
31skills it uses
9tools named
The skills this job leans on most
- 1Analysing error patterns and failure modes in models and AI systemsNeeded in 3 of 3 tasks
- 2Designing and running evaluations of AI systemsNeeded in 2 of 3 tasks
- 3Setting and applying a release and rollback bar for models and AI systemsNeeded in 2 of 3 tasks
ToolsMicrosoft Azure (portal, Cost Management, Monitor)Azure Machine LearningAzure OpenAI Service / Azure AI FoundryMLflowAmazon SageMakerAzure DevOpsGit / GitHubLangChain / LangGraphPython
All 3 tasks, where people get stuck and all 31 skills
Tasks
- Lead an incident through containment, restoration and communication
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Design and bound a tool-using agentic AI workflow
Where people get stuck
- Reviewing human- and AI-generated code changes for correctness and risk
- Designing testable data controls
- Escalating unresolved risks with evidence and clear ownership
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Setting and applying a release and rollback bar for models and AI systems
- Applying data classification and handling rules
- Assessing data-driven fairness and bias
- Bounding and securing AI systems' tool and data access
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Escalating unresolved risks with evidence and clear ownership
- Evaluating ML generalisation and fitness
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Independently checking analytical conclusions and claims
- Interpreting policy scope and delegated authority limits
- Judging the materiality of defects and deficiencies for the intended use
- Monitoring model fitness and drift
- Preserving incident evidence and timeline
- Protecting sensitive data in AI-assisted work
- Reading and tracing unfamiliar code
- Recognising cybersecurity threats to data work
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Reviewing human- and AI-generated code changes for correctness and risk
- Setting and reviewing action thresholds for scores and model predictions
- Testing competing explanations with discriminating evidence
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
Paths to start
Coming soonAI Evaluation and MonitoringMachine Learning: from Data to Validated Models and Decisions
6tasks in this job
108skills it uses
11tools named
The skills this job leans on most
- 1Escalating unresolved risks with evidence and clear ownershipNeeded in 6 of 6 tasks
- 2Reviewing human- and AI-generated code changes for correctness and riskNeeded in 5 of 6 tasks
- 3Analysing error patterns and failure modes in models and AI systemsNeeded in 5 of 6 tasks
- 4Designing and running evaluations of AI systemsNeeded in 4 of 6 tasks
- 5Designing testable data controlsCore to this role
ToolsMicrosoft Azure (portal, Cost Management, Monitor)LangfuseOneTrustAzure Machine LearningAzure OpenAI Service / Azure AI FoundryMLflow+ 5 more
All 6 tasks, where people get stuck and all 108 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Lead an incident through containment, restoration and communication
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Design and bound a tool-using agentic AI workflow
- Develop and evaluate an LLM-based application against an agreed evaluation bar
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Reviewing human- and AI-generated code changes for correctness and risk
- Escalating unresolved risks with evidence and clear ownership
- Setting and applying a release and rollback bar for models and AI systems
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Escalating unresolved risks with evidence and clear ownership
- Reviewing human- and AI-generated code changes for correctness and risk
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Assigning and operating data ownership and stewardship
- Bounding and securing AI systems' tool and data access
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Constructing clear and honest charts
- Controlling disclosure risk in shared data and outputs
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and evaluating retrieval for AI systems
- Designing and interpreting hypothesis tests
- Designing decision rights and dispute routes for data and AI work
- Designing efficient and reproducible investigations
- Designing enforceable data governance policies
- Designing implementable change plans
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating and improving chart accessibility
- Evaluating fitness and full cost of AI-assisted work
- Evaluating responsible AI use risks
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Governing shared data definitions
- Identifying and prioritising critical data elements
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Influencing change without formal authority
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Judging whether uncertainty changes the decision
- Mapping proposed process changes and control impacts
- Mapping stakeholders' decisions, constraints and needs
- Monitoring data quality and freshness in production
- Monitoring model fitness and drift
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Specifying and iterating AI task instructions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Working through stakeholder conflict over evidence and responsibility
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing testable work standards and review checklists
- Writing verifiable requirements and acceptance criteria
All tools
- Microsoft Azure (portal, Cost Management, Monitor)
- Langfuse
- OneTrust
- Azure Machine Learning
- Azure OpenAI Service / Azure AI Foundry
- MLflow
- LangChain / LangGraph
- Python
- Amazon Bedrock
- Docker
- Kubernetes
Paths to start
Coming soonAI Evaluation and MonitoringData Product Reliability: Release Gates, Service Levels and IncidentsMachine Learning: from Data to Validated Models and Decisions
7tasks in this job
116skills it uses
12tools named
The skills this job leans on most
- 1Escalating unresolved risks with evidence and clear ownershipNeeded in 7 of 7 tasks
- 2Reviewing human- and AI-generated code changes for correctness and riskNeeded in 6 of 7 tasks
- 3Analysing error patterns and failure modes in models and AI systemsNeeded in 6 of 7 tasks
- 4Designing and running evaluations of AI systemsNeeded in 4 of 7 tasks
- 5Designing testable data controlsNeeded in 2 of 7 tasks
ToolsAzure Machine LearningAmazon SageMakerMicrosoft Power BIAzure OpenAI Service / Azure AI FoundryAmazon BedrockMicrosoft Azure (portal, Cost Management, Monitor)+ 6 more
All 7 tasks, where people get stuck and all 116 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Evaluate an AI use case for suitability, value and risk before adoption
- Lead an incident through containment, restoration and communication
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Maintain a data and AI risk register and record risk treatment and acceptance
- Classify an AI system's risk and assure its documentation, evaluation and oversight evidence
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Reviewing human- and AI-generated code changes for correctness and risk
- Escalating unresolved risks with evidence and clear ownership
- Setting and applying a release and rollback bar for models and AI systems
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Designing testable data controls
- Escalating unresolved risks with evidence and clear ownership
- Reviewing human- and AI-generated code changes for correctness and risk
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Assigning and operating data ownership and stewardship
- Bounding and securing AI systems' tool and data access
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Constructing clear and honest charts
- Controlling disclosure risk in shared data and outputs
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and evaluating retrieval for AI systems
- Designing and interpreting hypothesis tests
- Designing decision rights and dispute routes for data and AI work
- Designing efficient and reproducible investigations
- Designing enforceable data governance policies
- Designing implementable change plans
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating and improving chart accessibility
- Evaluating fitness and full cost of AI-assisted work
- Evaluating responsible AI use risks
- Exercising human oversight of automated and AI outputs
- Explaining control types, dependencies and compensating controls
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Governing shared data definitions
- Identifying and prioritising critical data elements
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Influencing change without formal authority
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Judging whether uncertainty changes the decision
- Locating and acting on personal data for individual rights requests
- Mapping personal data processing and flows
- Mapping proposed process changes and control impacts
- Mapping stakeholders' decisions, constraints and needs
- Monitoring data quality and freshness in production
- Monitoring model fitness and drift
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Resolving consent and suppression status across systems
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Specifying and iterating AI task instructions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Working through stakeholder conflict over evidence and responsibility
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing testable work standards and review checklists
- Writing verifiable requirements and acceptance criteria
All tools
- Azure Machine Learning
- Amazon SageMaker
- Microsoft Power BI
- Azure OpenAI Service / Azure AI Foundry
- Amazon Bedrock
- Microsoft Azure (portal, Cost Management, Monitor)
- Langfuse
- MLflow
- Azure DevOps
- OneTrust
- Git / GitHub
- Docker
Paths to start
Coming soonAI Evaluation and MonitoringData Product Reliability: Release Gates, Service Levels and IncidentsMachine Learning: from Data to Validated Models and Decisions
About the role
Accountable for how machine learning models and AI systems are released, evaluated and monitored across several teams. Sets release, evaluation and guardrail standards and reviews other engineers' models and AI systems without line management. Covers both trained models and applications built on foundation models, with judgement on when a system is safe and reliable enough to release.
Also called Lead Machine Learning Engineer, Lead AI Engineer, Lead MLOps Engineer