Machine Learning Engineer
Accountable for deploying and operating machine learning models in production with monitoring, rollback and release safeguards.
3tasks in this job
10skills it uses
9tools named
The skills this job leans on most
- 1Designing and running evaluations of AI systemsNeeded in 2 of 3 tasks
- 2Analysing error patterns and failure modes in models and AI systemsNeeded in 2 of 3 tasks
- 3Monitoring model fitness and driftNeeded in 2 of 3 tasks
- 4Setting and applying a release and rollback bar for models and AI systemsNeeded in 2 of 3 tasks
- 5Exercising human oversight of automated and AI outputsCore to this role
ToolsMicrosoft Azure (portal, Cost Management, Monitor)DatadogMLflowGit / GitHubPythonAzure DevOpsAzure Machine Learningscikit-learnGitHub Actions
All 3 tasks, where people get stuck and all 10 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Deliver a tested software or API change to a data or AI service
Where people get stuck
- Exercising human oversight of automated and AI outputs
- Setting and applying a release and rollback bar for models and AI systems
All skills
- Debugging data and application services from evidence
- Gating releases on evidence with rollback ready
- Identifying keys and testing referential integrity
- Preparing change records with impact assessment and rollback
- Repairing data defects reversibly under owner authority
- Testing changes to data services and business systems for business meaning and downstream impact
- Tracing data lineage upstream and downstream
- Understanding and interrogating data and application APIs
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
Paths to start
Coming soonAI Evaluation and MonitoringMachine Learning: from Data to Validated Models and Decisions
4tasks in this job
84skills it uses
13tools named
The skills this job leans on most
- 1Analysing error patterns and failure modes in models and AI systemsNeeded in 3 of 4 tasks
- 2Designing and running evaluations of AI systemsNeeded in 2 of 4 tasks
- 3Monitoring model fitness and driftNeeded in 2 of 4 tasks
- 4Setting and applying a release and rollback bar for models and AI systemsNeeded in 2 of 4 tasks
- 5Exercising human oversight of automated and AI outputsCore to this role
ToolsAzure Machine LearningAmazon SageMakerGoogle Vertex AIMicrosoft Azure (portal, Cost Management, Monitor)DatadogKubernetes+ 7 more
All 4 tasks, where people get stuck and all 84 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Lead an incident through containment, restoration and communication
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Deliver a tested software or API change to a data or AI service
Where people get stuck
- Escalating unresolved risks with evidence and clear ownership
- Running incident command: severity, timeline and status updates
- Preserving incident evidence and timeline
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Monitoring model fitness and drift
- Setting and applying a release and rollback bar for models and AI systems
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Bounding and securing AI systems' tool and data access
- Building bounded, secure data-service APIs
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Constructing clear and honest charts
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing and interpreting hypothesis tests
- Designing efficient and reproducible investigations
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating and improving chart accessibility
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Identifying keys and testing referential integrity
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether uncertainty changes the decision
- Mapping proposed process changes and control impacts
- Monitoring data quality and freshness in production
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Selecting charts for evidence and decisions
- Setting and reviewing action thresholds for scores and model predictions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- Azure Machine Learning
- Amazon SageMaker
- Google Vertex AI
- Microsoft Azure (portal, Cost Management, Monitor)
- Datadog
- Kubernetes
- Python
- MLflow
- Azure DevOps
- GitHub Actions
- scikit-learn
- Docker
- Git / GitHub
Paths to start
Skill Path
Exploratory Data Analysis and Pattern Discovery
Describe a distribution honestly and decide which unusual records need checking before anyone acts.
- Modules
- 5 open
- Practice data
- Raket Parts
Skill Path
Responsible AI Governance and Assurance
Govern and assure AI across its lifecycle including third-party AI and redress.
- Modules
- 5 open
- Practice data
- Raket Parts
Coming soonAI Evaluation and MonitoringMachine Learning: from Data to Validated Models and Decisions
5tasks in this job
89skills it uses
11tools named
The skills this job leans on most
- 1Escalating unresolved risks with evidence and clear ownershipNeeded in 5 of 5 tasks
- 2Analysing error patterns and failure modes in models and AI systemsNeeded in 3 of 5 tasks
- 3Designing and running evaluations of AI systemsNeeded in 2 of 5 tasks
- 4Monitoring model fitness and driftNeeded in 2 of 5 tasks
- 5Running incident command: severity, timeline and status updatesNeeded in 1 of 5 tasks
ToolsAzure Machine LearningAmazon SageMakerGoogle Vertex AIMicrosoft Azure (portal, Cost Management, Monitor)MLflowAzure DevOps+ 5 more
All 5 tasks, where people get stuck and all 89 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Lead an incident through containment, restoration and communication
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Deliver a tested software or API change to a data or AI service
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Escalating unresolved risks with evidence and clear ownership
- Exercising human oversight of automated and AI outputs
- Setting and applying a release and rollback bar for models and AI systems
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Escalating unresolved risks with evidence and clear ownership
- Monitoring model fitness and drift
- Running incident command: severity, timeline and status updates
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Bounding and securing AI systems' tool and data access
- Building bounded, secure data-service APIs
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Constructing clear and honest charts
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing and interpreting hypothesis tests
- Designing efficient and reproducible investigations
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating and improving chart accessibility
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether uncertainty changes the decision
- Mapping proposed process changes and control impacts
- Monitoring data quality and freshness in production
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- Azure Machine Learning
- Amazon SageMaker
- Google Vertex AI
- Microsoft Azure (portal, Cost Management, Monitor)
- MLflow
- Azure DevOps
- GitHub Actions
- scikit-learn
- Python
- Git / GitHub
- Docker
Paths to start
Skill Path
Data Literacy for Business Decisions
Judge whether a business claim from a table, KPI, chart or dashboard really holds up.
- Modules
- 5 open
- Practice data
- Raket Parts
Skill Path
Exploratory Data Analysis and Pattern Discovery
Describe a distribution honestly and decide which unusual records need checking before anyone acts.
- Modules
- 5 open
- Practice data
- Raket Parts
Coming soonAI Evaluation and MonitoringData Product Reliability: Release Gates, Service Levels and Incidents
About the role
Accountable for deploying and operating machine learning models in production with monitoring, rollback and release safeguards. Decides how a model is packaged, released and watched for drift or failure. Differs from the Data Scientist who owns modelling method and the AI Engineer who builds on existing foundation models.
Also called MLOps Engineer, ML Engineer