AI Engineer
Accountable for building, evaluating and operating LLM-based applications, retrieval systems and bounded agentic workflows on existing models.
3tasks in this job
16skills it uses
8tools named
The skills this job leans on most
- 1Designing and running evaluations of AI systemsNeeded in 2 of 3 tasks
- 2Analysing error patterns and failure modes in models and AI systemsNeeded in 2 of 3 tasks
- 3Designing and evaluating retrieval for AI systemsNeeded in 1 of 3 tasks
ToolsLangfuseMicrosoft Azure (portal, Cost Management, Monitor)Git / GitHubDockerPythonLangChain / LangGraphAzure OpenAI Service / Azure AI FoundryJupyter (Notebook / JupyterLab)
All 3 tasks, where people get stuck and all 16 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Deliver a tested software or API change to a data or AI service
- Develop and evaluate an LLM-based application against an agreed evaluation bar
Where people get stuck
- Designing AI-assisted workflows with bounded tools and human authority
- Bounding and securing AI systems' tool and data access
- Evaluating fitness and full cost of AI-assisted work
All skills
- Assessing supplier dependency, data-handling and exit risk
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Identifying keys and testing referential integrity
- Judging whether and how to use AI for a task
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Protecting sensitive data in AI-assisted work
- Repairing data defects reversibly under owner authority
- Testing changes to data services and business systems for business meaning and downstream impact
- Tracing data lineage upstream and downstream
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
Paths to start
Coming soonAI Evaluation and MonitoringApplied AI at Work
8tasks in this job
71skills it uses
14tools named
The skills this job leans on most
- 1Bounding and securing AI systems' tool and data accessNeeded in 4 of 8 tasks
- 2Designing and running evaluations of AI systemsNeeded in 4 of 8 tasks
- 3Designing and evaluating retrieval for AI systemsNeeded in 2 of 8 tasks
- 4Designing AI-assisted workflows with bounded tools and human authorityNeeded in 2 of 8 tasks
- 5Evaluating fitness and full cost of AI-assisted workNeeded in 2 of 8 tasks
ToolsServiceNowLangfuseAzure AI SearchMicrosoft Azure (portal, Cost Management, Monitor)Jupyter (Notebook / JupyterLab)Git / GitHub+ 8 more
All 8 tasks, where people get stuck and all 71 skills
Tasks
- Review a proposed use, reuse, disclosure of or access to data and record the permitted-use decision
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Investigate application or API faults
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Deliver a tested software or API change to a data or AI service
- Design and bound a tool-using agentic AI workflow
- Develop and evaluate an LLM-based application against an agreed evaluation bar
- Evaluate and recommend a data, analytics or AI product or vendor
Where people get stuck
- Escalating unresolved risks with evidence and clear ownership
- Running incident command: severity, timeline and status updates
- Preserving incident evidence and timeline
All skills
- Bounding and securing AI systems' tool and data access
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and evaluating retrieval for AI systems
- Designing and running evaluations of AI systems
- Evaluating fitness and full cost of AI-assisted work
- Analysing error patterns and failure modes in models and AI systems
- Analysing event sequences, states and durations
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Building bounded, secure data-service APIs
- Building data protection by design into data products and pipelines
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Comparing groups
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing and interpreting hypothesis tests
- Designing efficient and reproducible investigations
- Diagnosing incorrect query and calculation results
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating responsible AI use risks
- Exploring data to find patterns that survive validation
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Identifying keys and testing referential integrity
- Interpreting averages, weighted measures and mix effects
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Monitoring data quality and freshness in production
- Monitoring model fitness and drift
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Specifying and iterating AI task instructions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- ServiceNow
- Langfuse
- Azure AI Search
- Microsoft Azure (portal, Cost Management, Monitor)
- Jupyter (Notebook / JupyterLab)
- Git / GitHub
- Docker
- Python
- LangChain / LangGraph
- Amazon Bedrock
- Azure OpenAI Service / Azure AI Foundry
- Pinecone
- Claude
- Hugging Face
Paths to start
Coming soonAI Evaluation and MonitoringDesigning Controlled Automations and AI-assisted WorkflowsApplied AI at Work
8tasks in this job
99skills it uses
7tools named
The skills this job leans on most
- 1Designing and running evaluations of AI systemsNeeded in 5 of 8 tasks
- 2Bounding and securing AI systems' tool and data accessNeeded in 4 of 8 tasks
- 3Evaluating fitness and full cost of AI-assisted workNeeded in 4 of 8 tasks
- 4Designing AI-assisted workflows with bounded tools and human authorityNeeded in 3 of 8 tasks
- 5Designing and evaluating retrieval for AI systemsNeeded in 2 of 8 tasks
ToolsLangfuseOneTrustMicrosoft Azure (portal, Cost Management, Monitor)Jupyter (Notebook / JupyterLab)LangChain / LangGraphPython+ 1 more
All 8 tasks, where people get stuck and all 99 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Evaluate an AI use case for suitability, value and risk before adoption
- Lead an incident through containment, restoration and communication
- Design and bound a tool-using agentic AI workflow
- Develop and evaluate an LLM-based application against an agreed evaluation bar
- Evaluate and recommend a data, analytics or AI product or vendor
- Classify an AI system's risk and assure its documentation, evaluation and oversight evidence
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Evaluating fitness and full cost of AI-assisted work
- Escalating unresolved risks with evidence and clear ownership
All skills
- Bounding and securing AI systems' tool and data access
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and evaluating retrieval for AI systems
- Designing and running evaluations of AI systems
- Evaluating fitness and full cost of AI-assisted work
- Analysing error patterns and failure modes in models and AI systems
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Building bounded, secure data-service APIs
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Constructing clear and honest charts
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing and interpreting hypothesis tests
- Designing efficient and reproducible investigations
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating and improving chart accessibility
- Evaluating responsible AI use risks
- Explaining control types, dependencies and compensating controls
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Judging whether uncertainty changes the decision
- Mapping proposed process changes and control impacts
- Monitoring data quality and freshness in production
- Monitoring model fitness and drift
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Specifying and iterating AI task instructions
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- Langfuse
- OneTrust
- Microsoft Azure (portal, Cost Management, Monitor)
- Jupyter (Notebook / JupyterLab)
- LangChain / LangGraph
- Python
- Hugging Face
Paths to start
Coming soonAI Evaluation and MonitoringDesigning Controlled Automations and AI-assisted WorkflowsData Product Reliability: Release Gates, Service Levels and Incidents
About the role
Accountable for building, evaluating and operating LLM-based applications, retrieval systems and bounded agentic workflows on existing models. Decides the prompts, retrieval design, evaluation sets and guardrails that make model behaviour acceptable for a use. Differs from the Machine Learning Engineer, who trains and operates models rather than building on foundation models.
Also called LLM Engineer, GenAI Engineer, AI Application Engineer