4tasks in this job
4skills it uses
10tools named
The skills this job leans on most
- 1Designing and running evaluations of AI systemsNeeded in 4 of 4 tasks
- 2Analysing error patterns and failure modes in models and AI systemsNeeded in 4 of 4 tasks
- 3Setting and applying a release and rollback bar for models and AI systemsNeeded in 3 of 4 tasks
ToolsLangfuseAmplitudeZendeskClaudeJiraOneTrustAzure OpenAI Service / Azure AI FoundryAmazon BedrockGoogle Vertex AIMicrosoft Excel
All 4 tasks, where people get stuck and all 4 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Evaluate an AI use case for suitability, value and risk before adoption
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Develop and evaluate an LLM-based application against an agreed evaluation bar
Where people get stuck
- Designing and evaluating retrieval for AI systems
- Judging whether and how to use AI for a task
- Evaluating fitness and full cost of AI-assisted work
All skills
Paths to start
Coming soonAI Evaluation and MonitoringMachine Learning: from Data to Validated Models and Decisions
8tasks in this job
75skills it uses
12tools named
The skills this job leans on most
- 1Analysing error patterns and failure modes in models and AI systemsNeeded in 7 of 8 tasks
- 2Designing and running evaluations of AI systemsNeeded in 5 of 8 tasks
- 3Judging whether and how to use AI for a taskNeeded in 5 of 8 tasks
- 4Evaluating fitness and full cost of AI-assisted workNeeded in 5 of 8 tasks
- 5Designing and evaluating retrieval for AI systemsNeeded in 2 of 8 tasks
ToolsJiraAmplitudeLangfuseClaudeZendeskCredo AI+ 6 more
All 8 tasks, where people get stuck and all 75 skills
Tasks
- Prioritise product outcomes
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Evaluate an AI use case for suitability, value and risk before adoption
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Design and bound a tool-using agentic AI workflow
- Develop and evaluate an LLM-based application against an agreed evaluation bar
- Maintain a data and AI risk register and record risk treatment and acceptance
- Conduct or review a data protection impact assessment for a data, analytics or AI processing
Where people get stuck
- Judging whether and how to use AI for a task
- Evaluating fitness and full cost of AI-assisted work
- Evaluating responsible AI use risks
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Evaluating fitness and full cost of AI-assisted work
- Judging whether and how to use AI for a task
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Bounding and securing AI systems' tool and data access
- Building cohort and retention analyses with correct denominators
- Building data protection by design into data products and pipelines
- Calculating and interpreting percentages, rates and ratios
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Communicating quantified uncertainty
- Comparing groups
- Controlling disclosure risk in shared data and outputs
- Decomposing one's own tasks for AI assistance
- Defining measurable outcomes and leading indicators
- Describing distributions with summaries and plots
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and interpreting hypothesis tests
- Designing and versioning data contracts for consumers
- Designing decision-oriented research with customers, users or employees
- Designing efficient and reproducible investigations
- Designing experiments
- Designing testable data controls
- Diagnosing incorrect query and calculation results
- Diagnosing operational handoffs and flow
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating product outcomes and metric integrity
- Evaluating responsible AI use risks
- Exercising human oversight of automated and AI outputs
- Explaining control types, dependencies and compensating controls
- Exploring data to find patterns that survive validation
- Gating releases on evidence with rollback ready
- Interpreting customer feedback and qualitative evidence
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Locating and acting on personal data for individual rights requests
- Making accountable decisions under uncertainty
- Mapping personal data processing and flows
- Mapping stakeholders' decisions, constraints and needs
- Monitoring model fitness and drift
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Prioritising work by evidenced value, cost, risk and confidence
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising cybersecurity threats to data work
- Recognising personal data and privacy duties
- Resolving consent and suppression status across systems
- Revising prior decisions when evidence changes
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Standardising and cleaning data under agreed rules
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Verifying AI-assisted outputs against sources
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
All tools
- Jira
- Amplitude
- Langfuse
- Claude
- Zendesk
- Credo AI
- OneTrust
- Azure OpenAI Service / Azure AI Foundry
- Amazon Bedrock
- Google Vertex AI
- Microsoft Excel
- Databricks
Paths to start
Tool Path
Excel for Analysis and Decision Models
Build and check Excel workbooks others can trust.
- Modules
- 5 open
- Runs in
- Your browser
Skill Path
Responsible AI Governance and Assurance
Govern and assure AI across its lifecycle including third-party AI and redress.
- Modules
- 5 open
- Practice data
- Raket Parts
Coming soonAI Evaluation and MonitoringApplied AI at Work
8tasks in this job
75skills it uses
11tools named
The skills this job leans on most
- 1Analysing error patterns and failure modes in models and AI systemsNeeded in 8 of 8 tasks
- 2Judging whether and how to use AI for a taskNeeded in 6 of 8 tasks
- 3Evaluating fitness and full cost of AI-assisted workNeeded in 6 of 8 tasks
- 4Designing and running evaluations of AI systemsNeeded in 5 of 8 tasks
- 5Designing and evaluating retrieval for AI systemsNeeded in 2 of 8 tasks
ToolsLangfuseAmplitudeZendeskCredo AIOneTrustClaude+ 5 more
All 8 tasks, where people get stuck and all 75 skills
Tasks
- Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
- Evaluate an AI use case for suitability, value and risk before adoption
- Prepare a model or AI system release decision with evaluation evidence, monitoring and rollback
- Design and bound a tool-using agentic AI workflow
- Develop and evaluate an LLM-based application against an agreed evaluation bar
- Maintain a data and AI risk register and record risk treatment and acceptance
- Conduct or review a data protection impact assessment for a data, analytics or AI processing
- Oversee an automated or AI system in operational use and correct, restrict or withdraw faulty outputs
Where people get stuck
- Judging whether and how to use AI for a task
- Evaluating fitness and full cost of AI-assisted work
- Evaluating responsible AI use risks
All skills
- Analysing error patterns and failure modes in models and AI systems
- Designing and running evaluations of AI systems
- Evaluating fitness and full cost of AI-assisted work
- Judging whether and how to use AI for a task
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing data source credibility and fitness for a question
- Assessing data-driven fairness and bias
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assessing privacy risk through data protection impact assessments
- Assessing supplier dependency, data-handling and exit risk
- Bounding and securing AI systems' tool and data access
- Building cohort and retention analyses with correct denominators
- Building data protection by design into data products and pipelines
- Calculating and interpreting percentages, rates and ratios
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Communicating quantified uncertainty
- Comparing groups
- Controlling disclosure risk in shared data and outputs
- Decomposing one's own tasks for AI assistance
- Defining measurable outcomes and leading indicators
- Describing distributions with summaries and plots
- Designing AI-assisted workflows with bounded tools and human authority
- Designing and interpreting hypothesis tests
- Designing and versioning data contracts for consumers
- Designing decision-oriented research with customers, users or employees
- Designing efficient and reproducible investigations
- Designing experiments
- Designing testable data controls
- Diagnosing incorrect query and calculation results
- Diagnosing operational handoffs and flow
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Estimating quantities and differences with intervals
- Evaluating ML generalisation and fitness
- Evaluating product outcomes and metric integrity
- Evaluating responsible AI use risks
- Exercising human oversight of automated and AI outputs
- Explaining control types, dependencies and compensating controls
- Exploring data to find patterns that survive validation
- Gating releases on evidence with rollback ready
- Interpreting customer feedback and qualitative evidence
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Locating and acting on personal data for individual rights requests
- Making accountable decisions under uncertainty
- Mapping personal data processing and flows
- Mapping stakeholders' decisions, constraints and needs
- Monitoring model fitness and drift
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Prioritising work by evidenced value, cost, risk and confidence
- Profiling data
- Protecting sensitive data in AI-assisted work
- Rating risks against appetite and choosing proportionate treatment
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising cybersecurity threats to data work
- Recognising personal data and privacy duties
- Resolving consent and suppression status across systems
- Revising prior decisions when evidence changes
- Selecting charts for evidence and decisions
- Setting and applying a release and rollback bar for models and AI systems
- Setting and reviewing action thresholds for scores and model predictions
- Standardising and cleaning data under agreed rules
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Verifying AI-assisted outputs against sources
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
All tools
- Langfuse
- Amplitude
- Zendesk
- Credo AI
- OneTrust
- Claude
- Azure OpenAI Service / Azure AI Foundry
- Amazon Bedrock
- Google Vertex AI
- Jira
- Microsoft Excel
Paths to start
Tool Path
Excel for Analysis and Decision Models
Build and check Excel workbooks others can trust.
- Modules
- 5 open
- Runs in
- Your browser
Skill Path
Responsible AI Governance and Assurance
Govern and assure AI across its lifecycle including third-party AI and redress.
- Modules
- 5 open
- Practice data
- Raket Parts
Coming soonAI Evaluation and MonitoringApplied AI at Work
About the role
Accountable for deciding where AI should and should not be used in a product. Bounds how AI features behave and judges their value against their risk. Works with AI engineers who build what the AI Product Manager specifies and with risk owners on what is acceptable.
Also called AI Product Owner