Explore Tasks
Monitor a deployed model or AI system in use and recommend retraining, recalibration, change or retirement
The model or AI system owner receives a monitoring finding with drift or quality evidence, business impact and a recommended action (retrain, recalibrate, change prompt or retrieval, roll back or retire), and records the decision.
2jobs do it
4jobs help with it
51skills it needs
- Machine Learning Engineer
- AI Engineer
- Data Scientist
- Lead Data Scientist
- Lead AI & ML Engineer
- AI Product Manager
Does itHelps with it
- Reading and tracing unfamiliar code
- Reviewing human- and AI-generated code changes for correctness and risk
- Sampling and reviewing a team's work against standards
- Diagnosing misleading data visualisations
- Verifying reports and dashboards before and after release
- Verifying AI-assisted outputs against sources
- Disclosing AI contribution to professional work
- Testing competing explanations with discriminating evidence
- Independently checking analytical conclusions and claims
- Escalating unresolved risks with evidence and clear ownership
- Giving specific developmental feedback on data and AI work
- Judging the materiality of defects and deficiencies for the intended use
- Writing maintainable code for analytical and data work
- Fitting, checking and interpreting inferential regression models
- Estimating quantities and differences with intervals
- Establishing and declaring data grain
- Combining tables safely: cardinality, fan-out and conservation of totals
- Calculating and interpreting percentages, rates and ratios
- Interpreting averages, weighted measures and mix effects
- Judging whether a change exceeds normal variation
- Formulating answerable data questions
- Assessing data source credibility and fitness for a question
- Profiling data
- Standardising and cleaning data under agreed rules
- Preparing analysis-ready datasets reproducibly
- Choosing and justifying missing-data treatments
- Describing distributions with summaries and plots
- Comparing groups
- Exploring data to find patterns that survive validation
- Investigating anomalies
- Designing and interpreting hypothesis tests
- Reading charts and dashboards correctly, including their state
- Selecting charts for evidence and decisions
- Writing evidence-led findings and auditable recommendations with their limits
- Defining the analysis population: filters, nulls, dates and time zones
- Analysing event sequences, states and durations
- Diagnosing incorrect query and calculation results
- Decomposing one's own tasks for AI assistance
- Training reproducible machine learning models
- Eliciting the decision behind an analytical request
- Designing efficient and reproducible investigations
- Choosing an analytical method to fit the question and data
- Gating releases on evidence with rollback ready
- Setting and reviewing action thresholds for scores and model predictions
- Bounding and securing AI systems' tool and data access
- Designing and running evaluations of AI systems
- Analysing error patterns and failure modes in models and AI systems
- Evaluating ML generalisation and fitness
- Monitoring model fitness and drift
- Assessing data-driven fairness and bias
- Setting and applying a release and rollback bar for models and AI systems
Paths that teach the skills it needs
Skill Path
Data Literacy for Business Decisions
Judge whether a business claim from a table, KPI, chart or dashboard really holds up.
- Modules
- 5 open
- Practice data
- Raket Parts
Tool Path
SQL for Analysis
Answer real business questions with SQL, from your first query to finding why a number moved.
- Modules
- 5 open
- Runs in
- Your browser
Skill Path
Data Preparation: Cleaning, Repair and Reconciliation
Check whether a table is fit for the job, then clean, repair and reconcile it with the evidence to back it.
- Modules
- 5 open
- Practice data
- Raket Parts
Coming soonFraming Questions and Sourcing Evidence for Investigations