Lead Data Engineer
Accountable for the design and review of pipelines that span several teams and for the reliability conventions engineers follow.
4tasks in this job
47skills it uses
9tools named
The skills this job leans on most
- 1Designing automated pipelines and processes that fail safely and recoverNeeded in 2 of 4 tasks
- 2Designing and validating batch and streaming ingestionNeeded in 2 of 4 tasks
- 3Engineering production data transformationsNeeded in 2 of 4 tasks
ToolsMicrosoft Azure (portal, Cost Management, Monitor)Amazon Web Services (console, Cost Explorer, CloudWatch)GitHub ActionsGit / GitHubTerraformPythonApache AirflowDatabricksApache Kafka
All 4 tasks, where people get stuck and all 47 skills
Tasks
- Diagnose a data pipeline failure and recover safely
- Prepare and verify a data-service release or migration for change approval
- Architect a bounded data and analytics service
- Build a tested batch or streaming data pipeline
Where people get stuck
- Reviewing human- and AI-generated code changes for correctness and risk
- Tracing data lineage upstream and downstream
- Designing and versioning data contracts for consumers
All skills
- Designing and validating batch and streaming ingestion
- Designing automated pipelines and processes that fail safely and recover
- Engineering production data transformations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data-service continuity and recoverability
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Coordinating accountable handoffs between teams
- Designing semantic models for governed analytics
- Diagnosing misleading data visualisations
- Disclosing AI contribution to professional work
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Evaluating service performance and quality trade-offs
- Evaluating software architecture for data and AI systems
- Explaining technical work to accountable non-specialists
- Facilitating evidence-led retrospectives and corrective actions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Judging the materiality of defects and deficiencies for the intended use
- Mapping proposed process changes and control impacts
- Monitoring data quality and freshness in production
- Preparing change records with impact assessment and rollback
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Recognising and containing suspected personal data breaches
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Recovering data pipelines safely
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Setting and reviewing reliability objectives and error budgets for data and AI services
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
Paths to start
Coming soonAnalytics Engineering: Tested, Version-controlled TransformationDesigning Controlled Automations and AI-assisted Workflows
7tasks in this job
102skills it uses
13tools named
The skills this job leans on most
- 1Reviewing human- and AI-generated code changes for correctness and riskNeeded in 6 of 7 tasks
- 2Testing changes to data services and business systems for business meaning and downstream impactNeeded in 6 of 7 tasks
- 3Tracing data lineage upstream and downstreamNeeded in 5 of 7 tasks
- 4Designing automated pipelines and processes that fail safely and recoverNeeded in 4 of 7 tasks
- 5Designing and versioning data contracts for consumersNeeded in 1 of 7 tasks
ToolsSQL (query language)Apache AirflowGitHub ActionsGit / GitHubJiraMicrosoft Azure (portal, Cost Management, Monitor)+ 7 more
All 7 tasks, where people get stuck and all 102 skills
Tasks
- Diagnose a data pipeline failure and recover safely
- Prepare and verify a data-service release or migration for change approval
- Lead an incident through containment, restoration and communication
- Define, monitor and review data-service reliability objectives
- Build a tested batch or streaming data pipeline
- Agree a data contract with source and consumer owners and assess the impact of upstream changes
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Diagnosing and remedying query and database performance
- Explaining data and AI platform architecture patterns and building blocks
- Analysing costs, allocation and margin drivers
All skills
- Designing and versioning data contracts for consumers
- Designing automated pipelines and processes that fail safely and recover
- Reviewing human- and AI-generated code changes for correctness and risk
- Testing changes to data services and business systems for business meaning and downstream impact
- Tracing data lineage upstream and downstream
- Analysing error patterns and failure modes in models and AI systems
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assigning and operating data ownership and stewardship
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and verifying KPIs against their specification
- Classifying AI uses against regulatory obligations
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Constructing clear and honest charts
- Controlling disclosure risk in shared data and outputs
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Designing and validating batch and streaming ingestion
- Designing and validating digital analytics event measurement
- Designing decision rights and dispute routes for data and AI work
- Designing dimensional models with explicit grain
- Designing efficient and reproducible investigations
- Designing enforceable data governance policies
- Designing implementable change plans
- Designing relational schemas: keys, constraints and normalisation
- Designing semantic models for governed analytics
- Diagnosing and remedying query and database performance
- Diagnosing data quality exceptions
- Diagnosing duplicate records
- Diagnosing misleading data visualisations
- Diagnosing operational handoffs and flow
- Disclosing AI contribution to professional work
- Enforcing and auditing data access
- Engineering production data transformations
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating and improving chart accessibility
- Evaluating service performance and quality trade-offs
- Evaluating software architecture for data and AI systems
- Explaining technical work to accountable non-specialists
- Facilitating evidence-led retrospectives and corrective actions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Governing shared data definitions
- Identifying and prioritising critical data elements
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Influencing change without formal authority
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Investigating procurement spend leakage and duplicate commitments
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Judging whether uncertainty changes the decision
- Managing master data identities and exceptions
- Mapping proposed process changes and control impacts
- Mapping stakeholders' decisions, constraints and needs
- Monitoring data quality and freshness in production
- Planning and validating controlled data migrations
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading business metrics
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Reconciling conflicting reports
- Recovering data pipelines safely
- Repairing data defects reversibly under owner authority
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and reviewing reliability objectives and error budgets for data and AI services
- Specifying BI measure semantics: additivity, aggregation and time
- Specifying decision-useful reports
- Standardising and cleaning data under agreed rules
- Testing competing explanations with discriminating evidence
- Understanding and interrogating data and application APIs
- Validating records by matching, recomputation and end-to-end tracing
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Working through stakeholder conflict over evidence and responsibility
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing testable work standards and review checklists
- Writing verifiable requirements and acceptance criteria
All tools
- SQL (query language)
- Apache Airflow
- GitHub Actions
- Git / GitHub
- Jira
- Microsoft Azure (portal, Cost Management, Monitor)
- Amazon Web Services (console, Cost Explorer, CloudWatch)
- Apache Spark / PySpark
- Databricks
- Python
- Apache Kafka
- ServiceNow
- Terraform
Paths to start
Skill Path
Data Literacy for Business Decisions
Judge whether a business claim from a table, KPI, chart or dashboard really holds up.
- Modules
- 5 open
- Practice data
- Raket Parts
Skill Path
Data Preparation: Cleaning, Repair and Reconciliation
Check whether a table is fit for the job, then clean, repair and reconcile it with the evidence to back it.
- Modules
- 5 open
- Practice data
- Raket Parts
Tool Path
SQL for Analysis
Answer real business questions with SQL, from your first query to finding why a number moved.
- Modules
- 5 open
- Runs in
- Your browser
Coming soonAnalytics Engineering: Tested, Version-controlled Transformation
7tasks in this job
124skills it uses
16tools named
The skills this job leans on most
- 1Reviewing human- and AI-generated code changes for correctness and riskNeeded in 6 of 7 tasks
- 2Designing automated pipelines and processes that fail safely and recoverNeeded in 4 of 7 tasks
- 3Diagnosing and remedying query and database performanceNeeded in 2 of 7 tasks
- 4Designing and validating batch and streaming ingestionNeeded in 2 of 7 tasks
- 5Engineering production data transformationsNeeded in 2 of 7 tasks
ToolsMicrosoft Azure (portal, Cost Management, Monitor)Amazon Web Services (console, Cost Explorer, CloudWatch)SnowflakeDatabricksMicrosoft Fabric (incl. OneLake)Apache Spark / PySpark+ 10 more
All 7 tasks, where people get stuck and all 124 skills
Tasks
- Evaluate platform performance and cost
- Lead an incident through containment, restoration and communication
- Define, monitor and review data-service reliability objectives
- Build a tested batch or streaming data pipeline
- Evaluate and recommend a data, analytics or AI product or vendor
- Agree a data contract with source and consumer owners and assess the impact of upstream changes
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
- Reviewing human- and AI-generated code changes for correctness and risk
- Escalating unresolved risks with evidence and clear ownership
All skills
- Designing and validating batch and streaming ingestion
- Designing automated pipelines and processes that fail safely and recover
- Diagnosing and remedying query and database performance
- Engineering production data transformations
- Reviewing human- and AI-generated code changes for correctness and risk
- Analysing error patterns and failure modes in models and AI systems
- Analysing event sequences, states and durations
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assessing supplier dependency, data-handling and exit risk
- Assigning and operating data ownership and stewardship
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and interpreting percentages, rates and ratios
- Calculating and verifying KPIs against their specification
- Choosing an analytical method to fit the question and data
- Choosing and justifying missing-data treatments
- Classifying AI uses against regulatory obligations
- Combining tables safely: cardinality, fan-out and conservation of totals
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Comparing groups
- Comparing options against explicit decision criteria
- Constructing clear and honest charts
- Controlling disclosure risk in shared data and outputs
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Decomposing one's own tasks for AI assistance
- Defining the analysis population: filters, nulls, dates and time zones
- Describing distributions with summaries and plots
- Designing and interpreting hypothesis tests
- Designing and validating digital analytics event measurement
- Designing and versioning data contracts for consumers
- Designing decision rights and dispute routes for data and AI work
- Designing dimensional models with explicit grain
- Designing efficient and reproducible investigations
- Designing enforceable data governance policies
- Designing implementable change plans
- Designing relational schemas: keys, constraints and normalisation
- Designing semantic models for governed analytics
- Diagnosing data quality exceptions
- Diagnosing duplicate records
- Diagnosing incorrect query and calculation results
- Diagnosing misleading data visualisations
- Diagnosing operational handoffs and flow
- Disclosing AI contribution to professional work
- Eliciting the decision behind an analytical request
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating and improving chart accessibility
- Evaluating service performance and quality trade-offs
- Evaluating software architecture for data and AI systems
- Explaining data and AI platform architecture patterns and building blocks
- Explaining technical work to accountable non-specialists
- Exploring data to find patterns that survive validation
- Facilitating evidence-led retrospectives and corrective actions
- Fitting, checking and interpreting inferential regression models
- Formulating answerable data questions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Governing shared data definitions
- Identifying and prioritising critical data elements
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Influencing change without formal authority
- Interpreting averages, weighted measures and mix effects
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Investigating procurement spend leakage and duplicate commitments
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether and how to use AI for a task
- Judging whether uncertainty changes the decision
- Making accountable decisions under uncertainty
- Managing master data identities and exceptions
- Mapping proposed process changes and control impacts
- Mapping stakeholders' decisions, constraints and needs
- Monitoring data quality and freshness in production
- Planning and validating controlled data migrations
- Preparing analysis-ready datasets reproducibly
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading business metrics
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Reconciling conflicting reports
- Recovering data pipelines safely
- Repairing data defects reversibly under owner authority
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and reviewing reliability objectives and error budgets for data and AI services
- Specifying BI measure semantics: additivity, aggregation and time
- Specifying decision-useful reports
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Training reproducible machine learning models
- Understanding and interrogating data and application APIs
- Validating records by matching, recomputation and end-to-end tracing
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Working through stakeholder conflict over evidence and responsibility
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing testable work standards and review checklists
- Writing verifiable requirements and acceptance criteria
All tools
- Microsoft Azure (portal, Cost Management, Monitor)
- Amazon Web Services (console, Cost Explorer, CloudWatch)
- Snowflake
- Databricks
- Microsoft Fabric (incl. OneLake)
- Apache Spark / PySpark
- ServiceNow
- Jira
- Apache Airflow
- Azure Data Factory
- Python
- Apache Kafka
- Informatica (IDMC, PowerCenter, Data Quality)
- Terraform
- Git / GitHub
- GitHub Actions
Paths to start
Skill Path
Data Literacy for Business Decisions
Judge whether a business claim from a table, KPI, chart or dashboard really holds up.
- Modules
- 5 open
- Practice data
- Raket Parts
Skill Path
Data Preparation: Cleaning, Repair and Reconciliation
Check whether a table is fit for the job, then clean, repair and reconcile it with the evidence to back it.
- Modules
- 5 open
- Practice data
- Raket Parts
Skill Path
Exploratory Data Analysis and Pattern Discovery
Describe a distribution honestly and decide which unusual records need checking before anyone acts.
- Modules
- 5 open
- Practice data
- Raket Parts
About the role
Accountable for the design and review of pipelines that span several teams and for the reliability conventions engineers follow. Decides pipeline designs and data contracts across team boundaries and leads the response to complex failures. Leads this work without line management, where a Data Engineer owns the pipelines they build.