Data Engineer
Accountable for engineering and operating reliable batch and streaming pipelines with validated ingestion and safe recovery.
2tasks in this job
14skills it uses
9tools named
The skills this job leans on most
- 1Designing automated pipelines and processes that fail safely and recoverNeeded in 2 of 2 tasks
- 2Monitoring data quality and freshness in productionNeeded in 2 of 2 tasks
- 3Designing and validating batch and streaming ingestionNeeded in 2 of 2 tasks
- 4Engineering production data transformationsNeeded in 2 of 2 tasks
- 5Recovering data pipelines safelyNeeded in 2 of 2 tasks
ToolsApache AirflowAzure Data FactoryDatabricksMicrosoft Azure (portal, Cost Management, Monitor)JiraPythonSQL (query language)Apache Spark / PySparkGit / GitHub
All 2 tasks, where people get stuck and all 14 skills
Tasks
Where people get stuck
All skills
- Designing and validating batch and streaming ingestion
- Designing automated pipelines and processes that fail safely and recover
- Monitoring data quality and freshness in production
- Recovering data pipelines safely
- Building decision-focused reports and dashboards
- Designing relational schemas: keys, constraints and normalisation
- Designing semantic models for governed analytics
- Evaluating service performance and quality trade-offs
- Gating releases on evidence with rollback ready
- Identifying keys and testing referential integrity
- Recognising statutory incident reporting duties and deadlines
- Standardising and cleaning data under agreed rules
- Tracing data lineage upstream and downstream
- Writing maintainable code for analytical and data work
Paths to start
Coming soonData Product Reliability: Release Gates, Service Levels and IncidentsAnalytics Engineering: Tested, Version-controlled Transformation
7tasks in this job
48skills it uses
15tools named
The skills this job leans on most
- 1Recovering data pipelines safelyNeeded in 7 of 7 tasks
- 2Monitoring data quality and freshness in productionNeeded in 6 of 7 tasks
- 3Designing automated pipelines and processes that fail safely and recoverNeeded in 5 of 7 tasks
- 4Designing and validating batch and streaming ingestionNeeded in 3 of 7 tasks
- 5Engineering production data transformationsNeeded in 3 of 7 tasks
ToolsApache AirflowMicrosoft Azure (portal, Cost Management, Monitor)Amazon Web Services (console, Cost Explorer, CloudWatch)JiraGit / GitHubGitHub Actions+ 9 more
All 7 tasks, where people get stuck and all 48 skills
Tasks
- Diagnose a data pipeline failure and recover safely
- Prepare and verify a data-service release or migration for change approval
- Lead an incident through containment, restoration and communication
- Define, monitor and review data-service reliability objectives
- Build a tested batch or streaming data pipeline
- Agree a data contract with source and consumer owners and assess the impact of upstream changes
- Build and secure the infrastructure for a data or AI service as reviewed code
Where people get stuck
- Tracing data lineage upstream and downstream
- Designing and versioning data contracts for consumers
- Testing changes to data services and business systems for business meaning and downstream impact
All skills
- Designing and validating batch and streaming ingestion
- Designing automated pipelines and processes that fail safely and recover
- Engineering production data transformations
- Monitoring data quality and freshness in production
- Recovering data pipelines safely
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data-service continuity and recoverability
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Designing dimensional models with explicit grain
- Designing relational schemas: keys, constraints and normalisation
- Designing semantic models for governed analytics
- Enforcing and auditing data access
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Evaluating service performance and quality trade-offs
- Evaluating software architecture for data and AI systems
- Explaining technical work to accountable non-specialists
- Facilitating evidence-led retrospectives and corrective actions
- Gating releases on evidence with rollback ready
- Identifying keys and testing referential integrity
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Mapping proposed process changes and control impacts
- Planning and validating controlled data migrations
- Preparing change records with impact assessment and rollback
- Preserving incident evidence and timeline
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading business metrics
- Recognising and containing suspected personal data breaches
- Recognising cybersecurity threats to data work
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Setting and reviewing reliability objectives and error budgets for data and AI services
- Standardising and cleaning data under agreed rules
- Testing changes to data services and business systems for business meaning and downstream impact
- Testing competing explanations with discriminating evidence
- Tracing data lineage upstream and downstream
- Understanding and interrogating data and application APIs
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- Apache Airflow
- Microsoft Azure (portal, Cost Management, Monitor)
- Amazon Web Services (console, Cost Explorer, CloudWatch)
- Jira
- Git / GitHub
- GitHub Actions
- GitLab
- Databricks
- Python
- Apache Spark / PySpark
- Azure Data Factory
- Apache Kafka
- SAP S/4HANA
- Salesforce
- Terraform
Paths to start
Coming soonData Product Reliability: Release Gates, Service Levels and IncidentsAnalytics Engineering: Tested, Version-controlled TransformationDesigning Controlled Automations and AI-assisted Workflows
8tasks in this job
96skills it uses
15tools named
The skills this job leans on most
- 1Monitoring data quality and freshness in productionNeeded in 6 of 8 tasks
- 2Testing changes to data services and business systems for business meaning and downstream impactNeeded in 6 of 8 tasks
- 3Designing automated pipelines and processes that fail safely and recoverNeeded in 5 of 8 tasks
- 4Tracing data lineage upstream and downstreamNeeded in 5 of 8 tasks
- 5Designing and versioning data contracts for consumersNeeded in 1 of 8 tasks
ToolsSQL (query language)Microsoft Azure (portal, Cost Management, Monitor)Apache AirflowApache Spark / PySparkAmazon Web Services (console, Cost Explorer, CloudWatch)GitHub Actions+ 9 more
All 8 tasks, where people get stuck and all 96 skills
Tasks
- Diagnose a data pipeline failure and recover safely
- Evaluate platform performance and cost
- Prepare and verify a data-service release or migration for change approval
- Lead an incident through containment, restoration and communication
- Define, monitor and review data-service reliability objectives
- Build a tested batch or streaming data pipeline
- Agree a data contract with source and consumer owners and assess the impact of upstream changes
- Set engineering standards for testing, change, data handling and AI-assisted code
Where people get stuck
All skills
- Designing and versioning data contracts for consumers
- Designing automated pipelines and processes that fail safely and recover
- Monitoring data quality and freshness in production
- Testing changes to data services and business systems for business meaning and downstream impact
- Tracing data lineage upstream and downstream
- Analysing error patterns and failure modes in models and AI systems
- Applying data classification and handling rules
- Assessing change risk, collisions and go or no-go
- Assessing data source credibility and fitness for a question
- Assessing data-service continuity and recoverability
- Assessing lawful basis, purpose compatibility and minimisation for data reuse
- Assigning and operating data ownership and stewardship
- Building data protection by design into data products and pipelines
- Building decision-focused reports and dashboards
- Calculating and verifying KPIs against their specification
- Classifying AI uses against regulatory obligations
- Communicating quantified uncertainty
- Communicating sensitive or unwelcome analytical findings
- Constructing clear and honest charts
- Coordinating accountable handoffs between teams
- Debugging data and application services from evidence
- Designing and validating batch and streaming ingestion
- Designing and validating digital analytics event measurement
- Designing dimensional models with explicit grain
- Designing efficient and reproducible investigations
- Designing enforceable data governance policies
- Designing relational schemas: keys, constraints and normalisation
- Designing semantic models for governed analytics
- Diagnosing and remedying query and database performance
- Diagnosing data quality exceptions
- Diagnosing duplicate records
- Diagnosing misleading data visualisations
- Diagnosing operational handoffs and flow
- Disclosing AI contribution to professional work
- Enforcing and auditing data access
- Engineering production data transformations
- Escalating unresolved risks with evidence and clear ownership
- Establishing and declaring data grain
- Estimating quantities and differences with intervals
- Evaluating and improving chart accessibility
- Evaluating service performance and quality trade-offs
- Evaluating software architecture for data and AI systems
- Explaining technical work to accountable non-specialists
- Facilitating evidence-led retrospectives and corrective actions
- Gating releases on evidence with rollback ready
- Giving specific developmental feedback on data and AI work
- Governing shared data definitions
- Identifying keys and testing referential integrity
- Independently checking analytical conclusions and claims
- Influencing change without formal authority
- Interpreting correlation without inferring cause
- Interpreting policy scope and delegated authority limits
- Investigating anomalies
- Investigating procurement spend leakage and duplicate commitments
- Judging the materiality of defects and deficiencies for the intended use
- Judging whether a change exceeds normal variation
- Judging whether uncertainty changes the decision
- Managing master data identities and exceptions
- Mapping proposed process changes and control impacts
- Mapping stakeholders' decisions, constraints and needs
- Planning and validating controlled data migrations
- Preparing change records with impact assessment and rollback
- Presenting and defending evidence-based recommendations to decision-makers
- Preserving incident evidence and timeline
- Profiling data
- Rating risks against appetite and choosing proportionate treatment
- Reading and tracing unfamiliar code
- Reading business metrics
- Reading charts and dashboards correctly, including their state
- Recognising and containing suspected personal data breaches
- Recognising and resolving ethical conflicts in data and AI work
- Recognising cybersecurity threats to data work
- Recognising misleading numerical claims
- Recognising personal data and privacy duties
- Recognising statutory incident reporting duties and deadlines
- Reconciling conflicting reports
- Recovering data pipelines safely
- Repairing data defects reversibly under owner authority
- Reviewing human- and AI-generated code changes for correctness and risk
- Running incident command: severity, timeline and status updates
- Sampling and reviewing a team's work against standards
- Selecting charts for evidence and decisions
- Setting and reviewing reliability objectives and error budgets for data and AI services
- Specifying BI measure semantics: additivity, aggregation and time
- Specifying decision-useful reports
- Standardising and cleaning data under agreed rules
- Testing competing explanations with discriminating evidence
- Understanding and interrogating data and application APIs
- Validating records by matching, recomputation and end-to-end tracing
- Verifying AI-assisted outputs against sources
- Verifying reports and dashboards before and after release
- Working through stakeholder conflict over evidence and responsibility
- Writing data quality rules
- Writing evidence-led findings and auditable recommendations with their limits
- Writing maintainable code for analytical and data work
- Writing verifiable requirements and acceptance criteria
All tools
- SQL (query language)
- Microsoft Azure (portal, Cost Management, Monitor)
- Apache Airflow
- Apache Spark / PySpark
- Amazon Web Services (console, Cost Explorer, CloudWatch)
- GitHub Actions
- Git / GitHub
- Terraform
- Jira
- Python
- Databricks
- Apache Kafka
- SAP S/4HANA
- Salesforce
- ServiceNow
Paths to start
Skill Path
Data Preparation: Cleaning, Repair and Reconciliation
Check whether a table is fit for the job, then clean, repair and reconcile it with the evidence to back it.
- Modules
- 5 open
- Practice data
- Raket Parts
Tool Path
SQL for Analysis
Answer real business questions with SQL, from your first query to finding why a number moved.
- Modules
- 5 open
- Runs in
- Your browser
Coming soonAnalytics Engineering: Tested, Version-controlled TransformationData Product Reliability: Release Gates, Service Levels and Incidents
About the role
Accountable for engineering and operating reliable batch and streaming pipelines with validated ingestion and safe recovery. Decides how data is extracted, landed and validated and how failures are detected and recovered. Differs from the Analytics Engineer, who models landed data for analysis.
Also called ETL Developer