Overview
Data Engineer Jobs in Sandton, South Africa at Discovery Limited
Discovery – Insure Senior Data Engineer – Vitality Drive International About Discovery
Discovery’s core purpose is to make people healthier and to enhance and protect their lives. We seek out and invest in exceptional individuals who understand and support our core purpose, and whose own values align with those of Discovery. Our fast‑paced and dynamic environment enables smart, self‑driven people to be their best. As global thought leaders, Discovery is passionate about innovating in order to not only achieve financial success, but to ignite positive and meaningful change within our society.
About
Discovery Insure
Discovery Insure is committed to creating a nation of great drivers through our innovative Shared‑value Insurance model. Discovery Insure is South Africa’s fastest growing short‑term insurance company with comprehensive products that provide protection against current and emerging risks facing clients in the motor and home insurance sectors. Vitality Drive, an internationally‑recognised and award‑winning programme, is a key differentiator in the market that incentivises and rewards clients for driving well.
The Vitality Drive programme has been scaled to local and international markets which now include Europe and the Middle East. The company employs over 1 000 people who are committed to putting our customers and financial advisers first by providing unique and innovative solutions and cover.
Key Purpose
The Senior Data Engineer is responsible for the design, development, and optimisation of scalable, enterprise‑grade data platforms on Azure, with Azure Synapse Analytics at the core. This role enables the delivery of high‑quality, trusted, and reusable data assets to support analytics, reporting, and AI use cases. The incumbent will play a key role in advancing a modern data lakehouse architecture, ensuring data is governed, performant, and aligned to business value outcomes.
Areas of responsibility may include but are not limited to:
Design
- Translate business requirements into technical designs adhering to Discovery Insure processes, standards and guidelines, considering performance, security, scalability, and cost.
- Define and evolve medallion architecture patterns in Microsoft Fabric (Bronze/Silver/Gold), including ingestion, transformation, and curated consumption layers.
- Produce and maintain modern data models:
- OLTP‑aligned models where needed
- Dimensional models (star/snowflake) for analytics
- Curated “Gold” layer data products (conformed dimensions, business‑ready facts)
- Design ingestion and transformation strategies across Fabric components (Lakehouse/Warehouse, Data Pipelines, Dataflows Gen2, Notebooks/Spark).
- Establish data quality, lineage, and governance requirements (e.g., data contracts, validation rules, auditing, retention).
- Liaise with relevant parties where clarification of business requirements or resolution to technical issues is needed.
- Research and recommend effective solutions to technical issues that arise (e.g., performance bottlenecks, modelling trade‑offs, orchestration patterns).
- Estimate development timelines based on business requirements and technical complexity; identify risks and dependencies early.
Development
- Construct robust, maintainable, scalable, optimally performing solution components in line with technical specifications, following prescribed process, standards and procedures.
- Develop and optimise:
- T‑SQL for Fabric Warehouse/Azure SQL (as applicable)
- ELT/ETL transformations (SQL/Spark) aligned to the medallion pattern
- Reusable frameworks for ingestion, validation, and incremental processing
- Build and maintain orchestration using Fabric Data Pipelines (scheduling, dependencies, retries, parameterisation).
- Implement patterns for incremental loads and change processing (e.g., CDC‑like approaches, watermarking, SCD Type 1/2 where appropriate).
- Apply performance engineering:
- Query optimisation and model simplification
- Efficient file formats and table design for analytical workloads
Testing
- Conduct unit testing and fix any defects found, aiming for high‑quality releases.
- Verify build stability and quality with development team before releasing to test team, aiming to release…
Title: Data Engineer
Company: Discovery Limited
Location: Sandton, South Africa
Category: