Overview
Intermediate Data Engineer Jobs in Cape Town, South Africa at Ignition Group
The Intermediae Data Engineer is accountable for creating and maintaining the technological infrastructure of Spot Money and implementing methods to improve data reliability and quality. Ensure the consistent delivery of data architecture throughout ongoing projects by creating data models, building data pipelines and coordinating ETL (extract, transform, load).
Responsibilities :
Data Ingestion & Processing
Assist in building ETL/ELT pipelines using AWS services like AWS Glue, Lambda, and Step Functions.
Work with Amazon S3 for data storage and ingestion.
Support real-time and batch data processing jobs.
Data Storage & Management
Load and manage data in AWS
Work with Parquet, Avro, JSON, and CSV file formats in data lakes.
Data Transformation &
Integration
Support data transformations using PySpark, SQL, or AWS Glue ETL.
Assist in writing SQL queries for data aggregation, cleansing, and structuring.
Automation & Orchestration
Monitor scheduled workflows
Write basic Python or Shell scripts for automating tasks.
Monitoring & Debugging
Assist in monitoring and troubleshooting failed data pipelines.
Perform unit testing and basic debugging of data workflows.
Title: Intermediate Data Engineer
Company: Ignition Group
Location: Cape Town, South Africa
Category: IT/Tech (Data Engineer, Cloud Computing, Big Data, Data Analyst)
