Overview
Data Engineer/Scientist/Advanced Jobs in Menlyn, South Africa at OpenSource Intelligent Solutions
Position: Data Engineer/Scientist (Advanced) 1210
Location: Menlyn
Our client is looking for a Data Engineer/Scientist (Advanced) with Bachelor’s or Master’s degree in Computer Science, Software Engineering. minimum of 3 years’ experience as a Data Engineer, 2 years of experience working with AWS services must be based in Gauteng, contract December 2027
Essential skills requirements:
Proficiency in Python 3.x for data processing and automation.
Experience with AWS Glue for ETL processes.
Strong knowledge of AWS Athena for querying large datasets.
Hands-on experience with AWS Lambda for serverless computing.
Familiarity with AWS EC2 for scalable computing resources.
Expertise in AWS Cloud Watch for monitoring and logging.
Proficiency in working with Postgre
SQL RDS for database management.
Experience with AWS Quick Sight for data visualization and reporting.
Strong understanding of data ingestion pipelines, particularly for Call Detail Records (CDRs).
Proficiency in Git and Git Hub for version control and collaboration, including experience with Git Hub CI/CD pipelines.
Any additional responsibilities assigned in the Agile Working Model (AWM) Charter
Advantageous skills requirements:
Extensive experience with Terraform for infrastructure as code, including creating, managing, and optimizing AWS resources.
Experience with other AWS services such as S3, Kinesis, and Dynamo
DB.
Knowledge of data formats such as Parquet, AVRO, JSON, and CSV.
Experience with Docker for containerization.
Understanding of Big Data technologies and frameworks.
Familiarity with Agile working models and tools like JIRA and Confluence.
Experience with data quality tools such as Great Expectations.
Knowledge of REST API development and integration.
Strong analytical skills for troubleshooting and optimizing data pipelines.
Experience in developing technical documentation and artefacts.
What will be your role and responsibilities?
Design, develop, and maintain data ingestion pipelines for BMW’s connected fleet, focusing on Call Detail Records (CDRs).
Utilize AWS Glue for ETL processes to transform and load data into the analytics platform.
Implement efficient querying solutions using AWS Athena.
Develop serverless applications and workflows using AWS Lambda.
Monitor and optimize data pipelines using AWS Cloud Watch.
Manage and maintain Postgre
SQL RDS databases for data storage and retrieval.
Create interactive dashboards and reports using AWS Quick Sight.
Leverage Terraform extensively to define, deploy, and manage AWS infrastructure as code, ensuring scalability and reliability.
Use Git and Git Hub for version control, collaboration, and maintaining code repositories.
Implement and manage Git Hub CI/CD pipelines to automate testing, deployment, and delivery processes.
Ensure data security and compliance with BMW’s information classification requirements.
Stay updated with the latest data engineering tools, technologies, and industry trends.
Identify opportunities for process improvements and automation to enhance pipeline efficiency.
Collaborate with cross-functional teams to understand data requirements and deliver solutions.
Develop and maintain technical documentation for data engineering processes and solutions.
Apply now
!!! send CV to
Title: Data Engineer/Scientist/Advanced
Company: OpenSource Intelligent Solutions
Location: Menlyn, South Africa
Category: IT/Tech (Data Engineer, Cloud Computing, AWS)
