Databricks Data Engineer

A forward-thinking technology company operating across the Middle East, Asia Pacific, Africa, and the United States, dedicated to advancing cloud innovation and developing AI solutions that positively impact society. In this role as a Databricks Developer, you will collaborate with a team delivering sophisticated AI, data, and cloud capabilities that support the banking and financial sectors. You will utilize Databricks to design scalable data pipelines, enhance analytical performance, and generate actionable insights that accelerate digital transformation.

Responsibilities:

• ETL/ELT Development: Develop, test, and deploy robust and efficient data pipelines using PySpark/Scala and the Databricks platform (including Delta Lake and Databricks Workflows).

• Data Transformation: Implement complex data transformation logic to clean, enrich, and aggregate financial data from various source systems (e.g., core banking, trading platforms).

• Cloud Integration: Integrate Databricks with native cloud services (AWS, Azure, or GCP) for data ingestion (e.g., S3, ADLS) and workflow orchestration (e.g., Azure Data Factory, AWS Glue).

• Quality and Testing: Write unit and integration tests for data pipelines and apply data quality checks to ensure accuracy in financial reporting and analysis.

• Compliance Support: Apply basic security and access control policies, such as those governed by Unity Catalog, to adhere to the firm's compliance requirements.

• Performance: Assist in monitoring and tuning Databricks cluster configurations and Spark job parameters to improve efficiency and reduce cost.

Qualifications and Experience:

• Databricks Platform: Strong hands-on experience with Databricks notebooks, clusters, job scheduling, and understanding of the Delta Lake transaction log for reliability.
• Programming: Proficient in Python (especially PySpark) and expert in SQL.
• Big Data Concepts: Solid understanding of Apache Spark fundamentals (e.g., RDDs, DataFrames, lazy evaluation) and distributed computing.
• Data Modeling: Practical experience with dimensional modeling (star/snowflake schemas) and implementing the Medallion Architecture (Bronze, Silver, Gold layers) on Delta Lake.
• DevOps/DataOps: Familiarity with version control (Git) and experience with basic CI/CD processes for deploying Databricks code.

The Reference Number for this position is NG60854 which is a Contract role in Sandton, Johannesburg offering a rate of up to R500 per hour salary negotiable based on experience. E-mail Nokuthula on nokuthulag@ e-Merge.co.za or call her for a chat on 011 463 3633 to discuss this and other opportunities.

Are you ready for a change of scenery? e-Merge IT recruitment is a niche recruitment agency. We offer our candidates options so that we can successfully place the right people with the right companies, in the right roles. Check out the e-Merge IT website www.e-merge.co.za for more great positions.

Posted on 04 Dec 09:53, Closing date 2 Feb