Data Engineer

-Design and deliver scalable ELT pipelines on Databricks using PySpark, Spark SQL, and medallion architecture (Bronze/Silver/Gold). -Implement data governance practices including Unity Catalog for centralized metadata management and data lineage tracking. -Apply partitioning and caching strategies to optimize query performance and reduce processing times. -Manage CI/CD orchestration with Azure DevOps for automated testing and deployment of data assets. -Integrate Databricks jobs with Airflow DAGs for end-to-end workflow orchestration and dependency management.

-Built scalable ELT pipelines with Databricks using PySpark/Spark SQL for high-volume energy data processing. -Leveraged Delta Lake with time travel and CDC logic to ensure data consistency, auditability, and rollback capabilities. -Applied partitioning and caching strategies for performance optimization across large-scale transformations. -Managed CI/CD pipelines using GitLab for version-controlled deployments across environments. -Conducted code reviews and cross-team collaboration to maintain engineering standards and code quality.

• Responsible for performing ETL/ELT operations with Azure Databricks using PySpark and Spark SQL. This involved extracting data from various sources, transforming it into a usable format, and loading it into a target database or data warehouse;
• Designed, developed, tested, debugged, and implemented QlikSense solutions based on specified requirements from business users related to dashboard goals and objectives. This included creating interactive data visualizations and reports to help stakeholders make informed decisions. The tools used in this role included Azure Databricks, Apache Spark (PySpark, Spark SQL), Qlik Sense, and Qlik NPrinting.

• Responsible for analyzing, designing, and implementing ETLs/ELT workflows for various projects. This included the CRM project used for the management of prospects, customers, and marketing campaigns, as well as the BI project for selling passenger train tickets;
• Designed and implemented automated ETL/ELT jobs with scheduling to feed the centralized supplier and purchasing management database from different data sources. The tools used in this role included Talend (DI, Big Data, ESB) (Java), Azure Databricks, Hadoop Ecosystem, Apache Spark (Spark SQL, PySpark), Databricks, PostgreSQL, and Power BI.

• Performance tuning: Optimizing the ETL process to ensure it runs efficiently and effectively, including tuning database queries and optimising data flows;
• Responsible for analyzing detailed functional specifications and elaborating technical specifications for ETL projects. This included designing and executing ETL projects to generate reports and destination files according to the client's needs;
• Ensured rigour in the validation and delivery phases of these projects. The tool used in this role was Talend ETL, along with PostgreSQL.