Gopal G.

Data Engineer

Gopal is a Data Engineer with over eight years of experience in regulated sectors like automotive, technology, and energy. He excels in GCP, Azure, AWS, and Snowflake, with expertise in full life cycle development, data modeling, database architecture, and performance optimization.

His proudest achievements include creating and optimizing ETL/ELT pipelines across multi-cloud environments. Gopal's Google Cloud, AWS, Microsoft Azure, and Snowflake certifications highlight his commitment to continuous learning and professional excellence.

He holds a Master's degree in Computer Engineering.

Main expertise

  • Fact Data Modeling 8 years
  • ETL
    ETL 8 years
  • Unix shell 7 years

Other skills

  • Pandas
    Pandas 4 years
  • MySQL
    MySQL 4 years
  • Apache ZooKeeper
    Apache ZooKeeper 4 years
Gopal

Gopal G.

United Kingdom

Get started

Selected experience

Employment

  • Data Engineer

    Nissan Motor Corporation - 1 year 10 months

    • Designed and implemented efficient and scalable data pipelines on Google Cloud Platform (GCP) to collect, process, and transform raw data into usable formats for analysis and consumption;
    • Led and managed offshore teams to successfully implement various data engineering tasks, ensuring alignment with project goals and maintaining high-quality standards through regular communication, clear documentation, and effective task delegation;
    • Oversaw governance and compliance of data stored in BigQuery, ensuring adherence to UK and EU GDPR regulations;
    • Conducted Data Privacy Impact Assessments (DPIA) for various projects at Nissan UK Limited and implemented necessary measures to mitigate or reduce risks;
    • Built and maintained data warehouses, data lakes, and data lake houses on GCP using services such as BigQuery, Google Cloud Storage (GCS), and Bigtable;
    • Integrated data from various sources into GCP using services like Cloud Storage, Cloud Pub/Sub, and Cloud SQL;
    • Implemented proper data governance and security measures using GCP Identity and Access Management (IAM) and Data Loss Prevention (DLP) to ensure compliance;
    • Built data pipelines using Google Dataflow to efficiently handle large volumes of data;
    • Implemented ETL/ELT processes to extract data from various sources and load it into data warehouses or data lakes;
    • Developed streaming pipelines for real-time data ingestion utilizing Kafka and Kafka Connect;
    • Implemented Python-based transformations and BigQuery procedures, orchestrating their execution seamlessly using Google Cloud Composer;
    • Engineered data transformations using Apache Beam, optimized for peak performance on Google DataProc clusters.

    Technologies:

    • Technologies:
    • Fact Data Modeling
    • ETL ETL
    • Unix shell
    • Performance Testing
    • Unit Testing
    • AWS S3 AWS S3
    • Data Analytics
    • Looker Looker
    • Snowflake Snowflake
    • BigQuery BigQuery
    • Pandas Pandas
    • MySQL MySQL
    • Data Modeling
    • Database testing
    • Apache ZooKeeper Apache ZooKeeper
    • AWS Athena AWS Athena
    • Redshift Redshift
    • Python Python
    • SQL SQL
    • Apache Kafka Apache Kafka
    • Apache Airflow Apache Airflow
    • Apache Spark Apache Spark
    • Hadoop Hadoop
    • Google Cloud Google Cloud
    • Data Engineering

  • Lead Data Engineer

    Technovert - 2 years 7 months

    • Developed ETL processes using Python and SQL to transform raw data into usable formats and load them into BigQuery for analysis;
    • Built and architected multiple data pipelines, managed end-to-end ETL and ELT processes for data ingestion and transformation in GCP, and coordinated tasks among the team;
    • Designed and implemented data pipelines using GCP services such as Dataflow, Dataproc, and Pub/Sub;
    • Migrated Oracle DSR to BigQuery using Dataproc, Python, Airflow, and Looker;
    • Designed and developed a Python ingestion framework to load data from various source systems, including AR modules, inventory modules, files, and web services, into BigQuery;
    • Developed pipelines to load data from customer-placed manual files in Google Drive to GCS and subsequently to BigQuery using BigQuery stored procedures;
    • Participated in code reviews and contributed to the development of best practices for data engineering on GCP;
    • Implemented data security and access controls using GCP's Identity and Access Management (IAM) and Cloud Security Command Center.

    Technologies:

    • Technologies:
    • Databricks Databricks
    • Fact Data Modeling
    • ETL ETL
    • Unix shell
    • Performance Testing
    • Unit Testing
    • AWS S3 AWS S3
    • Oracle Oracle
    • Salesforce Salesforce
    • Data Analytics
    • Microsoft Power BI Microsoft Power BI
    • Snowflake Snowflake
    • BigQuery BigQuery
    • Pandas Pandas
    • MySQL MySQL
    • Data Modeling
    • Database testing
    • Apache ZooKeeper Apache ZooKeeper
    • Azure Azure
    • Azure Data Factory Azure Data Factory
    • Azure Synapse Azure Synapse
    • Python Python
    • SQL SQL
    • Apache Kafka Apache Kafka
    • Apache Airflow Apache Airflow
    • Apache Spark Apache Spark
    • Hadoop Hadoop
    • Google Cloud Google Cloud
    • Data Engineering

  • Data Engineer

    Accenture - 1 year 8 months

    • Designed and implemented Snowflake data warehouses, developing schemas, tables, and views optimized for performance and data accessibility;
    • Extracted data from Oracle databases, transformed it into CSV files, and loaded these files into a Snowflake data warehouse stage hosted on AWS S3, ensuring secure and efficient data transfer and storage;
    • Created and utilized virtual warehouses in Snowflake based on business requirements, effectively tracked credit usage to enhance business insights and resource allocation;
    • Designed and configured Snowpipe pipelines for seamless and near-real-time data loading, reducing manual intervention and enhancing data freshness;
    • Parsed XML data and organized it into structured Snowflake tables for efficient data storage and seamless data analysis;
    • Designed and implemented JSON data ingestion pipelines, leveraging Snowflake's capabilities to handle nested and complex JSON structures;
    • Designed and deployed Amazon Redshift clusters, optimizing schema design, distribution keys, and sort keys for optimal query performance;
    • Leveraged AWS Lambda functions and Step Functions to orchestrate ETL workflows, ensuring data accuracy and timely processing;
    • Created and maintained data visualizations and reports using Amazon QuickSight to facilitate data analysis and insights.

    Technologies:

    • Technologies:
    • Fact Data Modeling
    • ETL ETL
    • Unix shell
    • Performance Testing
    • Unit Testing
    • Oracle Oracle
    • Data Analytics
    • Tableau Tableau
    • Data Modeling
    • Database testing
    • Python Python
    • SQL SQL
    • Data Engineering

  • BI Consultant, General Electric

    Tech Mahindra - 2 years 7 months

    • Designed and implemented Teradata packages to facilitate seamless data extraction, transformation, and loading (ETL) operations from diverse sources into data warehouses;
    • Developed interactive and dynamic reports using SSRS, providing stakeholders with timely and insightful data visualizations for informed decision-making;
    • Conducted rigorous data validation and quality checks to ensure the integrity and accuracy of processed data;
    • Optimized ETL performance by employing advanced techniques, resulting in a 25% reduction in processing time;
    • Developed the ingestion strategy for loading data from multiple source systems to the operational layer in the data warehouse using Python, SQL, and stored procedures;
    • Understood and developed design documents as deliverables for the project;
    • Implemented SCD Type 1 and Type 2 functionality and developed custom scripts in Teradata for integration and functionality development for different modules like Primavera P6 and Oracle Project module;
    • Managed and troubleshot issues as a DWH analyst to ensure the smooth flow of business operations;
    • Prepared unit test cases and performed end-to-end integration testing;
    • Actively participated in design discussions and reviewed solutions;
    • Participated in peer review discussions on development before moving to higher environments;
    • Loaded data from multiple files to a single target table using ODI variables;
    • Configured and developed ETL mappings to load data from XML and complex (unstructured/semi-structured) files;
    • Utilized Power BI to design and develop insightful visualizations and interactive dashboards, enabling data-driven decision-making for stakeholders and enhancing overall data engineering solutions.

    Technologies:

    • Technologies:
    • Fact Data Modeling
    • ETL ETL
    • Unix shell
    • Performance Testing
    • Unit Testing
    • Oracle Oracle
    • Data Analytics
    • Tableau Tableau
    • Data Modeling
    • SQL SQL
    • Data Engineering

Education

  • MSc.Computer Software Engineering

    University of West London · 2022 - 2023

  • MSc.Electronics and Communications

    Jawaharlal university of Hyderabad · 2012 - 2016

Find your next developer within days, not months

In a short 25-minute call, we would like to:

  • Understand your development needs
  • Explain our process to match you with qualified, vetted developers from our network
  • You are presented the right candidates 2 days in average after we talk

Not sure where to start? Let’s have a chat

First developer starts within days. No aggressive sales pitch.