End to end batch processing,data orchestration and real time streaming analytics on GCP
Google Cloud platform is catching up and a lot of companies have already started moving their infrastructure to GCP. I have worked on multiple projects to give the most practical solutions to real world use cases in terms of data engineering on the Cloud. These projects are designed keeping in mind end to end lifecycle of a typical Big data ETL project both batch processing and real-time streaming and analytics.
Considering the most important components of any batch processing or streaming jobs , this project makes use of
- Writing ETL jobs using Pyspark from scratch
- Storage components on GCP (GCS & Dataproc HDFS)
- Loading Data into Data-warehousing tool on GCP (BigQuery)
- Handling/Writing Data Orchestration and dependencies using Apache Airflow(Google Composer) in Python from scratch
- Batch Data ingestion using Sqoop , CloudSql and Apache Airflow
- Real Time data streaming and analytics using the latest API , Spark Structured Streaming with Python
- Micro batching using PySpark streaming & Hive on Dataproc
What I learnt
- Pyspark for ETL/Batch Processing on GCP using Bigquery as data warehousing component
- Automate and orchestrate SparkSql batch jobs using Apache Airflow and Google Workflows
- Sqoop for Data ingestion from CloudSql and using Airflow to automate the batch ETL
- Difference between Event-time data transformations and process-time transformations
- Pyspark Structured Streaming - Real Time Data streaming and transformations
- Save real time streaming raw data as external hive tables on Dataproc and perform ad-hoc queries using HiveSql
- Run Hive-SparkSql jobs on Dataproc and automate micro-batching and transformations using Airflow