- 1. Data Engineer - Brillio Technologies Bangalore
JUN 2019 - Present
- Project - Lineage Logistics [DATALAKE]
- Worked on automating various linux tasks through shell scripts.
- Python scripts for bulk table creation in impala and hive ,with validation.
- Python and sql scripts for data backfilling.
- Python script to clean up unwanted external tables hdfs data
- Python script for compression of hdfs data of external tables to increase performance.
- Pyspark script for ddl generation for source tables based on parquet format schema in impala.
- CDH admin works , package installation , job scheduling , prod deployement.
- AWS security checks , EC2 node spin up based on requirements , S3 backup BDR jobs.
- Airflow DAGs for sqoop incremental ingestion.
- Working on implementing realtime message bus data consumption using nifi
Appreciation
- Team Excellence award - Brillio
- 2. Full stack intern - YULU Bangalore
JAN 2019 - JUN 2019
-
- Involved in Yulu web app development with react js which is my final year project too
- With this Yulu web app , UBER successfully partnered with Yulu.
- Live Web app link