Project Portfolio
Project 11
Name – Visibility Report
Description – Creating real-time ERP process to get the supplier exchange and item related data from various sources (Oracle DB, Oracle UCM, FTP server) depending on the ERP and applying business rules based on the top of it. This data is exposed to Elastic search and unity catalog based on which the PowerBI dashboard is designed after generating meaningful insights for the business.
Technology Stack – AWS S3, Databricks, Github, S3, AWS Secret manager, Elastic search, Databricks Unity catalog
Responsibilities
1. Requirements gathering, and design technical document based on the functional requirements
2. Designed and implemented an efficient method of data collection from multiple sources like Oracle UCM, Oracle DB, FTP server and Transforming the business logic using databricks and storing back to S3 with delta format.
3. Exposed S3 data to unity catalog for analyzing the structured data.
4. Secrets manager to maintain the login credentials for the sources and use it run time for connections using boto3 library.
5. Exposing the dataset to Elastic search which would be consumed by the downstream process
6. Managed dependencies between multiple jobs to build end-to-end ETL workflows using Databricks workflow and scheduled the job to run every 6 hours.
7. Implemented SCD Type 2 logic to maintain the history of changes on a few tables.
8. Coordinated with the PowerBI developer to design the dashboard according to business requirements.
9. Coordinated with the team for Unit Testing & Data validation.
Project 10
Name – Item Maintenance
Description – Creating real-time process to get the items related data from Oracle UCM depending on the ERP and applying business rules on the top of it. This data is exposed to Elastic search index which is ultimately used by the downstream process to display the data on the Web portal.
Technology Stack – AWS S3, Databricks, Github, S3, AWS Secret manager, Elastic search, Databricks Unity catalog
Responsibilities
1. Requirements gathering, and design technical document based on the functional requirements
2. Designed and implemented an efficient method of data collection Oracle UCM and Transforming the business logic using databricks and storing back to S3 with delta format.
3. Exposed S3 data to unity catalog for analyzing the structured data.
4. Exposing the dataset to Elastic search which would be consumed by the downstream process
5. Managed dependencies between multiple jobs to build end-to-end ETL workflows using Databricks workflow and scheduled the job to run once a day.
6. Co-coordinating with team for Unit Testing & Data validation.
Project 9
Name – Dashx
Description – Migrate the on premise running movie scheduling website that performs some analytics on the historical data to determine the schedule of the movie in the theatre to maximize the profit. Using IaC(Cloudformation) the application is migrated to AWS. Setup the complete infrastructure using Cloudformation and implement the best practices from the security perspective and for making the application fault tolerant and highly available.
Technology Stack – AWS – Cloudformation, Batch, Autoscaling Group, VPC, Security Groups, Lambda, Cloudwatch, Loadbalancers, RDS, AWS SNS, AWS ECR, AWS Secret Manager, SonarQube, Github Actions, ZAP
Responsibilities
1. Implemented Infrastructure as Code (IaC) using cloudformation to provision security groups, EC2 instances, ASG, Load Balancer, VPC Endpoints, Lambda functions, Cloudwatch logs
2. Set up the application on Autoscaling group with an application load balancer to direct the traffic equally to all the instances.
3. Making the application fault tolerant by implementing load balancer health checks and custom health checks using lambda function.
3. Store the logs of the application to Cloudwatch logs for monitoring purposes.
4. Storing the metrics such as CPU utilization, memory utilization, network, usage, and number of hits for the application to Cloudwatch and create Alarms in case the metrics reach the maximum limit and notify the owner through SNS.
5. Using AWS Batch for automatically launching and terminating the EC2 instance for running the analytical process and using ECR for storing the image(with all the required packages).
6. Using SonarQube and ZAP tool for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs and code smells.
7. Automating the process of deployment using Github actions
8. Secrets manager to maintain the login credentials of the application and getting that in lambda function using boto3 package.
9. Using Github for SCM, Cloudformation as IaC and Github actions for build and deploy the code.
Project 8
Name – SDG Sunrise
Description – Creating real-time ERP process to get the data the from various sources(MySQL, SAP Hana, Oracle DB) depending on the ERP and applying business rules based on the top of it. This data is exposed to Kafka topics which is ultimately used by the downstream process to display the sales/order related data on the Web portal.
Technology Stack – AWS S3, Databricks, Github, S3, AWS Secret manager, Athena, Kafka
Responsibilities
1. Requirements gathering, and design technical document based on the functional requirements
2. Designed and implemented an efficient method of data collection from multiple sources ERP like Oracle DB, SAP Hana, MySQL and Transforming the business logic using databricks and storing back to S3 with delta format.
3. Exposed S3 data to Athena for analyzing the structured data.
4. Exposing the dataset to Kafka topics which would be consumed by the downstream process
5. Managed dependencies between multiple jobs to build end-to-end ETL workflows using Databricks workflow and scheduled the job to run after every 30 minutes.
6. Coordinating with team for Unit Testing & Data validation.
Project 7
Name – LeaClothing Workflow
Description – Scope of this project is to build a workflow to get the sales data from Google platform using Python code, setup RDS on EC2 instance and store the sales data in the RDS instance which will be further utilized by the downstream process to display on the shopping website and generate insights related to sales.
Technology Stack - Python, Google API, EC2, RDS, Airflow
Responsibilities
1. Requirements gathering, and design technical document based on the functional requirements
2. Building workflows on Airflow for getting full data extracted from the source using Google API.
3. Perform data quality checks and apply data cleansing techniques.
4. Agile environment. Jira is used for tracking user stories.
5. Setup RDS database functionality on EC2 instance and build a code to store data into the instance.
Project 6
Name – XO Content Tracker and Data Feeds
Description – Reading the data from Google Excel using the python package and performing cleansing and ETL operations on the data based on the business requirement and storing the data on S3 and Redshift which is further used in downstream processes like Tableau for generating visualizations.
Technology Stack - PySpark, SQL, AWS Athena, AWS S3, Sagemaker Notebooks, Github, Google API, AWS Redshift
Responsibilities
1. Requirements gathering, and designing technical documents based on the functional requirements
2. Building workflows on Airflow for getting full data extracted from the source using Google API.
3. Perform data quality checks and apply data cleansing techniques.
4. Carry out incremental and full loads operations with SCD Type 2 data loads.
5. Agile environment. Jira is used for tracking user stories.
6. Expose the data on Redshift/Athena/AWS S3 so that end users can easily query the data.
7. Create business rule documents and other technical documents.
Project 5
Name – MDM APL
Description – Scope of the project is to create workflows to get the master data from source using the API call to Oracle UCM and AWS SQS in the message format and extracting the valuable medical data from it which would be used by the end users to make decisions and draw conclusions. Build ETL pipeline to perform data cleansing operations and for storing historical data by applying SCD2 logic on data and exposing the dataset to S3 and Athena.
Technology Stack - PySpark, SQL, AWS Athena, AWS S3, Airflow, Github, AWS SQS, AWS Redshift
Responsibilities
1. Requirements gathering, and design technical document based on the functional requirements
2. Building workflows on Airflow for getting full data extracted from the source or getting changed entities in the message format using AWS SQS.
3. Perform data quality checks and apply data cleansing techniques.
4. Carry out incremental and full loads operations along with SCD Type 2 data loads.
5. Agile environment. Jira is used for tracking user stories.
6. Expose the data on Redshift/Athena/AWS S3 so that end users can easily query the data.
7. Create business rule document and other technical documents.
Project 4
Name - Oasis Claims Data Analytics – Building the Custom Queries
Description - Building some custom modules whose functionalities differ from the generic module designed previously. Leveraging the generic modules designed above and building a complete workflow that would ultimately result in the generation of claim and patient-level treatment details
Technology Stack - PySpark, SQL, Airflow, AWS Athena, AWS EMR, AWS S3, EC2 instances
Responsibilities
1. Building the modules per the business requirement. (Worked on 4 Therapy Areas – HCC, Heme, Alk, RA)
2. Conversion of MySQL queries/SAS code into the Spark Jobs.
3. Optimization of Spark Jobs using various techniques like broadcast joins, caching, and reducing unnecessary reprocessing of data.
4. Bug fixing and maintenance of the project.
5. Using Airflow to create a complete workflow by combining different Pyspark modules created
6. Requirements gathering, and designing technical documents based on the functional requirements
7. Testing the data to ensure the process quality.
Project 3
Name - G2O – Cloud Migration
Description - Project to migrate the on-premise process to cloud. Involved building of various healthcare trackers like HCC, RA, etc. in the AWS Sagemaker notebook for analysis purpose and creating the data that will be used for generating further insights. Exposing the dataset to Athena so that business can run simple SQL query to get the data and perform analysis.
Technology Stack - PySpark, SQL, Airflow, AWS Athena, AWS S3, Sagemaker Notebooks, Github
Responsibilities
1. Building tracker-specific Notebooks with the ETL logic based on the Business Rules
2. Automating the linking of the AWS Sagemaker Notebook with the EMR
3. Storing the data on AWS S3 and Athena for the end users
4. Creating business rule documents and other deliverable documents.
Project 2
Name - Oasis Claims Data Analytics – Building generic Modules
Description - Building some generic modules that would be leveraged along with some custom queries to get the complete patient claim history. These modules compute the patient’s complete treatment history on claim and patient level. These functionalities would be similar across various Therapy Areas which can be configured using the JSON Object passed.
Technology Stack - PySpark, SQL, AWS Jupyter Notebooks, AWS EMR, AWS RDS, Python libraries like Pandas, NumPy, Matplotlib, etc
Responsibilities
1. Building the modules per the business requirement.
2. Conversion of MySQL queries into the Spark Jobs.
3. Building the Jupyter Notebook to take the inputs from the users that would ultimately create a JSON Object and store it in RDS.
4. Creating Various Visualizations in the Jupyter Notebook based on the output generated after running the modules so that users get some insights into data in visual form using Python libraries like Pandas, Matplotlib, etc.
Project 1
Name - Canvas - Patient Journey
Description - Patient Journey is an analytics tool that processes the patient related medical data and mines some useful information from it. This information is then stored in intermediate files to generate visualizations. The visualizations are then used by data analysts/scientists for market predictions. Name - Oasis Claims Data Analytics – Building generic Modules Description - Building some generic modules that would be leveraged along with some custom queries to get the complete patient claim history. These modules compute the patient’s complete treatment history on claim and patient level. These functionalities would be similar across various Therapy Areas which can be configured using the JSON Object passed.
Technology Stack - Pyspark Developer PySpark, SQL, AWS Jupyter Notebooks, AWS Athena, AWS EMR, AWS S3, EC2
Responsibilities
1. RESTful API development using Python and Flask framework.
2. Conversion of MySQL queries into the Spark Jobs.
3. Optimization of Spark Jobs using various techniques like broadcast joins, caching, reducing unnecessary reprocessing of data
4. Consuming the data from S3 and building ETL pipeline using Airflow and running the jobs on EC2 cluster
5. Creating Jupyter notebooks for taking the parameters from end users which is passed as an input to the built process.
Experience 10+ Years
Work Style Remote
Duration Immediate
Location Ahmedabad