hirak06datascience

• Created and maintained optimal data pipeline architecture in cloud Microsoft Azure using Data Factory and Azure Databricks. • Developed pipelines in ADF using Linked services, Datasets, and Pipelines to extract, transform, and load data from sources such as Teradata, Blob storage, and Azure SQL Data Warehouse. • Built the Oozie pipeline for file movements, Sqoop data transfers from Teradata or SQL, and exports into Hive staging tables with subsequent business-driven aggregations and main table loads. • Established infrastructure for optimal ETL processes from diverse data sources using SQL and big data technologies like Hadoop Hive and Azure Data Lake Storage. • Collaborated on ETL tasks to maintain data integrity and pipeline stability. • Applied data modeling, ETL processes, and data warehousing concepts within Power BI and QlikView environments. • Designed and implemented database solutions with Azure Blob Storage for data storage and retrieval. • Deployed data factory pipelines to orchestrate data flows into SQL Databases. • Maintained data processing solutions using Azure HDInsight, Azure Databricks, and Azure Stream Analytics for real-time and batch data processing. • Cleansed, manipulated, and analyzed large datasets, including semi-structured and unstructured data (XML, JSON, CSV, PDFs) using Python. • Developed Python scripts for data filtering, cleansing, mapping, and aggregation. • Designed and developed ETL workflows in Informatica, leveraging Informatica Designer components to build complex mappings, sessions, and workflows for an Enterprise Data Warehouse. • Created informative dashboards and data visualizations in Power BI, enabling business users to make data-driven decisions. • Employed Big Query to manage and analyze large datasets, optimizing queries for efficient and scalable data processing. • Translated business requirements into secure and maintainable code, aligning technical solutions with business objectives. • Conducted extensive data analysis and root cause analysis to troubleshoot issues and improve data quality. • Reviewed and streamlined business processes, translating them into effective BI reporting and analysis solutions to enhance operational efficiency. • Adhered to SDLC processes and various project management methodologies to ensure timely and successful project delivery. • Utilized data modeling techniques (star/snowflake schema design, data marts, slowly changing dimensions) to create physical and logical data models for data warehouses. • Identified and resolved BI application performance bottlenecks through thorough analysis and tuning, enhancing overall system performance.

• Led various phases of SDLC from requirement gathering to testing, delivering scalable data solutions for healthcare analytics. • Designed and implemented a streaming platform using AMQ-Streams, Kafka, Camel, and Spring, enhancing real-time data processing capabilities. • Utilized Sqoop for efficient data loading into Spark SQL for RDDs, Datasets, and DataFrames creation. • Implemented data formats such as Avro, Parquet, ORC, and JSON, and developed UDFs in Hive and Pig for advanced data processing. • Managed Change Data Capture (CDC) using Qlik Replicate and automated data loading into HDFS. • Developed Airflow scheduling scripts in Python for orchestrating complex data workflows. • Led end-to-end machine learning workflows, from data gathering and preprocessing to model evaluation and deployment using Azure and Snowflake data sources. • Implemented cloud computing solutions with HDInsight, Azure Data Lake, Azure Data Factory, and Azure Machine Learning, leveraging PowerShell scripting. • Developed Spark scripts in PySpark for data processing and analytics. • Built applications using Django and Flask frameworks, integrating REST APIs and leveraging dependency injection with Spring Framework. • Designed and implemented data pipelines using Oozie and Airflow for efficient data processing and workflow automation. • Utilized K-Streams for real-time data streaming and analytics, enhancing decision-making capabilities. • Proficient in developing data-driven stories and advanced data analysis using QlikView and Power BI, creating intuitive visualizations for business users. • Deployed and configured Power BI in cloud and on-premises environments, ensuring data security and automatic report refresh. • Implemented Snowflake and Data Vault modeling approaches for managing large datasets and metadata. • Optimized data processing using Spark's in-memory capabilities, efficient joins, and transformations. • Implemented machine learning algorithms including linear regression, logistic regression, decision trees, random forests, and XGBoost for predictive analytics. • Developed machine learning models using recurrent neural networks (LSTM) for time series analysis and predictive modeling. • Automated processes using Ansible Python API and managed collections in Python for efficient data manipulation. • Integrated Workday HCM modules and managed data pipelines using Fivetran and HVR for data replication into Snowflake.

6% saving in costs for component manufacturing giant by designing end-to-end solution – cloud migration, building predictive models

Hadoop Ecosystem and Data Integration: • Utilized all Hadoop ecosystems (HDFS, YARN, MapReduce, Hive, Flume, Oozie, Zookeeper, Impala, HBase, Sqoop) through Cloudera Manager. • Collected and aggregated large volumes of log data using Apache Flume, staging it in HDFS for further analysis. • Created core Python API used across multiple modules for seamless integration. • Implemented data ingestion and managed clusters for real-time processing using Kafka. • Set up multi-hop, fan-in, and fan-out workflows in Flume to streamline data processing. • Imported transactional logs from web servers into HDFS using Flume. • Developed custom sterilizers and interceptors in Flume to mask confidential data and filter unwanted records from event payloads. Data Analytics and Machine Learning: • Designed and implemented data analytics solutions in the Hadoop ecosystem using MapReduce Programming, Spark, Hive, Pig, Sqoop, HBase, Oozie, Impala, and Kafka. • Converted Hive/SQL queries into Spark transformations using Spark RDDs, Python, and Scala. • Developed parser and loader MapReduce applications to extract data from HDFS and store it in HBase and Hive. • Wrote user-defined functions (UDFs) in Hadoop PySpark for data transformations and loads. • Built various machine learning models (Logistic Regression, KNN, Gradient Boosting) using Pandas, NumPy, Seaborn, Matplotlib, and Scikit-learn in Python. • Experimented with ensemble methods to enhance model accuracy, deploying models on AWS. Database Management and Development: • Extracted data from Teradata into HDFS/Dashboards using Spark Streaming. • Worked on MongoDB database concepts including locking, transactions, indexes, sharing, replication, and schema design. • Automated RabbitMQ cluster installations and configurations using Python/Bash. • Workflow Automation and Batch Processing • Utilized OOZIE for batch processing and dynamically scheduling workflows.