Vishnu Vikas Nallamalli

Designed and implemented a scalable AWS-based Enterprise Data Lake platform to support regulatory reporting, customer analytics, risk management, transaction monitoring, and enterprise-wide data consumption initiatives. Architected and developed batch and real-time data ingestion frameworks using Amazon EMR, AWS Glue, Amazon Kinesis Firehose, Amazon S3, and Apache Spark for processing high-velocity transactional and market datasets. Led end-to-end cloud data platform architecture assessments and implemented AWS services including Amazon EMR, Amazon Redshift, Amazon S3, Dynamo DB, Sage Maker, Lambda, and Step Functions to enable secure and scalable analytics solutions. Engineered robust data ingestion pipelines integrating data from core transactional systems, payment platforms, market feeds, relational databases, and external vendor sources into centralized cloud-based repositories. Developed enterprise-grade ETL and ELT frameworks using PySpark, Scala, Spark SQL, AWS Glue, and Hive to support large-scale data transformation and processing requirements. Built reusable migration accelerators leveraging Spark Data Sources, Hive Metastore, schema evolution strategies, and metadata-driven processing for modernizing legacy data platforms. Implemented secure data access controls using AWS IAM, Lambda, Dynamo DB, bucket policies, and encryption standards to meet stringent governance and compliance requirements. Established Kerberos-based authentication and authorization mechanisms across Hadoop ecosystems, enabling secure access to HDFS, Hive, Pig, YARN, and Map Reduce environments. Developed high-performance Spark applications using Scala and PySpark to process large-scale structured, semi-structured, and unstructured datasets across distributed environments.

Created optimized Spark SQL solutions utilizing Data Frames, Dataset APIs, partitioning strategies, and caching mechanisms to improve analytical processing efficiency. Developed advanced Teradata BTEQ scripts and SQL-based transformation processes supporting historical tracking, data reconciliation, deduplication, and master data management requirements. Managed enterprise data migration and integration initiatives using SQL Server Integration Services (SSIS), DTS packages, and cloud-native data movement services. Implemented Master Data Management processes and reference data governance standards through data cleansing, transformation, enrichment, and validation methodologies. Built machine learning data pipelines in Python to support predictive analytics, customer behavior modeling, operational forecasting, and intelligent decision-making use cases. Automated machine learning workflows using AWS Step Functions and Sage Maker for model training, validation, deployment, monitoring, and inference orchestration. Integrated Apache Airflow with AWS services to orchestrate complex multi-stage workflows across EMR clusters, Sage Maker environments, and cloud-based data platforms. Developed AWS Lambda functions using Boto3 SDK to automate infrastructure operations, governance controls, metadata management, and cloud resource optimization activities. Optimized Amazon Redshift data warehouses through workload management, query performance tuning, distribution strategies, sort key design, and storage optimization techniques.

Designed and maintained CI/CD pipelines using AWS Code Pipeline, Code Build, Code Deploy, Git, and Jenkins to automate deployment and release management processes. Implemented Infrastructure as Code solutions using Terraform and AWS Cloud Formation to provision secure, scalable, and repeatable cloud environments. Developed centralized observability and monitoring solutions using Elasticsearch, Kibana, Cloud Watch, and Cloud Trail to support operational intelligence and anomaly detection. Created governed analytical datasets using Alteryx, advanced SQL, and Tableau to deliver executive dashboards, operational reports, and self-service analytics capabilities. Established enterprise data governance frameworks encompassing metadata management, lineage tracking, audit controls, encryption using AWS KMS, and regulatory compliance standards. Led performance tuning initiatives for Spark workloads, EMR clusters, Redshift environments, and distributed data pipelines supporting large-scale enterprise datasets. Environment: Amazon Web Services, Apache Spark (Scala, PySpark, Spark SQL), Hadoop, Apache Airflow, Elasticsearch, Logstash, Kibana, Teradata (BTEQ), Microsoft SQL Server Integration Services, SQL Server, HBase, Python (Boto3, ML), Alteryx, Tableau, CI/CD, IAM, KMS

Built end-to-end data ingestion pipelines using Azure Data Factory to integrate on-premises sources such as MySQL and Cassandra with cloud platforms including Azure Blob Storage, ADLS Gen2, and Azure SQL Database, loading curated data into Azure Synapse Analytics. Developed batch and real-time data processing pipelines using Azure Databricks with Apache Spark, Spark SQL, PySpark, and Scala for processing claims and patient data feeds. Architected scalable data lake solutions using Azure Data Lake Storage Gen2 following medallion architecture principles for structured and semi-structured healthcare datasets. Migrated large-scale data from Hadoop and HDFS environments to Azure Databricks and Azure Synapse Analytics while ensuring data integrity and governance. Created and administered Databricks clusters with auto-scaling configurations and implemented Spark performance tuning strategies. Mounted Azure Blob Storage and ADLS Gen2 within Databricks for distributed processing and data curation. Built Spark Streaming applications consuming real-time event data from Kafka using both stateless and stateful transformations. Implemented ETL and ELT frameworks using Azure Data Factory, Databricks, Hive, and Snowflake SnowSQL to process structured and semi-structured data formats including JSON, Avro, Parquet, ORC, XML, and CSV.

Developed Hive data warehouse solutions implementing partitioning and bucketing strategies to optimize query performance. Engineered automated workflows using Azure Logic Apps and orchestrated containerized workloads using Azure Kubernetes Service for scalable data processing. Deployed and managed Spark applications in Kubernetes environments to support distributed analytics. Monitored Spark and Hadoop clusters using Azure Log Analytics, Azure Monitor, and Ambari Web UI to ensure operational stability. Transitioned log storage from Cassandra to Azure Synapse dedicated SQL pool to enhance reporting and analytics capabilities. Implemented secure data pipelines using Azure Key Vault for secrets management and deployed solutions across environments using ARM templates and Azure Dev Ops CI/CD practices. Developed reusable Databricks notebooks and PySpark libraries to standardize data transformation, cleansing, and enrichment processes. Worked extensively with Azure Cosmos DB using SQL API and Mongo API for scalable storage of -related semi-structured data.

Developed Map Reduce jobs, Pig scripts, Hive queries, and custom UDFs for large-scale data transformations within Hadoop ecosystems. Utilized Data Stax Spark Connector to integrate Cassandra with Spark-based analytics platforms. Designed and maintained Oozie workflows for scheduling and managing complex Hadoop job dependencies. Worked in hybrid cloud environments leveraging AWS Glue with PySpark dynamic frames, crawlers, and workflow scheduling for cross-platform data integration. Developed interactive dashboards and enterprise reports using SSRS and Power BI, including drill-down, parameterized, matrix, and executive reporting dashboards for stakeholders. Environment: Azure, Apache Spark (Spark SQL, PySpark, Scala), Delta Lake, Apache Kafka, Apache Hive, Apache Hadoop (HDFS, Map Reduce, Pig, Oozie), Apache Cassandra, Azure Cosmos DB (SQL & Mongo API), Snowflake, Power BI, SQL Server Reporting Services, Azure Dev Ops, Azure Key Vault; MySQL, JSON

Designed and developed scalable batch and real-time data processing applications using Apache Spark, PySpark, Scala, Kafka, Hive, and Hadoop ecosystem technologies to process large volumes of transactional and operational data. Built robust data ingestion frameworks to collect and integrate structured, semi-structured, and unstructured data from payment platforms, CRM systems, operational databases, flat files, and third-party service APIs. Developed Spark SQL and PySpark-based ETL pipelines to cleanse, validate, standardize, enrich, and transform raw data into analytics-ready datasets for reporting and advanced analytical use cases. Implemented Apache Kafka-based streaming solutions to capture, process, and distribute event-driven transactional data across enterprise data platforms with low-latency processing requirements. Designed and developed Apache Flink streaming applications to consume Kafka events, perform real-time transformations, enrich incoming datasets, and publish processed records for downstream analytical systems. Utilized Apache Avro schemas for data serialization, schema evolution, and efficient data exchange across distributed processing environments and streaming platforms. Developed Spark Structured Streaming applications to process near real-time transaction data, perform anomaly identification, and persist curated datasets into HBase and Amazon S3 repositories.

Created reusable shell scripts, automation utilities, and framework components for Hive, Sqoop, Flink, Spark, and Pig workloads to streamline data engineering development activities. Developed Hive external and managed tables, optimized partitioning strategies, and implemented efficient querying techniques to support large-scale analytical workloads. Performed data extraction from relational databases using Sqoop and integrated high-volume datasets into distributed Hadoop environments for enterprise analytics consumption. Assisted in administration, monitoring, troubleshooting, and operational support of Hadoop, Spark, Hive, Kafka, Splunk, and Tableau environments. Designed and maintained Tableau dashboards and analytical reports that enabled business users to monitor operational trends, transaction activity, customer behavior, and performance metrics. Developed REST API integrations to exchange data between enterprise applications, external systems, and cloud-based platforms while ensuring secure and reliable data movement. Built scalable cloud-based data ingestion and processing solutions using Amazon EMR, Apache Spark, Python, and Amazon S3 for enterprise data lake initiatives.

Worked extensively with AWS services including EMR, S3, Glue, Lambda, Redshift, Athena, IAM, EC2, VPC, Cloud Watch, and SNS to support cloud-native data engineering solutions. Developed AWS Glue ETL jobs using PySpark and Spark SQL to transform and load enterprise datasets into Redshift and other analytical repositories. Optimized Spark and PySpark application performance through effective partition management, join optimization techniques, memory tuning, caching strategies, and resource allocation improvements. Assisted in building CI/CD pipelines using Git, Jenkins, AWS Code Pipeline, and AWS Code Build to automate application deployment, testing, and release management processes. Environment: Apache Spark, PySpark, Scala, Apache Kafka, Apache Flink, Hadoop, HDFS, Hive, Sqoop, Pig, Spark SQL, Spark Structured Streaming, HBase, Apache Avro, Splunk, Tableau, Python, Shell Scripting, REST APIs, AWS, Git, Jenkins, CI/CD, Data Lake, ETL, Data Warehousing, Real-Time Streaming, Data Governance, Agile, SDLC

Migrated enterprise databases from SQLite to MySQL and Postgre SQL while maintaining data integrity, schema consistency, and seamless business continuity throughout the migration lifecycle. Developed and optimized complex SQL queries, joins, views, stored procedures, and database objects to support reporting, analytics, and operational data requirements. Performed data mapping, schema analysis, and data transformation activities to ensure consistency between source and target systems during migration projects. Built scalable data processing solutions using Python to handle structured, semi-structured, and transactional datasets from diverse sources. Developed automated data ingestion and conversion frameworks using Python and Bash scripting to streamline recurring integration processes. Created reusable Python modules for data cleansing, validation, normalization, and transformation to improve overall data quality and consistency.

Implemented database connectivity solutions using Python, SQLAlchemy, and database connectors for efficient data access and integration. Developed RESTful APIs using Python and Django to enable secure data exchange and integration between internal and external systems. Integrated messaging platforms using AMQP and Rabbit MQ to support asynchronous data processing and event-driven data workflows. Built data pipelines for processing, aggregating, and transforming large datasets to support reporting and operational business functions. Utilized Python libraries including Pandas, NumPy, Itertools, Functools, Glob, Random, and Math for advanced data manipulation and transformation tasks. Developed data visualizations using Matplotlib, Seaborn, Highcharts, HTML, CSS, and Java Script-based components for analytical reporting.

Generated operational and analytical reports using SQL Server Reporting Services (SSRS), including tabular, matrix, chart, and parameterized reports. Customized SSRS reporting solutions using URL access configurations to support dynamic report generation and distribution requirements. Performed root cause analysis, debugging, and issue resolution for database, ETL, and integration-related challenges. Environment: Python, Django, SQLAlchemy, MySQL, Postgre SQL, SQLite, SQL Server, Rabbit MQ, Git, Postman, SSRS, HTML, CSS, jQuery, Matplotlib, Seaborn, RESTful API, ETL