Projects

I am part of a team responsible for a business-critical service that captures Submissions from end users. A submission navigates through a series of approval workflows, ultimately generating side effects across various downstream services. These downstream services are independently managed and deployed by separate teams, each overseeing their own distinct applications Challenge The service played a role of a gatekeeper for data entering the system. Upon approval, the Submission triggered multiple CRUD operations across downstream systems. In our domain, the Submission in our domain was supposed to be IMMUTABLE. ...

I was working at a project that involved munching data generated by multiple data sources, transforming it and making it available for a downstream service that would index this data into a data store that could make it easy to run analytical workloads at run time. stack used: azure-data-factory flink azure-function azure-entra azure-storage-account elasticsearch What were the challenges? The original team that built this workflow used Flink which is an awesome tool for transforming data streams. However this decision involved the development team to manage its Flink installation (it was 2019, the company that I worked for was based out of Azure and at that point of time Azure did not have a managed Flink service). ...

The company that I worked for had an interesting problem to solve. They had recently “acquired” a consultant who provided custom reporting services to higher education institutions which was highly labor intensive and manual. We were asked to fully automate the workflow incrementally AND onboard existing clients powerbi azure-storage azure-data-factory azure-web-app powerbi-embeded azure-functions azure-entra What were the challenges? The existing process was very custom and manual in nature. The consultant custom transformed data files from the institutions, loaded them into an on prem database to perform transformations and imported the results into a power bi report. The report was then emailed or shared via powerbi service. The institution would then go back and forth with the consultant about to tweek the reporting bits. This was NOT scalable engineering bits We setup a dotnet based website for presentation would leverage powerbi embedded with app owns the data strategy to render embedded reports consultant would get the data files from the clients and upload it to azure storage using the web app. we used azure data factory to orchestrate ETLing the data files into the OLAP data model needed for powerbi we setup custom dotnet service that would load / refresh powerbi report from the processed output files from azure data factory. learnings powerbi is a PITA unless you know what you are doing. there are quite a few levels that you need to get it right for it to work as you expect. It greatly reduced the efforts to put reports in a consumable form, however scaling it has its challenges we delivered the workflow iteratively where initially consultants were given access to load the files directly into azure, which was later replaced by a webapp. we continued to deliver the reports by still running the most of the transformations on top of SQL server, thereby avoid releasing large changes with unwanted side effects.

I was working as a full time Senior Software Developer in a project that indexed data generated by multiple data sources. This legacy system had outgrown from its initial single responsibility of “managing indicies in ElasticSearch” to handle more things were vague but still could be bucketed in the managing indicies bucket The product that used this service had a requirement for a data source that could handle analytical query workload. This was 2018, and with the skills and experience available at hand Elasticsearch was chosen the data store. As this was a multi-tenant service it was decided to have an index per client and a metadata index to hold a mapping between tenant-id and index ...

In 2018, I was involved in designing and building a workflow that would allow sharing data between multiple products of the same campany on a single enterprise platform. The company had an enterprise contract with Microsoft Azure, with almost all of their infrastructure built and deployed across regions in azure. Building this workflow using the tools available in azure was a no brainer. Stack azure-data-factory azure-kubernetes-services azure-functions dotnet-core node elasticsearch kanban OKRs azure-devops ...

I worked on a project that involved sending surveys to students enrolled in higher education degree institutions at end of the semester. serverless azure-functions storage queue micro-service scaling What were the challenges? We used a legacy windows service built on dotnet framework with a crusty cron job system built on top of Quartz to send emails The process was chunky and could not scale to the demands of the growing product This was a multi-tenant service however, issues in one tenant affected performance and quality of service offered to other tenants There were too many unknowns about if and when the emails would be delivered, this was more of a spaghetti implementation of the older service It lacked tools for internal support people and clients to manage expectations and achieve their goals Solution? scalable We designed and implemented an end to end replacement built on top of azure functions and azure storage tables and queues. Azure Durable functions were still out yet. Emphasis was given to make the system transparent to troubleshoot and manage the workflow for developers, internal support staff and clients alike release strategy Email as a presentation layer gives you zero room for making mistakes that can be corrected. We released this iteratively moving few tenants over at a time engineering bits the system consisted of four actors ...