I am part of a team that works on a business critical microservice that captures a Submission from end users. A request submitted by a party flows through a workflow of approvals and at the end creates / generates sides effects in downstream services. There are several downstream services, each managed and deployed as separate applications by their own team. Challenges The service I worked on acted as a gatekeeper for data entering into these downstream service....
I was working at a project that involved munching data generated by multiple data sources, transforming it and making it available for a downstream service that would index this data into a data store that could make it easy to run analytical workloads at run time. stack used: azure-data-factory flink azure-function azure-entra azure-storage-account elasticsearch What were the challenges? The original team that built this workflow used Flink which is an awesome tool for transforming data streams....
The company that I worked for had an interesting problem to solve. They had recently “acquired” a consultant who provided custom reporting services to higher education institutions which was highly labor intensive and manual. We were asked to fully automate the workflow incrementally AND onboard existing clients powerbi azure-storage azure-data-factory azure-web-app powerbi-embeded azure-functions azure-entra What were the challenges? The existing process was very custom and manual in nature. The consultant custom transformed data files from the institutions, loaded them into an on prem database to perform transformations and imported the results into a power bi report....
I was working as a full time Senior Software Developer in a project that indexed data generated by multiple data sources. This legacy system had outgrown from its initial single responsibility of “managing indicies in ElasticSearch” to handle more things were vague but still could be bucketed in the managing indicies bucket The product that used this service had a requirement for a data source that could handle analytical query workload....
In 2018, I was involved in designing and building a workflow that would allow sharing data between multiple products of the same campany on a single enterprise platform. The company had an enterprise contract with Microsoft Azure, with almost all of their infrastructure built and deployed across regions in azure. Building this workflow using the tools available in azure was a no brainer. Stack azure-data-factory azure-kubernetes-services azure-functions dotnet-core node elasticsearch kanban OKRs azure-devops...
I worked on a project that involved sending surveys to students enrolled in higher education degree institutions at end of the semester. serverless azure-functions storage queue micro-service scaling What were the challenges? We used a legacy windows service built on dotnet framework with a crusty cron job system built on top of Quartz to send emails The process was chunky and could not scale to the demands of the growing product This was a multi-tenant service however, issues in one tenant affected performance and quality of service offered to other tenants There were too many unknowns about if and when the emails would be delivered, this was more of a spaghetti implementation of the older service It lacked tools for internal support people and clients to manage expectations and achieve their goals Solution?...