KEDA ScaledJobs

This is a typical use-case that crops up in a backend service. A user calls an endpoint to change the state of a system. Say they wanted to enforce Multi Factor Authentication. This setting needs to be cascaded across ALL users for the system (or the tenant the user manages). Rolling out these changes is an open ended operation that needs to be batched and can’t be handled within the life cycle of a request.

• You have to manage updates to states of N items that might be too big for an http request
• You want to off load the processing to a separate backend service as you don’t want side effects of this operating affecting (processing N items in the current process might be resource intensive)
• You still want the end user to get an accurate state of the operation (failed, succeeded, operation_queued, processing)

The current architecture of the backend service that I received ownership of, had a pattern where an in- process thread ran in the background, polled the database for such a request (a record in the database with a State column) every xyz seconds. The operation was processed in this long running thread that was outside http request life cycle.

There were several advantages to this approach namely, keep it simple the workflow being simple maintenance and deployment wise. Observability wise it was a dream as all associated to the process were logged under the same owner process.

However there were few points that stuck out. tight coupling The backend-api was tightly coupled to authorizations needed to downstream subsystems (for our multi factor authentication example, say it needed management permissions on AWS Cognito) that were touched in the operation. wasted cpu If the operation was not high frequency, it would lead to uncessary polling. concurrency controls Horizontal scaling employed on the service meant we needed a concurrency system that used a database lock to ensure only a single instance would pick up and do the actual work which was not scalable.

Requirements

• backend api queues a job
• separate process picks up job and processes it. this de-couples the level of authorization on resources between the api and the background worker
• de-couple the scaling between api and worker

We decided the http endpoint would trigger a SQS event. The challenge here was to setup a workload lifecycle in kubernetes.

Solution

The HTTP endpoint would persist the details of the job in the database and then hand off the actual work to run by publishing a SQS message. The real challenge was designing the workload lifecycle on Kubernetes that would trigger a work load + scale it via configuration.

Enter KEDA

KEDA is an operator that can be installed on kubernetes cluster (we used helm package management for handling operator installations) and can be configured to listen to variety of triggers (SQS in our case) and can be made to trigger workloads and helps with scaling the workloads.

helm install keda kedacore/keda --namespace keda --create-namespace

Once installed, KEDA is made up of three parts worth understanding:

• Operator. It watches a custom resource (the ScaledJob) whose definition includes a trigger. The operator monitors that trigger and connects to the external event source. It runs under a service account that we assign an identity to, that identity is what grants access to the SQS queue. Triggers come in many flavors (aws-sqs-queue, redis, and more).
• Metrics server. This is what actually drives scaling. Behind the scenes it feeds the Horizontal Pod Autoscaler (HPA), which decides how many pods/jobs to run.
• Admission webhook. A guardrail that catches syntax mistakes and duplicate/conflicting definitions before they land in the cluster.

ScaledJob vs ScaledObject

KEDA offers two scaling primitives, and the choice matters here:

• A ScaledObject scales a long-running Deployment up and down. That’s still a persistent process — just an autoscaled version of the poller I was trying to get rid of.
• A ScaledJob spins up an ephemeral Kubernetes Job per unit of work. The job starts, drains its messages, and exits.

For “pick up a message, do a big chunk of work, then stop,” the ScaledJob was the right fit. Two properties made it click:

• Scale to zero. When the queue is empty, nothing runs. No idle poller, no idle cost.
• Scale on queue depth. As the queue grows, KEDA launches more jobs — scaling is driven by the actual backlog instead of a number I guessed.

A stripped-down version of the resource looked like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: async-worker
spec:
  jobTargetRef:
    template:
      spec:
        serviceAccountName: async-worker   # IAM identity with access to the queue
        containers:
          - name: worker
            image: <registry>/async-worker:latest
        restartPolicy: Never
  minReplicaCount: 0        # scale to zero when the queue is empty
  maxReplicaCount: 20
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: <sqs-queue-url>
        queueLength: "5"    # target messages per job
        awsRegion: us-east-1
      authenticationRef:
        name: keda-aws-credentials

Authorizations

• We handled setting up infrastructure using terraform for IaC
• KEDA operator was assigned an IAM idenity (keda-operator-iam) that had permissions to assume roles
• A trigger was created with iAM identity (trigger-iam) that had permissions on SQS queue
• A kubernetes service account was setup associated to a IAM role (worker-iam) that had permissions on SQS queue (read, delete). The worker pod would use this service account

Putting it all together

• user would trigger the action
• backend-api would persist a record in the database + raise an event in SQS (more on this to do in a two phase commit way)
• keda metrics server + operator identity a trigger threshold is met, uses HPA to schedule worker pod
• worker pod fetches SQS messages (1 or more) processes the message
• important the worker process is resposible for deleting the queue message marking the request was processsed

Learnings

• Event-driven beats polling for bursty async work. Letting queue depth drive scaling — and scaling to zero when it drains — was both cheaper and simpler than a service that ran around the clock.
• Ephemeral jobs isolate failures. A per-message Job that starts and exits is far easier to reason about than a shared, long-lived poller holding state in memory.
• Let the platform own scaling. Handing scaling decisions to KEDA and the HPA removed a whole category of manual tuning and “is it keeping up?” anxiety from our plate.