Event Driven Security Automation

The automation examples presented in the previous blog posts, asset data collection and EASM, were time triggered.

In this post we are going to use external events as triggers for launching security tasks. This allows to respond to something happening in other systems.

For reading this blog post you should be familiar with the post that introduced the security automation platform and have a basic understanding of Kubernetes concepts.

Argo Events

Argo Workflows only supports triggering workflows manually or via a time schedule. Argo Events can be used to trigger a workflow from a variety of event sources like webhooks, message queues, etc.

There are four main components in Argo Events to achieve this, all of them custom Kubernetes resources:

EventSource: defines where events are consumed from and how, e.g. message queue parameters and credential to access it
EventBus: events consumed by the EventSource from an external system are sent to this Argo Events internal event bus.
Sensor: defines how events from the internal event bus will trigger a workflow ^[1].
A trigger has it’s own resource type called Trigger

These components and how they interact with each other is shown in the diagram below.

The event message can be passed to the Argo Workflow as an input artifact.

We are going to walk through all of this with a specific example later, but first we have to deploy Argo Events into our Kubernetes cluster.

Deployment

Similar to Argo WF, Argo Events is Kubernetes (K8S) native. We therefore have to deploy it in a Kubernetes cluster.

There is a community maintained Helm chart that can be used.

I suggest to create new dedicated K8S namespaces:

One namespace for Argo Events itself
At least one additional namespace where the workflows will be executed that are triggered by Argo Events

You will have to modify the configuration of the already existing Argo WF deployment: the Argo controller must be aware of and granted access to the namespace(s) where the event driven workflows will be executed.

Assuming Argo Events has been successfully deployed, we have to create the event bus:

apiVersion: argoproj.io/v1alpha1
kind: EventBus
metadata:
  name: default
spec:
  nats:
    native:
      replicas: 3
      auth: none
      persistence:
        storageClassName: standard
        accessMode: ReadWriteOnce
        volumeSize: 4Gi

As there is exactly one event bus in Argo Events you only have to deploy this once with kubectl -n $(NS) apply -f eventbus.yml (use the namespace of the Argo Events deployment).

Argo Events is now ready. So let’s look at an implementation example now.

Ticketing System Credential Scans

Tickets in support systems can contain credentials. We have seen this in security incidents in the past where customer credentials were stolen from a support system. One way of addressing this problem is to scan all tickets for credentials. Argo WF, in combination with Argo Events, can be used to achieve this.

The high-level goal is: every time a ticket is created or updated, or files are attached to a ticket, we want to scan this content for credentials.

Architecture

The solution consists of multiple components. These are, as also shown in the figure below:

Ticketing system: basically every ticketing system supports configuring webhooks that can be triggered when certain events occur ^[2]. Such a webhook has to be configured so that it is triggered every time someone creates or updates a ticket.
Webhook: the ticketing system will call this webhook, whose implementation will take the provided message and publish it to a message queue
Message queue: messages from the ticketing systems will be stored here until Argo Events consumes them
Argo Events will take a message from the queue and trigger an Argo Workflow
The Argo Workflow, using the message as input, fetches the ticket data from the ticketing systems and performs a credential scan of the ticket data

If credentials have been found, those findings will be published by the workflow.

This is pretty straightforward. But before diving into the implementation details, we first have to define the workflow that performs the actual credential scans.

Workflow Model

As with all posts in this series, we define the workflow model as a graph:

The first task has to fetch the ticket content and it’s file attachments from the ticketing system.

Those files are then made available to the credentials scanning tasks, potentially using different scanners. There are various options available: Trufflehog, gitleaks, etc.

As mentioned in the previous blog post, findings can be sent to a SIEM, or directly to human users via email or a messaging system.

Implementation

With the architecture in place and the workflow model defined, we can now look into the implementation details. We will focus on the Argo Events deployment and configuration and how it integrates with Argo Workflows. In contrast to previous posts, we will only briefly discuss the workflow template that performs the actual credential scans.

We are making the following assumptions for our implementation, derived from the fact that we are deploying on GCP:

Argo WF and Argo Events are deployed on GCP GKE, GCP’s managed Kubernetes service
GCP Pub/Sub is used as message queue
GCP Cloud Run is used to deploy the webhook
GCP service accounts are created with sufficient permissions to access the required resources

The need for service accounts will be explained in each section where they are required.

Webhook

Unless you are managing the ticketing system yourself, the Webhook will usually have to be publicly reachable over the Internet ^[3], so that it can be called by the ticketing system.

The implementation should therefore require the caller (=ticketing system) to provide an API key or something similar for authentication purposes ^[4].

The rest of the implementation is rather simple: take the provided message and send it to a message queue. This can be implemented in only a few lines of code (omitted for brevity).

Info

For the remainder of this post, we assume that the message provided by the ticketing system is a JSON payload.

On GCP, the easiest way to deploy a web hook is by using the already mentioned Cloud Run Functions. On AWS, the equivalent is a Lambda function with a function URL that requires no AWS IAM authentication ^[5]. If you want to operate your own webhook infrastructure, there are frameworks for different languages available out there.

For our GCP deployment, we have to make sure that the Cloud Function (that hosts the web hook) has permissions to send messages to the GCP Pub/Sub based message queue. We therefore associate a GCP service account with the Cloud Function, where the service account has the required permissions.

Message Queue

As already mentioned, we are using GCP Pub/Sub as our message queue. A dedicated subscription is created, exclusively for publishing events that originate from one ticketing system.

We need some additional things so that Argo Events can access those messages:

Create a GCP service account that has the required permissions to consume messages from the GCP Pub/Sub subscription
The K8S namespace where Argo Events has been deployment must have a K8S service account
This K8S service account must be permitted access to the GCP Pub/Sub API. This can be achieved with workload identity that allows a K8S service account to obtain a credential for the GCP service account from (1).

When you are deploying to AWS you can use AWS SQS. When using AWS EKS, you can then use either IAM roles for service accounts or alternatively EKS Pod Identities for granting the K8S service account access to AWS SQS API.

If you want to host your own messaging system, alternatives are Kafka, NATS, etc.

Whatever solution you choose, it must be supported by Argo Events! You can find a list of supported messaging systems in the Argo Events documentation.

Next, we are finally diving into the specification of the Argo resources.

Event Source

We first need to configure the EventSource object that defines from where Argo Events will consume messages.

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: gcp-pubsub-ticket-system
spec:
  template:
    serviceAccountName: argo-events 
  pubSub:
    jira_cloud:                     
      jsonBody: true
      subscriptionID: ticket_sub

	Argo Events will use this K8S service account when accessing the event source
	Our event source is GCP Pub/Sub (`pubSub`) and Argo Events will listen for events/messages on the subscription with ID `ticket_sub`

Argo Events is going to monitor the GCP Pub/Sub subscription for new messages that were pushed into GCP Pub/Sub by the webhook.

Sensor

The sensor connects the event source with a trigger that defines what is going to happen when a message arrives in the GCP Pub/Sub subscription.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: sensor-ticketing
spec:
  template:
    serviceAccountName: argo-events
  dependencies:                              
    - name: dep-ticketing
      eventSourceName: gcp-pubsub-ticket-system
      eventName: jira_cloud
  triggers:                                  
    - template:
        conditions: "dep-ticketing"
        name: dep-ticketing
        argoWorkflow:                         
          operation: submit
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: credentialscan-tickets-
                namespace: argo-events-workflows
              spec:
                workflowTemplateRef:         
                  name: ticket-secrets-scan
                arguments:
                  artifacts:
                  - name: message            
          parameters:
            - src:
                dependencyName: dep-ticketing
                dataKey: body
              dest: spec.arguments.artifacts.0.raw.data

	This sensor listens for events from the previously defined event source (`gcp-pubsub-ticket-system`)
	Only one trigger is defined that waits for an event emitted by the event source
	When the trigger fires it will launch the Argo Workflow defined in this section (`argoWorkflow`)
	Reference to the `WorkflowTemplate` that contains the actual credential scan implementation
	The workflow is provided with one input file (artifact) that is taken from the body of the message that originated from the event source (message queue)

When a new message arrives in the message queue (represented by event source gcp-pubsub-ticket-system), a new Argo Workflow is created. The content of the event message (key body) is provided as input (parameter name message) to the workflow. The implementation of the workflow logic itself is defined in a template (ticket-secrets-scan).

WorkflowTemplate

We only provide a basic overview of the template that implements the credential scanning logic.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: ticket-secrets-scan
spec:
  entrypoint: main
  templates:
  - name: main
    inputs:
      artifacts:             
      - name: message
    dag:
      tasks:
        - name: fetch-ticket                    
          templateRef:
            name: atlassian-event-payload
            template: main
          arguments:
            artifacts:
            - name: message
              from: "{{inputs.artifacts.message}}"
        - name: scans                           
          templateRef:
            name: helper-secrets-scans
            template: main
          depends: "fetch-ticket"
          arguments:
            artifacts:
            - name: files-to-scan
              from: "{{tasks.fetch-ticket.outputs.artifacts.files_to_scan}}"
        - name: publish                         
          templateRef:
            name: publish-credential-findings
            template: main
          arguments:
            artifacts:
            - name: results
              from: "{{tasks.scans.outputs.artifacts.results}}"

	The original message from the ticketing system is provided as input
	Download the ticket data from the ticketing system
	Scan the files that have been fetched in the previous task for credentials
	Publish findings

This is basically the workflow model that we defined before, implemented as an Argo WorkflowTemplate.

We do not provide the details for the sub-tasks. Their implementation is rather straightforward though.

Deploying

All resources can be deployed as follows, assuming the Kubernetes namespace $NS is used for the Argo Events deployment and $NS2 is used for the workflows to be executed.

$ argo template create scan_workflow_template.yml -n $NS2
$ argo template create event_source.yml -n $NS
$ argo template create sensor.yml -n $NS

The order is important: the workflow template (scan_template.yml) has to be created before the event source and the sensor, as there are dependencies between these objects.

Once deployed, every time somebody creates or updates a ticket in the ticketing system, the webhook will be called that will forward the message from the ticketing system to the GCP Pub/Sub message queue. The Argo Events event source will consume this message and the sensor will then trigger the workflow that will perform a credential scan of the affected ticket.

The following screenshot shows a slightly more complex deployed sensor configuration on the Argo WF web UI:

The sensor itself is in the center of the image, with three event sources (different message queues) to the left and three triggers (all launching workflows) to the right.

The credential scan workflows executed by this configuration are visible on the Argo WF web UI, just like any other workflow:

Summary

This post shows how to trigger workflows from external events, using Argo Events. This integrates nicely into the overall Argo Workflows ecosystem. While setting up the sensors might take some time to get used to at first, you will become accustomed to this really fast.

For the provided use case, the time duration from a ticket being created or updated to having a credential scan result available is just a few seconds. ^[6]

This is a very powerful framework for driving various types of security automation! The ability to launch tasks in almost "real-time" in response to various external events enables a lot of security use cases.

1. For completeness: Argo Events can not only launch Argo Workflows but can also create other Kubernetes objects.

2. This is for example supported by Jira

3. For 3rd party managed ticketing systems, the source IP might not be known, or the source IP CIDR range is too large for setting up firewall rules.

4. When your webhook is publicly reachable over the Internet it can be called by anyone. It is therefore important to require some form of authentication.

5. You should still implement something like an API key authentication within your webhook though.

6. Most of the time is actually spent in the credential scans. Everything else happens in milliseconds.