Qualys vulnerability management automation guide

To begin to manage risk for your digital footprint, you require some form of Vulnerability Management (VM). Much of the work revolves around education, awareness, and overcoming friction. Embedding security baselines as a gating mechanism into project lifecycles is often a hard-fought battle, but one that works best for everyone if done as early as possible. 

Expectation setting around any security gating should happen from the outset of a project. Still, it can sometimes be perceived as a surprise, labeled as a blocker, or result in acrimonious finger-pointing right before go-live. Yet, by empowering non-security staff and project management to perform the initial scans themselves (i.e. to self-serve), security teams can frictionlessly enable product and development teams to adhere to their go-live requirements faster and more easily.

Let’s look at how to automate aspects of your vulnerability management, allowing you to scale yourself, your team, and even get some precious time back, all without any ensuing grumpiness!

Qualys Automation and APIs 

These days Qualys is so much more than just Vulnerability Management software (and related scanning), yet enumerating vulnerabilities is still as relevant as it ever was. As a cornerstone of any objective security practice, identifying known unknowns is not just achievable, but something that’s countable and measurable in terms of real risk. Today, we’re going to focus on leveraging some basic Qualys automation to maximize your impact and efficacy. Let’s start by familiarizing ourselves with the Qualys VM/PC REST API. We will combine some simple steps into a more complex (but not complicated) outcome.

When you first encounter any API, the first few questions tend to be:

1. Where and what sort of documentation does the API have?

2. What specific endpoint do I access?

3. How do I authenticate?

4. Are there any limitations (including rate limits) or 'gotchas'?

5. What tooling can I use to quickly prototype and test? 

Qualys API Documentation 

• Searchable Web API Guide here.

• Official PDF API guide here and ‘quick’ reference here.

• API client examples here for a whole range of languages!

• Developer API forum here.

Note: We were, unfortunately, unable to locate an OpenAPI / Swagger file for the VM/PC API. 

Qualys API Endpoints 

Your API endpoint is determined by, and dependent on your account, explained here.

Qualys API Authentication 

Basic HTTP authentication (RFC 2617) and a form of session-based authentication are available. A custom header detailing the client type or user action must also be provided irrespective: 'X-Requested-With: xxxxxx'

Qualys API Limitations 

Qualys VM/PC API rate and concurrency limits per account type here. Your remaining API call limits (and related time windows) are reflected back in the headers of each API response.

Tooling 

cURL on the CLI or a more fully-featured environment like Postman (Postman collection v3.0 for Qualys). Later we will also look at using Tines for workflow automation.

Starting Simple with Curl 

Let’s start simply with cURL to perform 4 fundamental API actions. Let’s add an IP address as a host-based asset, list available host assets, request a scan, and then request the scan report. 

By using the command line -d switch we automatically issue a POST request with cURL. Also, note the 'Content-Type' used below:

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "X-Requested-With: Tines Demo" \
-d "ips=138.68.142.188&enable_vm=1" \
"https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/asset/ip/?action=add" \
| xq '.'

With this POST operation, we also need to ensure we’ve set enable_vm=1 to enable vulnerability management for the asset. The '|' pipe to xq is also an additional handy tool to return JSON from XML (XML being the default response type from Qualys). We then get the below response:

{
  "SIMPLE_RETURN": {
    "RESPONSE": {
      "DATETIME": "2020-09-29T14:39:28Z",
      "TEXT": "IPs successfully added to Vulnerability Management"
    }
  }
}

OK, let’s check if it really has been added as an asset:

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/json" \
-H "X-Requested-With: Tines Demo" \
"https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/asset/host/?action=list" \
| xq '.'

And the response:

{
  "HOST_LIST_OUTPUT": {
    "RESPONSE": {
      "DATETIME": "2020-09-29T15:15:24Z",
      "HOST_LIST": {
        "HOST": [
          {
            "ID": "13245001",
            "IP": "138.68.142.188",
            "TRACKING_METHOD": "IP",
            "DNS": "box.tines.xyz",
            "OS": "Linux 2.6"
          }
        ]
      }
    }
  }
}

Now let’s run a custom scan on this test host. We’ve also created two new scan types (over and above the default pre-loaded scan templates) called 'FastScan' and 'FullScan' which have the respective option_id's of 1181621 and 1181652. Let’s invoke our custom 'FullScan' and ensure it gets processed quickly by asking for it with a priority of 1.

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "X-Requested-With: Tines Demo"  \
-d "scan_title=Tines Demo Scan&ip=138.68.142.188&option_id=1181652&priority=1" \
"https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/scan/?action=launch" \
| xq '.'

And the response is:

{
  "SIMPLE_RETURN": {
    "RESPONSE": {
      "DATETIME": "2020-09-30T16:01:31Z",
      "TEXT": "New vm scan launched",
      "ITEM_LIST": {
        "ITEM": [
          {
            "KEY": "ID",
            "VALUE": "1988712"
          },
          {
            "KEY": "REFERENCE",
            "VALUE": "scan/1601481690.88712"
          }
        ]
      }
    }
  }
}

Ok, so far, so good. Now let’s ask for the scan results, and then subsequently, we’ll ask for a full PDF scan report. We’re not sure how long the initial scan takes or if the scan has been fully processed yet, but let’s ask for it and see what happens?

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "X-Requested-With: Tines Demo"  \
-d "scan_ref=scan/1601481690.88712&output_format=json" \
"https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/scan/?action=fetch" \
| jq '.'

This time, the response is not such a favorable one:

{
  "SIMPLE_RETURN": {
    "RESPONSE": {
      "DATETIME": "2020-09-30T16:02:15Z",
      "CODE": "15003",
      "TEXT": "Unable to fetch: No scan with ref scan/1601481690.88712 exists in state FINISHED,CANCELED,PAUSED,ERROR,NOHOSTALIVE,NOVULNSFOUND,INTERRUPTED."
    }
  }
}

So let’s wait a minute or so and then try again! Success, we now get a large array of scan data, including specific vulnerability results (a little too big to show all of them here):

    "result": "Some of the ports filtered by the firewall are: 20, 21, 22, 23, 111, 135, 445, 1, 7, 11.\n\nListed below are the ports filtered by the firewall.\nNo response has been received when any of these ports are probed.\n1-3,5,7,9,11,13,15,17-24,27,29,31,33,35,37-39,41-52,54-79,81-223,242-246,\n256-265,280-282,309,311,318,322-325,344-351,363,369-442,444-581,592-593,\n598,600,606-620,624,627,631,633-637,666-674,700,704-705,707,709-711,729-731,\n740-742,744,747-754,758-765,767,769-777,780-783,786,799-801,860,873,886-888,\n900-901,911,950,954-955,990-992,995-1001,1008,1010-1011,1015,1023-1100,\n1109-1112,1114,1123,1155,1167,1170,1207,1212,1214,1220-1222,1234-1236,\n1241,1243,1245,1248,1269,1313-1314,1337,1344-1625,1636-1774,1776-1815,\n1818-1824,1900-1909,1911-1920,1944-1951,1973,1981,1985-2028,2030,2032-2036,\n2038,2040-2049,2053,2065,2067,2080,2097,2100,2102-2107,2109, and more.\nWe have omitted from this list 702 higher ports to keep the report size manageable."

Finally, let’s ask for the creation of a nicely formatted PDF report (and just as before, we may have to poll the endpoint until the report is ready. However, we’ll skip that part for brevity and go on to show an example of tying the whole workflow together faster!).

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "X-Requested-With: Tines Demo"  \
-d "template_id=1781809&output_format=pdf&report_title=Tines Demo Report&report_type=Scan&ips=138.68.142.188&report_refs=scan/1601481690.88712" \
"https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/report/?action=launch" \
| xq '.'

Qualys' response:

{
  "SIMPLE_RETURN": {
    "RESPONSE": {
      "DATETIME": "2020-09-30T16:12:01Z",
      "TEXT": "New report launched",
      "ITEM_LIST": {
        "ITEM": {
          "KEY": "ID",
          "VALUE": "1419259"
        }
      }
    }
  }
}

Looking good, let’s get the report (assuming it’s ready!)...

curl -s -u "qualys_username:qualys_password" \
-H "Content-Type: application/x-www-form-urlencoded" \
-H "X-Requested-With: Tines Demo"  \
-d "id=1419259" "https://qualysapi.qg2.apps.qualys.eu/api/2.0/fo/report/?action=fetch" \
--output "Tines Demo Report.pdf"

… which results in the below fully-fledged 12 page PDF report (though we’re only showing pages 1 and 2 for now). This was a test Linux-based host running only a few services.

This is by no means automated or self-serve yet, so let’s see how we can simply orchestrate these steps into something a little smarter and smoother (without using any code) by using Tines.

Further and Faster with Tines 

Let’s evolve this use case a little further by creating a simple visual workflow in Tines. It will also benefit from the addition of a self-serve form on the front-end of the Story while allowing for both logic and flow control on the back-end. This will also email the scan report to the requester and also update a Jira case (as the chosen case management system, albeit ServiceNow or TheHive could also be easily used).

The blue ‘HTTP’ Action steps above mimic our curl steps (but are now simply dragged and dropped onto the Storyboard). They look like this (below) behind the scenes:

{
  "url": "https://{% global_resource qualys %}/api/2.0/fo/asset/ip/",
  "method": "post",
  "payload": {
    "action": "add",
    "ips": "{{ .receive_form_request.body.ip_address_or_multiple_addresses}}",
    "enable_vm": "1",
    "ud1": "Email: {{ .receive_form_request.body.requester_email}}, Code/Note: {{ .receive_form_request.body.project_wbs_code_or_note}}"
  },
  "basic_auth": [
    "{% global_resource qualys_username %}",
    "{% credential qualys_password %}"
  ],
  "headers": {
    "X-Requested-With": "Tines"
  }
}

We wire these building blocks together to create a repeatable and consistent Story workflow. At the top of the Story, we will receive the details from a form using the 'Webhook' Action. At the bottom, we will use a 'Send To Story' Action to make it modular and enable further chaining of this Story to others. In between, we handle simple extraction of any information we need, including looping with delays until Qualys has finished processing our scans and built our reports.

We can rapidly create a reusable form using some simple fields to capture the IP asset and project details:

Now let’s submit the above form to see what happens next! Qualys has been tasked to scan and report on a Windows server that’s, unfortunately, been misconfigured and is running a whole host of exposed and unneeded services. First, we receive an email with the scan report directly attached. This is a simple way for a Project Manager to be empowered to talk to their technical team about security-related issues without directly engaging the security team (yet!).

The first two pages are shown below but feel free to download the full report here, which includes all vulnerability and prescriptive remediation tasks.

Secondly, we update our case management system e.g. Jira, with the security report and related findings (something we’ve covered in detail in previous blogs). 

So, scan early, and scan often to empower your project teams to help themselves (while making your life easier too)! What fields would you add to a self-serve security form? What other repetitive tasks could you make 'self-serve' to help optimize and empower teams?

You can download the above automation and try it out (or modify it to your needs) using the free Community Edition of Tines.

‍

*Please note we recently updated our terminology. Our "agents" are now known as "Actions," but some visuals might not reflect this.*