So what are my ideas on fixing the ASV process?
Modify The ASV Program
The conditions that drove the ASV process originally made sense. Vulnerability scanning tools were predominately open source and anyone could do scanning and just about anyone was doing vulnerability scanning. The results produced out of the open source tools could be highly questionable at best and the reporting was haphazard and about as trustworthy at times as a three dollar bill. Even in large organizations, the people doing the vulnerability scanning did not necessarily have networking, security or even IT backgrounds. Then there was a tremendously high false positive rate out of the open source tools. As a result, most organizations ignored the results they received because they found that they could not be trusted.
The purpose of the ASV program was to bring some sanity and professionalism to the vulnerability scanning process. MasterCard invented the ASV program (it was not called ASV then) back in 2005. A test network was built and prospective ASVs were required to run their vulnerability scanners against this network and produce results which were then reviewed by MasterCard. It was a much a test of the vulnerability scanning tool as it was of the person running the tool. When the program transitioned to the PCI SSC, the Council added a multiple choice test to the process, but the virtual network testing and report review is still part of the process.
The trouble with this process is that the vulnerability scanning tool is no longer the problem. Every ASV uses a commercial vulnerability scanning tool from either Tenable, Qualys, Saint, Tripwire or similar commercial tool vendor these days because they cannot afford to do otherwise. Since these tool vendors are also ASVs, requiring a vulnerability scan for ASV certification has become a truly pointless exercise. Other than the possibility of not properly entering the IP addresses to be scanned and running the wrong scanning policy, there really is very little that someone can screw up with a scanning tool.
The skill in vulnerability scanning today is reviewing the results, dealing with false positive results, working to address results with compensating controls and, with the Councils new edict on combining reports, working to get passing quarterly scans.
Therefore, in my opinion, training and testing of ASVs should be focused on the following.
- Determining the scope of vulnerability scanning.
- Vulnerability scanning methodology.
- Interpreting vulnerability scanning reports to confirm knowledge of the process and the meaning of the results.
- What constitutes a false positive result and how to document a false positive result.
- Development and documentation of an appropriate compensating control for a vulnerability.
- Process for how to produce an acceptable passing scanning report from multiple reports.
And let us not limit ASV certification to just independent consulting firms. As with the internal security assessor (ISA) program, open the ASV program to internal personnel as well. Most large companies have independent vulnerability scanning teams that are as capable to more than capable than their ASV brethren. There is no longer any reason that these internal people cannot do the ASV scans particularly if they meet the same standards and qualifications.
Approved Vulnerability Scanning Tools
I am not suggesting that the Council needs to develop a certification process for these tools as there are already plenty of sources that assess such tools.
The Council would publish a list based on the criteria developed by one or more independent tool assessment sources. This list would define those tools acceptable to use for ASV vulnerability scanning. The PCI DSS should then require that the QSA confirm that the vulnerability scanner used by the ASV is on the list in addition to confirming scope and the scanning policy used.
Require A Vulnerability Scanning Methodology
With the PCI DSS v3, the Council now requires penetration testers to use a documented and industry accepted penetration testing methodology. Yet, there is no such requirement for vulnerability testing.
Most vulnerability scanning is done using what I call the “toss it against the wall and see what sticks” approach. Basically, every possible vulnerability is run against every device. Most commercial vulnerability scanners interpret banners, signatures and other markers to trim the list of vulnerabilities to be tested based on what they believe the target to be. However, when you are scanning an external network blind, scanners cannot always properly interpret what an IP address resolves to as a device because of the mix of responses that they receive. As a result, scanners do not necessary trim tests increasing false positive results or they trim them too much and the test is not complete.
Then there is the automated nature of today’s vulnerability scanning. While I understand the desire to reduce costs of vulnerability scanning, the “point and click” nature of today’s ASV scanning has made it flawed. And it gets worse as organizations get passing scans. As a QSA, I cannot tell you how many passing scans I have reviewed where an organization could be hacked six ways to Sunday with the remaining vulnerabilities. As a security professional, it scares me to death. But as a QSA, while I can bring these up, they get no play because they do not have a CVSS of 4.0 or greater. You hope that these vulnerabilities get picked up in an organization’s penetration test.
But there is no guarantee of that happening because the penetration tester’s vulnerability scanner may or may not pick up the same vulnerabilities. As a result, part of the penetration testing methodology should include a review of all vulnerabilities found since the last penetration test and those should be tested for in the current penetration test to ensure they have been addressed.
Obviously, I have a preference to the methodology I discussed back in Part 2. But there are a number of methodologies posted out on the Internet from a variety of good sources. All I ask is that the vulnerability scanning methodology be integrated with the penetration testing methodology so that there are not gaps in coverage.
Require Monthly External Vulnerability Scanning
Before everyone panics, I am not asking that ASV scans be run monthly. Although if the ASV program is modified, for organizations with internal ASVs that is a possibility. I would still require the quarterly ASV scan, but I would add in monthly scans run by anyone deemed qualified as is allowed for internal vulnerability scans.
My primary rationale for this recommendation is driven by this simple fact. When the dominant solution vendor releases patches on the second Tuesday of every month and the vast majority of those fixes have a CVSS score of 4.0 or greater, anyone that thinks quarterly scanning keeps them secure is seriously kidding themselves. Not that a lot of security professionals bought into the quarterly vulnerability scanning requirement, even as a bare minimum. But without the standard requiring it, a QSA has no leg to stand on other than to intimidate and shame people into doing monthly scanning.
Even if you are not Microsoft centric in your external environment, with the breaches that have occurred and the revelations of Shellshock and Poodle, it is painfully obvious that the quarterly requirement is not going to keep organizations secure. I got a lot of calls after both of these vulnerabilities were announced with clients asking if their passing scans were no longer valid. I was a bit schizophrenic in my thoughts. On the one hand, I was glad they were at least thinking about the security implications of these vulnerabilities. But their concern about their passing scans just highlights the importance of meeting a PCI requirement and passing their PCI assessment versus being secure. Because, while I only got a few calls, you know that there are too many people that are congratulating themselves on dodging the bullets of Shellshock and Poodle because of the fortuitous timing of their quarterly scans and that they got an additional 30, 60 or even 90 days to address them.
Then there are those organizations that run solutions such as IBM’s Websphere or Oracle’s eCommerce suites. Both of these vendors not only patch their own application frameworks, but they also release those patches to the underlying operating systems that are compatible with their application frameworks. But worse, these vendors do not release monthly patch releases, they do patch releases on quarterly, semi-annual or even annual bases. As a result, there is a high likelihood that some operating system patches could be left out of these releases due to compatibility or timing issues. The work around is to mitigate any remaining vulnerabilities through additional logging, additional monitoring, changes in firewall rules, changes in IDS/IPS rules, etc. The additional vulnerability scanning could help organizations identify these issues and address them quicker than quarterly.
A side benefit of monthly scanning will be improving the ability of organizations and their QSAs to determine if an organization’s patching and mitigation processes are working according to requirement 6.1. Quarterly scans typically document a lot of vulnerabilities, mostly those under a CVSS of 4.0. As a result, whether or not an organization is properly managing their environment can be very difficult and time consuming leading to missing items that should be addressed. Having reports more often can facilitate getting these issues addressed sooner rather than later and keeping the volume lower and less daunting.
The bottom line in all of this is that monthly scanning is required to even have a chance at being secure these days. Yet the vast majority of organizations are only doing quarterly scans and thinking they are secure. That practice must change.
So there we have it. My thoughts on the ASV process and how I would go about fixing it.
