Reanalyzing the Web for its Security Headers

By Corinne Smith –

In November of 2012 Veracode performed an experiment where they scanned Alexa’s top 1,000,000 sites for their security related headers. They looked for the headers X-Frame-Options, Strict-Transport-Security, X-Content-Security-Policy, and Access-Control-Allow-Origin. (Their full results can be found here)

VeraCode’s Results

In 2012 out of the one million websites there were 16,109 that had at least one of the four security related headers. Coming in at 12,812 unique sites X-Frame-Options headers were on the major of the websites that had security headers. Next most is Access-Control-Allow-Origin with 2,539 sites. There were 980 URLs with valid Strict-Transport-Security headers which are from the server telling the client to use HTTPS going forward. Veracode found a whopping 111 URL which used either X-Content-Security-Policy or X-Toolkit-CSP, 32 of which allow scripts or evals inline.

A Note on Content Security Policy

The Content Security Policy (CSP) version one was only introduced as a W3C Standard on 15th of November in 2012. However, there were experimental implementations in both Firefox (from Firefox 4) and Chrome (in 2011), using the headers X-Content-Security-Policy and X-Webkit-CSP respectively. When Veracode performed their analysis of CSP headers they were looking for the X-* headers which have been depreciated from the onset of the standard for the Content-Security-Policy header. This makes the results between Veracode and my own experiment as I searched for the header defined by the standard.

My Experiment

For my experiment, I wrote a simple python script to poll the top ten thousand of Alexa’s one million websites. I chose to do just the first ten thousand of the sites because of time, however the script given time would work for all one million. The script pulls the websites out of a csv file (which can be downloaded here), and leverages the python requests library to GET the main site over HTTP.

My Code

import requests

def main():

websites = open("websites.csv", "r")

results = open("results.csv", "a")

i = 0

redirectsneeded = []

for line in websites:

if i%100 == 0:

print i

result = []

i += 1

line = line.split()[1]

line = line.strip()

line = "http://" + line

try:

r = requests.get(line)

except:

redirectsneeded.append(line)

continue

result.append(line)

result.append(str(r.status_code))

if r.status_code == 200:

if 'x-Frame-Options' in r.headers:

result.append(r.headers['X-Frame-Options'])

else:

result.append("0")

if 'Content-Security-Policy' in r.headers:

result.append(r.headers['Content-Security-Policy'])

else:

result.append("0")

if 'Strict-Transport-Security' in r.headers:

result.append(r.headers['Strict-Transport-Security'])

else:

result.append("0")

string = ",".join(result) + "\n"

results.write(string)

string = ""

else:

redirectsneeded.append(line)

print redirectsneeded

websites.close()

results.close()

main()

My Results

X-Frame-Options

Out of the top 10,000 websites 2865 had some form of X-Frame-Options header. This means that approximately thirty percent of the top websites use headers for protection against click jacking. This value is greatly increased from the 12,812 out of 1,000,000 that Veracode reported in 2012. However not all of the responses contained valid values for the X-Frame-Options header. In the invalid responses, there were a few headers with GOFORIT as the value and others contained the value allow-all. Both of these values cause the header to fail and allow the resource to be framed.

Value Occurances
DENY 360
ALLOW-FROM 10
GOFORIT 3
ALLOW-ALL 33
SAMEORIGIN 2405

 

HSTS

There were a number of sites that use HSTS and have the Strict-Transport-Security header in their responses. Out of the 10,000 sites I scanned, 1,110 of them included the header in question. Of the sites that included the header, all of them specified a Max-age of some value.
290 of the headers included an IncludeSubdomain tag as well.

Value Occurances
MAX-AGE 1110
INCLUDESUBDOMAIN 58

 

CSP

Content Security Policy is a header that works to prevent cross site scripting and other code injection attacks on websites. Out of the ten thousand URLs scanned a disappointing total of 375 URLs had the Content-Security-Policy header. A number of the URLs allowed for either “unsafe-eval” or “unsafe-inline” for script-src. This number is still much larger when compared to the number Veracode came up with (111 out of 1,000,000). A few sites had Content-Security-Policy-Report-Only, so hopefully this number will continue to rise.

Conclusion

Overall the top sites on the web have increased their inclusion of security focused headers. This makes sense as four years has passed between when Veracode performed their analysis, and my own undertaking. In the past four year the web has evolved a lot and security in web is becoming a larger and larger issue so more companies are taking steps to combat attacks against their sites.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s