GS cookie protection – effectiveness and limitations

The Microsoft C/C++ compiler supports the GS switch which aims to detect stack buffer overruns at runtime and terminate the process, thus in most cases preventing an attacker from gaining control of the vulnerable machine.  This post will not go into detail about how GS works, so it may be helpful to refer to these MSDN articles for an overview and loads of detail on how GS works and what a GS cookie is. It’s important to note that depending on the exact vulnerability, even if a function is protected by a GS cookie, a stack-based buffer overrun may still be exploitable – for example if the attacker can gain control prior to the cookie check. However even in these circumstances, GS can often be a significant obstacle to exploitation and/or reliability of exploitation. There have been some stack-based attacks recently that were not mitigated by GS – this post takes a couple of examples and looks at why that was.

This companion post looks at how Visual Studio 2010 improves GS in light of some of the limitations covered in this post.

MS08-067 – netapi32 vulnerability in parsing path name

In MS08-067 (see also associated SDL blog entry), there is a fixed size buffer on the stack and the vulnerability is in the code that parses the path for “\..” substrings, replacing them with the explicit directory name as required. For example \\server\A\B\C\..\D\..\..\E” would be resolved to “\\server\A\E”. The vulnerability lies in the fact that when searching backwards through the path string buffer for a ‘\’ character the pointer returned by this search can sometimes end up being before the start of the path buffer.

Let’s look at how this key fact impacts the effectiveness of GS when data is then copied into memory starting at that address. The call stack looks like:


The path string buffer is defined in CanonicalizePathNames and that function is GS-protected. The ConvertPathMacros function is not GS-protected: it takes a pointer to the path buffer defined in CanonicalizePathNames as one of its arguments and has itself no local variables that would lead to it being GS-protected.

The initial stack layout is as below, with the ConvertPathMacros function having a pointer to the pathBuffer variable defined in CanonicalizePathName:


Step 1: the search for ‘\’ character causes a pointer to reference address before the start of path buffer.


Step 2: attacker-controlled data is then written starting at that address, overwriting ConvertPathMacro’s stack frame data:


Step 3: ConvertPathMacro function returns, but its return address has been overwritten so that the attacker gains control. In the diagram above the GS cookie in CanonicalizePathName is not overwritten. Note that it is irrelevant whether the overflow overwrote the GS cookie in CanonicalizePathName or not: this is because the cookie in CanonicalizePathName’s stack frame is only checked when CanonicalizePathName returns. And the attacker gains control long before that, when ConvertPathMacros returns.

This is an example of a stack-based vulnerability and attack that GS is simply not designed to mitigate: GS will only protect against an overflow if:

1. The cookie is overwritten as part of the overflow.

2. The cookie check at function return is reached.

In this example neither of the two criteria above need apply.

MS07-017 – ANI file parsing vulnerability

By way of contrast, compare this with the ANI vulnerability in MS07-017 (see also associated SDL blog entry). The corresponding call stack was:



ReadChunk was effectively just copying data from the ANI file to a pointer provided by LoadAniIcon. The attacker could not control this pointer so exploiting MS07-017 could not be achieved by overwriting ReadChunk’s stack frame as in the previous case. However the LoadAniIcon function where the buffer vulnerable to overflow was defined was not GS-protected at all! So a traditional overflow targeting LoadAniIcon’s return address was feasible.


The ANIHEADER local variable overflowed in the ANI vulnerability was a pure data structure:

typedef struct _ANIHEADER {
DWORD cbSizeof;
DWORD cFrames;
DWORD cSteps;
DWORD cx, cy;
DWORD cBitCount, cPlanes;
DWORD jifRate;

The compiler uses a heuristic to decide which functions to GS-protect, and this is targeted mainly at protecting against string buffer overflows. As LoadAniIcon contained no such string buffers then it was not GS-protected.

Unlike the previous example then, the ANI vulnerability could in principle have been mitigated by GS, if GS were applied more extensively; eg if LoadAniIcon had been GS-protected then the picture would have looked like:


The overflow would have overwritten the GS cookie and when LoadAniIcon returned then the GS cookie check would have detected the overflow and terminated the process. As noted at the start, depending on the exact control flow in LoadAniIcon between the overflow and the cookie check at function exit it may still be possible to exploit this; however GS has removed the generic method of exploitation making any exploit harder to develop and (experience tells us) often far less reliable.

It turns out that there is a way of instructing the compiler to be more aggressive in what functions it GS-protects via a pragma: #pragma strict_gs_check (blogged about here by Mike Howard). In fact partly as a result of this ANI vulnerability, MSEC worked with product teams in Windows to apply the strict GS pragma to a number of parser components as part of Windows Vista Service Pack 1.


GS is designed to mitigate a specific class of stack-based attacks, making stack-based exploits harder to develop, less reliable, and in some cases reducing what would have been attacker code execution to a denial-of-service. For example at the time of writing, we do not know of an exploit for MS06-040 on Windows XP SP2+ or Windows Server 2003 SP1 platforms. The SDL requires all Microsoft products to be built with GS enabled. Many third-party products use GS too – including recent versions of Quicktime, Adobe Acrobat, and Flash for example.

Where the attacker has more fine-grained control of where to start the overflow, or the direction of an overflow/underflow then the GS mitigation won’t help. In some cases however GS would help if present, and its absence is purely related to the default heuristics that the compiler uses. For high-risk code – code that handles untrusted data for example – consider making use of the strict_gs_check pragma.

Looking ahead I’m also excited at the prospect of “Enhanced GS” – what’s “Enhanced GS”? Well, it’s a whole other article: check back here over the next couple of days and read all about it!

And of course there is no substitute for the code being secure in the first place!

– Tim Burrell, MSEC Security Science

Links to related articles

/GS (Buffer Security Check), MSDN Visual C++ compiler options entry

Compiler Security Checks in Depth, MSDN Visual Studio Technical Articles, Brandon Bray, February 2002

MS08-067 and the SDL, SDL blog entry, Michael Howard, October 2008

Lessons learned from the Animated Cursor Security Bug, SDL blog entry, Michael Howard , April 2007

Hardening stack-based buffer overrun detection in VC2005 SP1, Michael Howard’s blog, April 2007

#pragma strict_gs_check , MSDN C/C++ pre-processor reference.