Analyzing Kernel Stack Crashes Related to Microsoft February 2011 MS11-011 (KB2393802) Patch

If you are having trouble with machines bugchecking with stop codes 7F or 1000007F trap code 8 after applying the February 2011 Microsoft patches, then this article might be relevant.

Often 0x7F or 0x1000007F stop codes trap code 8 are due to overflowing a fixed size (12000 byte) kernel stack space resource. The lion’s share of variations of crashes related to MS11-011 (KB2393802) seem to fall in this bucket.

While you can measure total kernel stack space using MemInfo (http://www.winsiderss.com/tools/meminfo/meminfo.htm), to diagnose the problem to the root cause you need to break kernel stack space usage down by module. Below is what we are seeing for kernel stack usage in sample crash dumps from affected machines. In this particular crash, Intel video driver related modules consumed 6,712 bytes of the fixed size 12,000 byte kernel stack space resource. We are finding that upgrading the Intel video driver is providing relief. YMMV.

Crashing systems examined so far have pre-2010 vintage igxpmp32.sys drivers installed. You can see the installed drivers versions by issuing a “lm t n” command in the kernel debugger with the crash dump file opened. Here is an example from a crashing system:

b1fb4000 b2187020 igxpmp32 igxpmp32.sys Fri Dec 18 12:59:39 2009 (4B2BC30B)

Upgrading to the latest Intel video driver (look to Lenovo, Dell, or whatever your hardware vendor is first for the latest tested version), seems to be helping in at least the cases we have examined.

Below is an example of how you can use the Microsoft kernel debugger’s knf command to dump out a stack backtrace with the kernel stack space per frame listed out. The number of bytes consumed per frame is in the second column in the output. Just take that list and add up each entry by OS component type to see who the offender(s) is/are. If you use Excel to do the math, you can use the HEX2DEC function to convert the hexadecimal values from the debugger output to decimal.

Example:
BugCheck 1000007F, {8, ba360d70, 0, 0}

*** WARNING: Unable to verify timestamp for igxpmp32.sys
*** ERROR: Module load completed but symbols could not be loaded for igxpmp32.s
s
Unable to load image igxpdx32.DLL, Win32 error 0n2
*** WARNING: Unable to verify timestamp for igxpdx32.DLL
*** ERROR: Module load completed but symbols could not be loaded for igxpdx32.D
L
Probably caused by : igxpmp32.sys ( igxpmp32+44184 )

Followup: MachineOwner
———

3: kd> knf
# Memory ChildEBP RetAddr
00 9bb12004 805362cb nt!ExpFindCurrentThread+0x8
01 24 9bb12028 8062c345 nt!ExAcquireResourceSharedLite+0x51
02 c 9bb12034 8063791f nt!CmpLockRegistry+0x27
03 38 9bb1206c 805bfe5b nt!CmpSecurityMethod+0x17
04 40 9bb120ac 805c01c8 nt!ObpGetObjectSecurity+0x99
05 30 9bb120dc 8062f28f nt!ObCheckObjectAccess+0x2c
06 4c 9bb12128 8062ff30 nt!CmpDoOpen+0x2d5
07 200 9bb12328 805bf488 nt!CmpParseKey+0x5a6
08 78 9bb123a0 805bba14 nt!ObpLookupObjectName+0x53c
09 54 9bb123f4 80625696 nt!ObOpenObjectByName+0xea
0a fc 9bb124f0 8054167c nt!NtOpenKey+0x1c8
0b 0 9bb124f0 80500699 nt!KiFastCallEntry+0xfc
0c 84 9bb12574 805e701e nt!ZwOpenKey+0x11
0d 270 9bb127e4 805e712a nt!RtlpGetRegistryHandleAndPath+0x27a
0e 48 9bb1282c 805e73e3 nt!RtlpQueryRegistryGetBlockPolicy+0x2e
0f 28 9bb12854 805e79eb nt!RtlpQueryRegistryDirect+0x4b
10 50 9bb128a4 805e7f10 nt!RtlpCallQueryRegistryRoutine+0x369
11 29c 9bb12b40 b1ff8184 nt!RtlQueryRegistryValues+0x482
WARNING: Stack unwind information not available. Following frames may be wrong.
12 a8 9bb12be8 b1fbd85b igxpmp32+0x44184
13 678 9bb13260 b1fb9a7b igxpmp32+0x985b
14 14 9bb13274 b2196729 igxpmp32+0x5a7b
15 c4 9bb13338 804ef19f VIDEOPRT!pVideoPortDispatch+0xabf
16 10 9bb13348 bf85e8c2 nt!IopfCallDriver+0x31
17 30 9bb13378 bf85e93c win32k!GreDeviceIoControl+0x93
18 24 9bb1339c bf376769 win32k!EngDeviceIoControl+0x1f
19 1288 9bb14624 bf3b9f19 igxpdx32+0x8769
1a 7c 9bb146a0 8054167c igxpdx32+0x4bf19
1b 0 9bb146a0 00000000 nt!KiFastCallEntry+0xfc

email: david @ sharpesecurity.com
website: www.sharpesecurity.com
Twitter: twitter.com/sharpesecurity
Twitter: twitter.com/patchmanagement

Share

7 Responses to “Analyzing Kernel Stack Crashes Related to Microsoft February 2011 MS11-011 (KB2393802) Patch”

  1. Little John said:

    Feb 22, 11 at 1:44 am

    Thanks for posting this. Can you provide the steps to gather the module list? I’d like to cross-check this against my machine with errors. Thanks.

  2. David Sharpe said:

    Feb 22, 11 at 4:45 am

    Do you mean how can you get to the point where you can successfully issue a “lm t n” command against a crash dump file to see the list of loaded drivers?

  3. Anonymous said:

    Mar 12, 11 at 12:31 am

    Not sure what Little John was looking for specifically, but if you could provide the switches/steps you used for MemInfo to break down the kernel stack space by module, I’d be greatly appreciative.

  4. David Sharpe said:

    Mar 12, 11 at 12:41 am

    I think MemInfo can only report the summary total usage figures. To get the breakdown you need to isolate the cause of a crash like this, I fell back to a kernel debugger (e.g. the free Microsoft kd was used in the article). Are you analyzing a crash dump file?

  5. Shaun (previously Anonymous because I simply forgot to type a name) said:

    Mar 12, 11 at 1:14 am

    The module we’re seeing in the dump is part of PGP Desktop. The information from PGP is that their footprint is at 500 or 600 depending on the version installed and that they’re generally the last driver loaded (potentially because of installation order). To quote a forum post, this leaves the finger pointing at them when other modules are using a considerable amount of the stack space. Knowing about Intel’s graphics driver has helped in most cases, but there are a select population of systems experiencing this issue that do not have an Intel graphics adapter. We need to pinpoint the stack space hog such that it can be addressed.

  6. David Sharpe said:

    Mar 12, 11 at 1:18 am

    If you have a small minidump file (the 64K-ish ones) from one of the affected machines, just email that to me at david –at– sharpesecurity.com if you want.

  7. Shaun said:

    Mar 12, 11 at 1:19 am

    I appear to have misread a section of the article. That or I need more coffee. This is what the knf command is used for. I’ll give that a go. Thanks for helping me work through my mental block.


Leave a Reply