MoreNOPs Feed

FreeBSD kernel debugging with GDB

2011-10-01T22:00:00Z

In today's post we will cover how to build a testing setup (to test kernel mode rootkit components for instance) in a FreeBSD environment. For this matter we'll build a client-server scheme using VMWare and it's serial port emulation capabilities as a bridge between the two machines. In this setup, one of the boxes will act as a debugger and the other one as the debuggee thanks to GDB's remote debugging feature. The exact versions covered in this article are the following:

FreeBSD 8.2-RELEASE (i386)
VMware Workstation 7.1.4 running under Ubuntu 11.04 (amd64)

First of all we need to grab a fresh copy of FreeBSD. Once we got our copy ISO file, we create a single new Virtual Machine (VM) with default FreeBSD settings (8GB of disk space + default disk layout should be more than enough for our setup). During the install process we should take care of selecting kernel developer packages as well as the ports system. The rest of the install process is fairly straightforward if we keep this guide at hand.

With a fresh system just ready for action, the next step will consist in installing rsync, which we'll use to move compiled source + binaries to the debugger on a later step. To install rsync one of the easiest methods is to go straight into the ports system and building it from there:

# cd /usr/ports/net/rsync  
# make install

And after a few seconds of output we will be back in a shell prompt with a fully functioning rsync deployment. Next, we will avoid ourselves the burden of having to install a second VM by making use of VMWare's VM cloning capability. In this case we will perform a full copy so that we get two separate disk images rather than a single linked copy. From now on, one of the VMs will work as a debuggee (VM A) and the other one as debugger (VM B).

In the next phase, inside VM A we will recompile the kernel such that we enable full debugging capabilities and tracing. For this, we will make a copy of the default kernel compilation configuration file and then add a couple of flags. After completing the kernel compilation and installing the new boot image, we will copy the generated debugging kernel + symbols to VM B. Let's begin:

# cd /usr/src/sys/i386/  
# cp conf/GENERIC conf/MYDEBUGKERNEL

With our favorite text editor we edit MYDEBUGKERNEL and add (note that some of the options might already be enabled and needn't be re-defined):

DEBUG -g  
OPTION KDB  
OPTION DDB  
OPTION GDB

And now, it's compile time:

# cd /usr/src/  
# make buildkernel KERNCONF=MYDEBUGKERNEL  
# make installkernel KERNCONF=MYDEBUGKERNEL

After compilation and installation succeed, we will copy the generated resources to VM B using rsync. Assuming that a folder is ready at /usr/debug in VM B (192.168.45.139 in this case), we pull the trigger:

# cd /usr/obj/usr/src/sys  
# rsync -avz MYDEBUGKERNEL root@192.168.45.139::/usr/debug

The transfer shouldn't take very long. After it's over, we shutdown both machines to let the VM magic begin. By magic, I mean that we will emulate a serial port connection by connecting both machines through a named pipe. Procedure is as follows:

In VM A's tab, right click -> Settings -> Add
Select "Serial Port" from the list then "output to socket"
In the next window, choose a filename for the file that will act as a named pipe (e.g. /tmp/com_1)
Select "From server" to "Virtual Machine"
That's it. Now repeat the previous steps with VM B but instead of "From Server" choose "From Client"

Now that both machines are ready and connected, we can start descending into the catacombs. Fire up both machines. Go to VM A and edit the file /boot/device.hints and change the first serial connection's flag parameter, which in my particular case is "hint.uart.0.flags" to the value "0xC0". Reboot VM A.

Now shift to VM B and navigate to the directory where we copied the symbols using rsync. From there launch GDB connected to the serial port (serial port device name might not match yours):

# cd /usr/debug/MYDEBUGKERNEL  
# kgdb -r /dev/cuau0 kernel.debug  
GNU gdb 6.1.1 [FreeBSD]  
Copyright 2004 Free Software Foundation, Inc.  
GDB is free software, covered by the GNU General Public License, and you are  
welcome to change it and/or distribute copies of it under certain conditions.  
Type "show copying" to see the conditions.  
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...    
switching to remote protocol  
ignoring packet error, continuing...  
Couldn't establish connection to remote target  

(kgdb)

Notice at this point how GDB enters a semi-idle state waiting for active connections from VM A. Now back to VM A, directly in the shell we issue:

# sysctl debug.kdb.enter=1  
debug.kdb.enter: 0KDB: enter: sysctl debug.kdb.enter  
[thread pid 1219 tid 100083]  
Stopped at  kdb_enter+0x3a: movl    $0,kdb_why  
db> gdb  
Step to enter the remote GDB backend.  
db>

Notice how now in VM B the debugging prompt changed to an active state. Once we type 's' to step in VM A, the debugger in VM B will break and take control of VM A (with full system control, yay!).

It's possible to confirm that we're actually debugging VM A by issuing a simple command such as 'bt', which will show the current executing thread's call stack:

(kgdb) bt  
#0  kdb_enter (why=0xc0c311a6 "sysctl",  
    msg=0xc0cd9a83 sysctl debug.kdb.enter)  
    at /usr/src/sys/kern/subr_kdb.c:354  
#1  0xc08eeaa in kdb_sysctl_enter (oidp=0xc0dbed00, arg1=0x0, arg2=0,  
    req=0xe6c89b74) at /usr/src/sys/kern/subr_kdb.c:174  
    (...output trimmed...)

Happy digging!

Disaster recovery plus plus

2011-03-20T23:00:00Z

Apparently a week or so ago my hosting service decided that wordpress wasn't much of his liking and without much explanation the website went down just minutes after I finished a new post. Annoyed by the fact that wordpress required both runtime generation and database backend, and after failing several times trying to put the service back up I decided to switch to this compile-to-static content format. No MySQL, no PHP, no nothing. Fortunately I made fresh SQL dump's from the database just the day before and after several tests I was able to recover all the posts in a XML file. Anyhow, welcome everyone to the new home and apologies for the downtime.

On Python, Ruby and dangling handlers

2011-03-07T23:00:00Z

Last time we saw how to build a fuzzing engine with Python and vtrace. Unfortunately the implementation I presented in that post generated certain issues once a decent amount of iterations were completed. Apparently Python doesn't like simultaneous thread's launching and finishing every 5 seconds. Analysis with Process Explorershowed that although past threads aren't used anymore the garbage collector fails coping with them. This ultimately lead to handle exhaustion (+10000 danglingthread+file handles) and subsequent fuzzer crash. A change in the approach failed as well. In this case the second tree architecture tested looked like so:

Main fuzz controller
- Iteration monitor
  - Debuggee

Making extensive use of the subprocessmodule in this case led to the same result, the only difference in this case is that process handles get leaked instead of thread handles. Somebody might shed more light on this but looks more like a Python garbage collector issue than anything else. Annoyed by this apparent fact, we move onto the next section. #### Fuzzing engine with Ruby and Ragweed This seemed like a nice chance to get a good grasp of Ruby and it's low-level foo capabilities. Looking for a already proven programmatic debugging choices are few; ragweedseems like the most mature decision. I'm not going to extend myself much on how to get a hand of the tool but I'll point a few steps:

Download the latest package from the FFI branch (0.2.0.pre2 right now).
Install any ruby release from the 1.8 branch.
Install dependencies (CMD.exe commands):

gem install rake

cd tduehr-ragweed/

gem build ragweed.gemspec

gem install ragweedXXXXX.gem

And that's about it. In this incarnation of the fuzzing engine I followed a slightly different approach to the one described in the beginning:

Main fuzz controller
- Debuggee
- Iteration Monitor

It's worth mentioning that both the debuggeeand the iteration monitor are launched as full-fledged processes. The code that accompanies this post is still fairly dirty and definitely not polished, but it works and does NOT leave dangling handlers messing around with OS resources. Also, note that the code is essentially a file fuzzer that lacks certain (albeit fundamental) features like sample logging and so on. This lacking features might make it as a future release Egurra, who knows... :-) To make the PoC code work you'll need to satisfy a couple more of dependencies, namely:

win32-process from Win32Utils.
sys-proctable from Sys Utils.

References:

ragweed on github.
ruby fuzzing engine.

Fuzzing engine with vtrace

2011-02-23T23:00:00Z

Ever since I first started building my first fuzz-like tools, I always felt that having a solid monitoring engine was a must. While developing Egurra, and motivated by the learnings from Gray Hat Python by Justin Seitz, I leaned pretty straightforwardly towards PyDbg. Unfortunately the component in charge of loading the debuggee in each iteration did so using the load() method. This entry point kept failing in an inconsistent fashion, as also confirmed by other users. Even the creator, Pedram Amini itself, acknowledged so in the official documentation (source):

To Do: This routines needs to be further tested ... I nomally just attach.

Hence, looking for other alternatives two names stuck out for me:

PyDbgEng: Python wrapper around the official windows debugging engine. Also, current underlying engine for the Peach fuzzer.
vtrace: VDB is a debugger written using the vtrace API built by Invisigoth from Kenshoto.

Althought PyDbgEng has the richest documentation of the two, I'll cover the latter one for the rest of this article because I got the most consistent results with it. vtrace comes with no documentation at all, aside from the source code itself. This inevitably forces us to take a look at the source in order to get some clues as to what is what and how to build anything useful out of it. Appart from the code, atlas's atlasutils came very handy for example reference. The main class that will support the meat of our debugging entity is the Trace class stored in the vtrace folder. Let's see it in action:

def fuzz(exe_path,format):  

    print "[*]"
    print "[*] Starting fuzzer"
    print "[*]"

    try:
        it = 0
        while 1:
            print "[*]"
            print "[*] running iteration %d" % it
            print "[*]"
            mutate(format)
            me = getTrace()
            handler = VerboseNotifier(format)
            me.registerNotifier(NOTIFY_SIGNAL,handler)
            thread = threading.Thread(target=proc,args=(me,exe_path,format))
            thread.setDaemon(0)
            thread.start()
            time.sleep(5)
            if me.isRunning():
                print "[*]"
                print "[*] killing process %d " % me.getPid()
                print "[*]"
                me.kill()
            time.sleep(1)
            it += 1

    except KeyboardInterrupt:
        print "[*]"
        print "[*] Closing fuzzing session, %d iterations completed." % it
        print "[*]"
        sys.exit(0)In this block we instantiate a Trace entity calling

getTrace() . In the next two lines, we register a handler for SIGNAL events inside the debugged process, which we'll cover deeply latter on. In this context, formatrefers to the file format currently being fuzzed. Apart from that, the rest of the code is standard Python threading code: Launch a debugging thread and wait for 5 seconds and then check if the process is still running (hence, no crash) and kill it if that's the case. In the following code, we will take a quick glance at the proc function that will launch the debugged app:

def proc(dbg,exe_path,format):
    dbg.execute(exe_path+" test.%s" % format)
    dbg.run()No voodoo here either, just load the executable and let it run.

Finally we will cover the most juicy aspect of all this process, the signaling and exception handling. For this matter, althought not the most elegant and most correct solution, I opted for tweaking directly some code in the vtrace libs. The following tweaks happened inside vtrace/notifiers.py:

class VerboseNotifier(Notifier):
    def __init__(self,format):
        self.format = format

    def notify(self, event, trace):
        dict = trace.getMeta("Win32Event")
        if dict['ExceptionCode'] == 0xC0000005L:
            print "[*] Access violation spotted!"
            stamp = time.localtime()

            fd = open("crashes\\crash-%s.log" % stamp, "w")
            fd.write("ESP: 0x%08x" % trace.getRegisterByName('esp'))
            fd.write("\n")
            fd.write("EIP: 0x%08x" % trace.getProgramCounter())
            fd.write("\n")
            fd.write(repr(trace.getMeta("Win32Event")))
            fd.close()

            shutil.copy("test.%s" % self.format, "crashes\\%s.%s" % (stamp,self.format))

            trace.kill()
        else:
            print "[*] First chance...passing over."
            trace.setMode("RunForever", True)

VerboseNotifier() is, as the name implies, a very informative type of Notifier. It simply spits out generic information regarding every event that happens inside the debugged process. In this particular case we are interested in catching ACCESS_VIOLATION exceptions only. Every time a event happens inside the debugged process, the notify() routine will be referenced with the parameters properly populated. All we need to do is check whether the event was ACCESS_VIOLATION exception or not. If this is the case, we will log both the crash and the sample file for latter analysis. Yet again, the formatparameter refers to the file format being fuzzed, pure convenience. Althought we focused on a particular use case here, vtrace contains A LOT of functionality and could definitely be used in a lot of different scenarios. Happy hunting! References:

atlas wandering
DEFCON 16: VulnCatcher: Fun with Vtrace and Programmatic Debugging (Youtube Video)
vtrace (GoogleCode mirror)
PyDbgEng

Internet Explorer clip:rect(0) Memory Corruption Vulnerability

2010-11-15T23:00:00Z

Today we're going to talk about a currently unpatched vulnerability in IE reportedly affecting versions 6/7/8. In this brief analysis we're going to focus on Internet Explorer version 7.0.5730.13 in Windows XP SP3. This bug tagged CVE-2010-3962, made it into the public because apparently it was found being actively exploited in the wild (SYMANTEC). Trigger code has been widely published in many sites, and for documentation purposes, it's reproduced here:


style
=
position
:absolute;clip:rect(0)>
There isn't much to say about it. It's seems that the
original
purpose of this properties (besides allowing system compromise :-), is to manipulate table borders in certain elements. But do we care about fancy HTML shapes here? not really... The essence of the bug lies in the implementation of certain methods inside
mshtml.dll,
the HTML rendering engine in Windows. First we need to look at
CTableLayoutBlock::EnsureTableDispNode
which allocates a new object:
.text:7E9A93E5                 push    esi
.text:7E9A93E6                 lea     eax, [ebx+0Ch]
.text:7E9A93E9                 push    eax
.text:7E9A93EA                 call    ?New@CDispContainer@@SGPAV1@PAVCDispClient@@K@Z ; CDispContainer::New(CDispClient *,ulong)
.text:7E9A93EF                 mov     esi, eax
.text:7E9A93F1                 test    esi, esi
.text:7E9A93F3                 jz      loc_7E9EA792
.text:7E9A93F9                 or      dword ptr [esi+4], 40000000h
.text:7E9A9400                 mov     ecx, [ebx+8]Next, this very object gets passed to
CDispNode::SetUserClip
inside the same
CTableLayoutBlock::EnsureTableDispNode
function:
.text:7E9EA691                 xor     eax, eax
.text:7E9EA693                 mov     [ebp+var_9C], eax
.text:7E9EA699                 mov     [ebp+var_A0], eax
.text:7E9EA69F                 mov     [ebp+var_A4], eax
.text:7E9EA6A5                 mov     [ebp+var_A8], eax
.text:7E9EA6AB                 lea     eax, [ebp+var_A8]
.text:7E9EA6B1                 push    eax
.text:7E9EA6B2                 mov     ecx, esi ; our object
.text:7E9EA6B4                 call    ?SetUserClip@CDispNode@@QAEXABUtagRECT@@@Z ; CDispNode::SetUserClip(tagRECT const &)And here comes the funny part. Inside
CDispNode::SetUserClip
the
vftable
address gets dereferenced and OR'ed with 0x1, corrupting the table as a result:
.text:7EA3FB95                 mov     eax, edi ; load object in eax
.text:7EA3FB97                 shl     ecx, 2
.text:7EA3FB9A                 sub     eax, ecx
.text:7EA3FB9C                 or      dword ptr [eax], 1 ; BUG!
.text:7EA3FB9F                 mov     eax, [edi+4]Once we have a corrupted entity, we need to actually call any method in the
vftable
because it will point to corrupted memory. This happens inside
CTableLayout::CalculateLayout:
.text:7E9A9604                 mov     ecx, [ebp+corruptObj_ptr]
.text:7E9A9607                 and     [ebp+var_8], 0
.text:7E9A960B                 lea     edx, [ebp+var_8]
.text:7E9A960E                 mov     [ebp+var_4], eax
.text:7E9A9611                 mov     eax, [ecx]      ; vtable[0]
.text:7E9A9613                 push    edx
.text:7E9A9614                 call    dword ptr [eax+44h] ; PWN!
.text:7E9A9617                 jmp     loc_7E8A3954As many other researchers pointed out this bug is a bit difficult to exploit reliably because it depends largely on mshtml.dll's version (7.0.5730.13 was used here). Heap Spray anyone?

Links:

  
    http://www.exploit-db.com/exploits/15421/
  
  
    http://www.microsoft.com/security/portal/Threat/Encyclopedia/Entry.aspx?Name=Exploit:Win32/CVE-2010-3962.A
  
  
    http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-3962
  

P.D: While I was writing this I noticed that the guys at
hdlsec.com
had a nice work relating to this very vulnerability. Unfortunately their site seems to be down, suggested read when it's running again.