Nathan Dautenhahn

Abstractions, Mechanisms, and Policies for Trustworthy System Design and Implementation

"All problems in computer science can be solved by another level of indirection."
-- David Wheeler

Indeed. Consequently, an exponential number of security gaps have arisen!

Computing systems contain vast amounts of sensitive information, but most systems have numerous layers of abstractions, making it challenging to guarantee high-level security properties. My research identifies fundamental abstraction gaps and closes them with cross-layer solutions. I am most interested in building trustworthy systems software, where a single exploit could compromise the whole system. My research in operating system security has applied two key approaches: 1) decompose systems in ways that enhance trustworthiness and 2) harden them to attack.

Systems Security

Operating Systems

The foundation of virtual worlds, powerful, yet dangerous if exploited.


Abstractions at the right location to shift the culture of trust in overprivileged software.

Program Analysis

Static and dynamic analysis techniques to aid in policy derivation, transformation, and enforcement.


Hardware support for zero-cost safe execution.

Project Summaries

Secure Operating Systems

PerspicuOS (pdf) introduced the Nested Kernel architecture, and demonstrated how to virtualize the MMU at a single hardware privilege level without global memory safety or control-flow integrity. PerspicuOS combines read-only page table mappings with instruction capabilities. The base implementation mitigates code injection. Using the nested kernel we provided intra-kernel write-protection abstractions, and protected the system call vector table, a system call logging facility, and a tamper-evident process list data structure.

Nexen (pdf|analysis) deconstructed Xen into a Nested Kernel security monitor, a shared services domain, and per VM slices. The slice abstraction replicates code for each VM and sandboxes it. Nexen defends against 107 out of 144 Xen Security Advisories including denial-of-service attacks.

KCoFI (pdf) is the first kernel control-flow integrity system, demonstrating how to securely transition between threads during interrupts, signals, and bi-directionally between user space and system execution. KCoFI employs SVA memory protections to protect interrupt stacks.

Virtual Ghost (pdf) protects applications from untrusted operating systems by using ghost memory and controlled mappings, while enforcing control-flow integrity using KCoFI.

Automated Decomposition

μSCOPE. (in preparation) Systematizing Compartmentalization Opportunities for Privilege Encapsulation (μSCOPE) is a systematic methodology and framework for analyzing privileges within large scale code bases. The core idea is to synthesize decompositions directly from program behavior and measure the opportunity for least-privilege separation. We are currently applying it to Linux and JavaScript.

BreakApp (pdf) automatically decomposes large-scale third-party applications along module boundaries and provides abstractions for developers to specify cross-module policies.

Hardware-Assisted Safe Execution

NFP + XGPS (on request) The Nested Flow Path (NFP) is a system wide control-flow abstraction that traces control-flows between subroutine invocations and specifies how to traverse back on subsequent return-like operations for all transfers within a thread and across thread boundaries. eXecution Graph Path Security (XGPS) is an implementation of the NFP that modifies micorarchitecture to automatically construct, protect, and enforce the NFP for the full software stack, including the operating system.

Microstache (on request) is a new in-address space abstraction along with associated microachitectural mechanisms. MicroStache replaces coarse-grained, slow, and complex isolation abstractions with the simple stache memory segment, which is accessed through a special purpose CPU extension that isolates it from normal load and store operations, as well as cache based micro-architectural side-channels. MicroStache combines with static compiler analysis to protect a large class of safely accessed data significantly impairing code-reuse attacks.

Deterministic Record and Replay

QuickRec (pdf) is a hardware/software system that records non-deterministic events and then injects them for deterministic replay. QuickRec is the first x86 based FPGA system to track multithreaded micro-operation interleaving.

PinCap (patent 9501340) is the first relaxed consistency memory model replayer of Total Store Order interactions from a real Intel x86 CPU. PinCap uses Pin to virtualize main memory, and emulates micro-ops with internal CPU register and cache state.

Secure Web Browsers

Cocktail (pdf) is a web browser that replicates each user input to three diverse browsers and does simple 2 out of 3 voting before returning the data to the client. This is a form of opportunistic N-version programming. We found non-determinism and browser event models to create challenges for replicating state.

Xan (pdf) automatically transforms several web application behaviors to employ new browser security features for protection. We found that in many cases secure variants could be deterministically applied without loss of functionality.


Doctoral Education Perspectives

What does it mean to obtain a doctorate? How can we do this systematically? Why is it that way? Will I know when I'm there? These are all core questions we analyze in the Doctoral Education Perspectives Seminar, characterizing the transition from dependent to independent scholar. I created this course while at UIUC to aid us in our journey. Please see the syllabus for a taste. I also created a panel on diverse and at-odds advising styles, which led to a great heated collegial debate.

Advanced Operating Systems

Operating Systems are a core element of computing. This syllabus was created as part of a course I took on research pedagogy, and included novel elements for peer reviewed grading and large systems design and implementation.

Operating Systems Seminar

As a teaching assistant for the OS seminar at UIUC I created a project where student proposals and final reports were shepherded by the instructors. I also devised a debate that gave students the opportunity to decide the best system organization: VMM, OS, or Microkernel? I hadn't created the Nested Kernel yet, otherwise it likely would have won.

Promoting Undergraduate Research in Engineering (PURE)

PURE is a program that pairs undergraduate freshman and sophomore students with research mentors. I mentored two students who worked on building a buddy allocator for PerspicuOS (Nested Kernel implementation in FreeBSD 9.0). Code.


Nested Kernel

Main project site.


FreeBSD 9.0 Implementation of the Nested Kernel.


Fine-grained object tracing in Linux.

KCoFI + VirtualGhost

Managed runtime for FreeBSD, enforces kernel control-flow integrity and sandboxes malicious operating systems.


Passive fingerprinting attack on Tor network.

Let's Get In Touch!

Ready to start the adventure? That's great! Give me a call or send me an email and I will get back to you as soon as possible!