Nathan Dautenhahn

Abstractions, Mechanisms, and Policies for Trustworthy System Design and Implementation

"All problems in computer science can be solved by another level of indirection."
-- David Wheeler

Indeed. Consequently, an exponential number of security gaps have arisen!

Computing systems contain vast amounts of sensitive information, but most systems have numerous layers of abstractions, making it challenging to guarantee high-level security properties. My research identifies fundamental abstraction gaps and closes them with cross-layer solutions. I am most interested in building trustworthy systems software, where a single exploit could compromise the whole system. My research in operating system security has applied two key approaches: 1) decompose systems in ways that enhance trustworthiness and 2) harden them to attack.


PhD Students

  • Ian Bridges -- Kernel Hardening and Memory Safety Analysis
  • Bruce Chen -- Privilege Analysis and Browser Separation
  • Fangfei Yang -- Hardware-Assisted Program Safety
  • Bumjin Im -- Automated Privilege Separation using Commodity Hardware
  • Yufeng Zhou -- Automated Privilege Separation and Nested Kernel Abstractions

Undergraduate Students

  • Lily Gray -- Python Privilege Analysis
  • Clara Tian -- Browser Privilege Analysis
  • Peiru Yang -- Nested Kernel Instruction Rewriting and Privilege Analysis


Are you interested in hacking?

I am interested in folks who like to build real systems, taming complexity to simplify and introduce solutions. Types of hacking you might be interested in? How about A little compiler hacking with LLVM (hopefully never GCC :-))? Maybe you want to explore FreeBSD, Linux or even build your own microkernel? Maybe you want to model security and quantify it so we can convert security from art into a science? Maybe you want to do security analysis? Maybe you like hardware/software boundaries and prototyping FGPAs? How about formal modeling of low-level abstractions to prove relevant high level properties?

What do we do?

In my group students are already doing all of these and we are excited to do more! I primarily have three research thrusts: 1) abstractions, mechanisms, and policies for nesting security monitors into larger monolithic environments, i.e., security kernel design and implementation, 2) developing a least-privilege security compiler that translates an application into a minimal low-level but rich representation for automated analysis, transformation, and code generation; and 3) hardware/software co-design for retrofitting memory safety into existing ecosystems.

Want to join?

Please get in touch with me if you are interested (Email, Twitter DM, etc.). Make sure to describe relevant experiences that would indicate your level of hacker skillz as well as describe what projects are of interest to you. If you don't connect the dots for me I likely won't see the connection. I'm looking for PhD students and will also consider adding a Postdoc.

Systems Security

Operating Systems

The foundation of virtual worlds, powerful, yet dangerous if exploited.


Abstractions at the right location to shift the culture of trust in overprivileged software.

Program Analysis

Static and dynamic analysis techniques to aid in policy derivation, transformation, and enforcement.


Hardware support for zero-cost safe execution.

Project Summaries

Secure Operating Systems

PerspicuOS (pdf) introduced the Nested Kernel architecture, and demonstrated how to virtualize the MMU at a single hardware privilege level without global memory safety or control-flow integrity. PerspicuOS combines read-only page table mappings with instruction capabilities. The base implementation mitigates code injection. Using the nested kernel we provided intra-kernel write-protection abstractions, and protected the system call vector table, a system call logging facility, and a tamper-evident process list data structure.

Nexen (pdf|analysis) deconstructed Xen into a Nested Kernel security monitor, a shared services domain, and per VM slices. The slice abstraction replicates code for each VM and sandboxes it. Nexen defends against 107 out of 144 Xen Security Advisories including denial-of-service attacks.

KCoFI (pdf) is the first kernel control-flow integrity system, demonstrating how to securely transition between threads during interrupts, signals, and bi-directionally between user space and system execution. KCoFI employs SVA memory protections to protect interrupt stacks.

Virtual Ghost (pdf) protects applications from untrusted operating systems by using ghost memory and controlled mappings, while enforcing control-flow integrity using KCoFI.

Automated Decomposition

μSCOPE. (in preparation) Systematizing Compartmentalization Opportunities for Privilege Encapsulation (μSCOPE) is a systematic methodology and framework for analyzing privileges within large scale code bases. The core idea is to synthesize decompositions directly from program behavior and measure the opportunity for least-privilege separation. We are currently applying it to Linux and JavaScript.

BreakApp (pdf) automatically decomposes large-scale third-party applications along module boundaries and provides abstractions for developers to specify cross-module policies.

Hardware-Assisted Safe Execution

NFP + XGPS (on request) The Nested Flow Path (NFP) is a system wide control-flow abstraction that traces control-flows between subroutine invocations and specifies how to traverse back on subsequent return-like operations for all transfers within a thread and across thread boundaries. eXecution Graph Path Security (XGPS) is an implementation of the NFP that modifies micorarchitecture to automatically construct, protect, and enforce the NFP for the full software stack, including the operating system.

Microstache (on request) is a new in-address space abstraction along with associated microachitectural mechanisms. MicroStache replaces coarse-grained, slow, and complex isolation abstractions with the simple stache memory segment, which is accessed through a special purpose CPU extension that isolates it from normal load and store operations, as well as cache based micro-architectural side-channels. MicroStache combines with static compiler analysis to protect a large class of safely accessed data significantly impairing code-reuse attacks.

Deterministic Record and Replay

QuickRec (pdf) is a hardware/software system that records non-deterministic events and then injects them for deterministic replay. QuickRec is the first x86 based FPGA system to track multithreaded micro-operation interleaving.

PinCap (patent 9501340) is the first relaxed consistency memory model replayer of Total Store Order interactions from a real Intel x86 CPU. PinCap uses Pin to virtualize main memory, and emulates micro-ops with internal CPU register and cache state.

Secure Web Browsers

Cocktail (pdf) is a web browser that replicates each user input to three diverse browsers and does simple 2 out of 3 voting before returning the data to the client. This is a form of opportunistic N-version programming. We found non-determinism and browser event models to create challenges for replicating state.

Xan (pdf) automatically transforms several web application behaviors to employ new browser security features for protection. We found that in many cases secure variants could be deterministically applied without loss of functionality.


Doctoral Education Perspectives

What does it mean to obtain a doctorate? How can we do this systematically? Why is it that way? Will I know when I'm there? These are all core questions we analyze in the Doctoral Education Perspectives Seminar, characterizing the transition from dependent to independent scholar. I created this course while at UIUC to aid us in our journey. Please see the syllabus for a taste. I also created a panel on diverse and at-odds advising styles, which led to a great heated collegial debate.

Advanced Operating Systems

Operating Systems are a core element of computing. This syllabus was created as part of a course I took on research pedagogy, and included novel elements for peer reviewed grading and large systems design and implementation.

Operating Systems Seminar

As a teaching assistant for the OS seminar at UIUC I created a project where student proposals and final reports were shepherded by the instructors. I also devised a debate that gave students the opportunity to decide the best system organization: VMM, OS, or Microkernel? I hadn't created the Nested Kernel yet, otherwise it likely would have won.

Promoting Undergraduate Research in Engineering (PURE)

PURE is a program that pairs undergraduate freshman and sophomore students with research mentors. I mentored two students who worked on building a buddy allocator for PerspicuOS (Nested Kernel implementation in FreeBSD 9.0). Code.


Nested Kernel

Main project site.


FreeBSD 9.0 Implementation of the Nested Kernel.


Fine-grained object tracing in Linux.

KCoFI + VirtualGhost

Managed runtime for FreeBSD, enforces kernel control-flow integrity and sandboxes malicious operating systems.


Passive fingerprinting attack on Tor network.

Let's Get In Touch!

Ready to start the adventure? That's great! Give me a call or send me an email and I will get back to you as soon as possible!