Nathan Dautenhahn

Abstractions, Mechanisms, and Policies for Trustworthy System Design and Implementation

"All problems in computer science can be solved by another level of indirection."
-- David Wheeler

Indeed. Consequently, an exponential number of security gaps have arisen!

Computing systems contain vast amounts of sensitive information, but most systems have numerous layers of abstractions, making it challenging to guarantee high-level security properties. My research identifies fundamental abstraction gaps and closes them with cross-layer solutions. I am most interested in building trustworthy systems software, where a single exploit could compromise the whole system. My research in operating system security has applied two key approaches: 1) decompose systems in ways that enhance trustworthiness and 2) harden them to attack.


PhD Students

  • Bruce Chen -- Privilege Analysis and Browser Separation
  • Fangfei Yang -- Hardware-Assisted Program Safety
  • Bumjin Im -- Automated Privilege Separation using Commodity Hardware

Undergraduate Students

  • Lucas Atayde (Junior) -- Privilege Analysis and Runtime Behavior Monitoring
  • Tiffany Xiang (Freshman) -- Biosec DNA Sequence Detection


  • Lily Gray (UG) -- Python Privilege Analysis
  • Peiru Yang (UG) -- Nested Kernel Instruction Rewriting and Privilege Analysis


Are you interested in hacking?

I am interested in folks who like to build real systems, taming complexity to simplify and introduce solutions. Types of hacking you might be interested in? How about A little compiler hacking with LLVM (hopefully never GCC :-))? Maybe you want to explore FreeBSD, Linux or even build your own microkernel? Maybe you want to model security and quantify it so we can convert security from art into a science? Maybe you want to do security analysis? Maybe you like hardware/software boundaries and prototyping FGPAs? How about formal modeling of low-level abstractions to prove relevant high level properties?

What do we do?

In my group students are already doing all of these and we are excited to do more! I primarily have three research thrusts: 1) abstractions, mechanisms, and policies for nesting security monitors into larger monolithic environments, i.e., security kernel design and implementation, 2) developing a least-privilege security compiler that translates an application into a minimal low-level but rich representation for automated analysis, transformation, and code generation; and 3) hardware/software co-design for retrofitting memory safety into existing ecosystems.

Want to join?

Please get in touch with me if you are interested (Email, Twitter DM, etc.). Make sure to describe relevant experiences that would indicate your level of hacker skillz as well as describe what projects are of interest to you. If you don't connect the dots for me I likely won't see the connection. I'm looking for PhD students and will also consider adding a Postdoc.

Systems Security

Operating Systems

The foundation of virtual worlds, powerful, yet dangerous if exploited.


Abstractions at the right location to shift the culture of trust in overprivileged software.

Program Analysis

Static and dynamic analysis techniques to aid in policy derivation, transformation, and enforcement.


Hardware support for zero-cost safe execution.

Project Summaries

Secure Operating Systems

PerspicuOS (pdf) introduced the Nested Kernel architecture, and demonstrated how to virtualize the MMU at a single hardware privilege level without global memory safety or control-flow integrity. PerspicuOS combines read-only page table mappings with instruction capabilities. The base implementation mitigates code injection. Using the nested kernel we provided intra-kernel write-protection abstractions, and protected the system call vector table, a system call logging facility, and a tamper-evident process list data structure.

Nexen (pdf|analysis) deconstructed Xen into a Nested Kernel security monitor, a shared services domain, and per VM slices. The slice abstraction replicates code for each VM and sandboxes it. Nexen defends against 107 out of 144 Xen Security Advisories including denial-of-service attacks.

KCoFI (pdf) is the first kernel control-flow integrity system, demonstrating how to securely transition between threads during interrupts, signals, and bi-directionally between user space and system execution. KCoFI employs SVA memory protections to protect interrupt stacks.

Virtual Ghost (pdf) protects applications from untrusted operating systems by using ghost memory and controlled mappings, while enforcing control-flow integrity using KCoFI.

Automated Decomposition

μSCOPE. (in preparation) Systematizing Compartmentalization Opportunities for Privilege Encapsulation (μSCOPE) is a systematic methodology and framework for analyzing privileges within large scale code bases. The core idea is to synthesize decompositions directly from program behavior and measure the opportunity for least-privilege separation. We are currently applying it to Linux and JavaScript.

BreakApp (pdf) automatically decomposes large-scale third-party applications along module boundaries and provides abstractions for developers to specify cross-module policies.

Hardware-Assisted Safe Execution

NFP + XGPS (on request) The Nested Flow Path (NFP) is a system wide control-flow abstraction that traces control-flows between subroutine invocations and specifies how to traverse back on subsequent return-like operations for all transfers within a thread and across thread boundaries. eXecution Graph Path Security (XGPS) is an implementation of the NFP that modifies micorarchitecture to automatically construct, protect, and enforce the NFP for the full software stack, including the operating system.

Microstache (pdf) is a new in-address space abstraction along with associated microachitectural mechanisms. MicroStache replaces coarse-grained, slow, and complex isolation abstractions with the simple stache memory segment, which is accessed through a special purpose CPU extension that isolates it from normal load and store operations, as well as cache based micro-architectural side-channels. MicroStache combines with static compiler analysis to protect a large class of safely accessed data significantly impairing code-reuse attacks.

Deterministic Record and Replay

QuickRec (pdf) is a hardware/software system that records non-deterministic events and then injects them for deterministic replay. QuickRec is the first x86 based FPGA system to track multithreaded micro-operation interleaving.

PinCap (patent 9501340) is the first relaxed consistency memory model replayer of Total Store Order interactions from a real Intel x86 CPU. PinCap uses Pin to virtualize main memory, and emulates micro-ops with internal CPU register and cache state.

Secure Web Browsers

Cocktail (pdf) is a web browser that replicates each user input to three diverse browsers and does simple 2 out of 3 voting before returning the data to the client. This is a form of opportunistic N-version programming. We found non-determinism and browser event models to create challenges for replicating state.

Xan (pdf) automatically transforms several web application behaviors to employ new browser security features for protection. We found that in many cases secure variants could be deterministically applied without loss of functionality.


Doctoral Education Perspectives

What does it mean to obtain a doctorate? How can we do this systematically? Why is it that way? Will I know when I'm there? These are all core questions we analyze in the Doctoral Education Perspectives Seminar, characterizing the transition from dependent to independent scholar. I created this course while at UIUC to aid us in our journey. Please see the syllabus for a taste. I also created a panel on diverse and at-odds advising styles, which led to a great heated collegial debate.

Advanced Operating Systems

Operating Systems are a core element of computing. This syllabus was created as part of a course I took on research pedagogy, and included novel elements for peer reviewed grading and large systems design and implementation.

Operating Systems Seminar

As a teaching assistant for the OS seminar at UIUC I created a project where student proposals and final reports were shepherded by the instructors. I also devised a debate that gave students the opportunity to decide the best system organization: VMM, OS, or Microkernel? I hadn't created the Nested Kernel yet, otherwise it likely would have won.

Promoting Undergraduate Research in Engineering (PURE)

PURE is a program that pairs undergraduate freshman and sophomore students with research mentors. I mentored two students who worked on building a buddy allocator for PerspicuOS (Nested Kernel implementation in FreeBSD 9.0). Code.


Nested Kernel

Main project site.


FreeBSD 9.0 Implementation of the Nested Kernel.


Fine-grained object tracing in Linux.

KCoFI + VirtualGhost

Managed runtime for FreeBSD, enforces kernel control-flow integrity and sandboxes malicious operating systems.


Passive fingerprinting attack on Tor network.


  • [2021] Serving on IEEE S&P 2022 PC       
  • [2020] Serving on USENIX Security and IEEE S&P 2021 PCs
  • [2020] Serving on RAID and USENIX Security 2020 PCs
  • [SP 2020] Awarded NSF Grant for CNS Core: Small: eXecution Graph Path Security (XGPS)
  • [FA 2019] Serving on the USENIX Security PC
  • [FA 2019] Awarded Intel Gift for exploring Automated Privilege Separation
  • [FA 2019] Three PhD Student joining the group!
  • [SP 2019] Serving and serving on USENIX ATC PC
  • [FA 2018] Serving on USENIX Security and ICDCS PCs
  • [August 2018] Beginning assistant professorship @ Rice!
  • [May 2018] Invited to serve on the ASPLOS '18 ERC.
  • [May 2018] Paper "MicroStache" has been accepted at RAID '18!
  • [May 2018] Accepted Assistant Professor position at Rice University!
  • [October 2017] Create a Mac OSX High Sierra iso image for XNU hacking.
  • [October 2017] Paper "BreakApp: Automated, Flexible Application Compartmentalization" has been accepted at NDSS '18!
  • [October 2017] Invited to serve on PCs: Usenix Security Symposium '18 and International Conference on Distributed Computing Systems '18.
  • [July 2017] Invited talk with LinuxKit Sig
  • [July 2017] Released alpha version of Memorizer, a fine-grained memory access tracing tool for internal Linux objects github.
  • [June 2017] Our ONR grant (BAA N00014-17-S-B010) "REVOLVER: Recurrent EVOLution and Verification of Encapsulated Rights" has been awarded. The grant is in collaboration with Steve Zdancewic and JMS at UPenn.
  • [March 2017] Deconstructing Xen has been covered by several bloggers (the morning paper, Trevor Jim, Computer Science Blog) and featured on Hacker News ( thanks to Adrian Colyer with "the morning paper".
  • [January 2017] Apparently I've been Discovered: Article #46. Thanks to Stephen Ornes for the opportunity, it is truly an honor to comment.
  • [December 2016] Paper on "Deconstructing Xen" accepted at NDSS '17!
  • [December 2016] Served on Oakland Shadow PC
  • [November 2016] Workshop paper on DIRECT accepted at FEAST '16!
  • [August 2016] Officially graduated! Received my diploma from UIUC!
  • [May 2016] Postdoc at the department of Computer and Information Science, at the University of Pennsylvania working with Prof. Jonathan M. Smith
  • [May 2016] Successfully defended and deposited my thesis!  
  • [November, 17 2015] Nested Kernel Featured on Reddit: reddit OSDEV  
  • [June 2015] Latest work published at USENIX ATC '15: Slipstream  
  • [April 17, 2015] Nested Kernel Featured on Hacker News, and at the top for a second: hacker news  
  • [May 2015] Released the source code for the nested kernel:  
  • Let's Get In Touch!

    Ready to start the adventure? That's great! Give me a call or send me an email and I will get back to you as soon as possible!