Nathan Dautenhahn


Founder and CEO, Serenitix
Research Scientist, Computer Science, Dartmouth College
Previously Assistant Professor of Computer Science, Rice University

Abstractions, Mechanisms, and Policies for Trustworthy System Design and Implementation


"All problems in computer science can be solved by another level of indirection."
-- David Wheeler

Indeed. Consequently, an exponential number of security gaps have arisen!


Computing systems contain vast amounts of sensitive information, but most systems have numerous layers of abstractions, making it challenging to guarantee high-level security properties. My research identifies fundamental abstraction gaps and closes them with cross-layer solutions. I am most interested in building trustworthy systems software, where a single exploit could compromise the whole system. My research in operating system security has applied two key approaches: 1) decompose systems in ways that enhance trustworthiness and 2) harden them to attack.

Students at Rice (2018–2023)


PhD Graduates

  • Yi (Bruce) Chien, Ph.D. 2025 -- Toward Secure Compartmentalization. Now at Microsoft
  • Fangfei Yang, Ph.D. 2024 -- Towards Fine-Grained Isolation Mechanisms. Now at Microsoft
  • Bumjin Im, Ph.D. 2021 -- Automated Privilege Separation using Commodity Hardware. Now at Amazon

Alumni

  • Yudi Yang (PhD) -- Program Analysis for Automated Compartmentalization
  • Weijie Huang (PhD) -- Subprocess Isolation in Node
  • Lucas Atayde (UG) -- Privilege Analysis and Runtime Behavior Monitoring
  • Tiffany Xiang (UG) -- Biosec DNA Sequence Detection
  • Lily Gray (UG) -- Python Privilege Analysis
  • Peiru Yang (UG) -- Nested Kernel Instruction Rewriting and Privilege Analysis

Systems Security

Operating Systems

The foundation of virtual worlds, powerful, yet dangerous if exploited.

Virtualization

Abstractions at the right location to shift the culture of trust in overprivileged software.

Program Analysis

Static and dynamic analysis techniques to aid in policy derivation, transformation, and enforcement.

Microarchitecture

Hardware support for zero-cost safe execution.

Project Summaries


Secure Operating Systems

PerspicuOS (pdf) introduced the Nested Kernel architecture, and demonstrated how to virtualize the MMU at a single hardware privilege level without global memory safety or control-flow integrity. PerspicuOS combines read-only page table mappings with instruction capabilities. The base implementation mitigates code injection. Using the nested kernel we provided intra-kernel write-protection abstractions, and protected the system call vector table, a system call logging facility, and a tamper-evident process list data structure.

Nexen (pdf|analysis) deconstructed Xen into a Nested Kernel security monitor, a shared services domain, and per VM slices. The slice abstraction replicates code for each VM and sandboxes it. Nexen defends against 107 out of 144 Xen Security Advisories including denial-of-service attacks.

KCoFI (pdf) is the first kernel control-flow integrity system, demonstrating how to securely transition between threads during interrupts, signals, and bi-directionally between user space and system execution. KCoFI employs SVA memory protections to protect interrupt stacks.

Virtual Ghost (pdf) protects applications from untrusted operating systems by using ghost memory and controlled mappings, while enforcing control-flow integrity using KCoFI.

Automated Decomposition

μSCOPE. (in preparation) Systematizing Compartmentalization Opportunities for Privilege Encapsulation (μSCOPE) is a systematic methodology and framework for analyzing privileges within large scale code bases. The core idea is to synthesize decompositions directly from program behavior and measure the opportunity for least-privilege separation. We are currently applying it to Linux and JavaScript.

BreakApp (pdf) automatically decomposes large-scale third-party applications along module boundaries and provides abstractions for developers to specify cross-module policies.

Hardware-Assisted Safe Execution

NFP + XGPS (on request) The Nested Flow Path (NFP) is a system wide control-flow abstraction that traces control-flows between subroutine invocations and specifies how to traverse back on subsequent return-like operations for all transfers within a thread and across thread boundaries. eXecution Graph Path Security (XGPS) is an implementation of the NFP that modifies micorarchitecture to automatically construct, protect, and enforce the NFP for the full software stack, including the operating system.

Microstache (pdf) is a new in-address space abstraction along with associated microachitectural mechanisms. MicroStache replaces coarse-grained, slow, and complex isolation abstractions with the simple stache memory segment, which is accessed through a special purpose CPU extension that isolates it from normal load and store operations, as well as cache based micro-architectural side-channels. MicroStache combines with static compiler analysis to protect a large class of safely accessed data significantly impairing code-reuse attacks.

Deterministic Record and Replay

QuickRec (pdf) is a hardware/software system that records non-deterministic events and then injects them for deterministic replay. QuickRec is the first x86 based FPGA system to track multithreaded micro-operation interleaving.

PinCap (patent 9501340) is the first relaxed consistency memory model replayer of Total Store Order interactions from a real Intel x86 CPU. PinCap uses Pin to virtualize main memory, and emulates micro-ops with internal CPU register and cache state.

Secure Web Browsers

Cocktail (pdf) is a web browser that replicates each user input to three diverse browsers and does simple 2 out of 3 voting before returning the data to the client. This is a form of opportunistic N-version programming. We found non-determinism and browser event models to create challenges for replicating state.

Xan (pdf) automatically transforms several web application behaviors to employ new browser security features for protection. We found that in many cases secure variants could be deterministically applied without loss of functionality.

Teaching


Doctoral Education Perspectives

What does it mean to obtain a doctorate? How can we do this systematically? Why is it that way? Will I know when I'm there? These are all core questions we analyze in the Doctoral Education Perspectives Seminar, characterizing the transition from dependent to independent scholar. I created this course while at UIUC to aid us in our journey. Please see the syllabus for a taste. I also created a panel on diverse and at-odds advising styles, which led to a great heated collegial debate.

Advanced Operating Systems

Operating Systems are a core element of computing. This syllabus was created as part of a course I took on research pedagogy, and included novel elements for peer reviewed grading and large systems design and implementation.

Operating Systems Seminar

As a teaching assistant for the OS seminar at UIUC I created a project where student proposals and final reports were shepherded by the instructors. I also devised a debate that gave students the opportunity to decide the best system organization: VMM, OS, or Microkernel? I hadn't created the Nested Kernel yet, otherwise it likely would have won.

Promoting Undergraduate Research in Engineering (PURE)

PURE is a program that pairs undergraduate freshman and sophomore students with research mentors. I mentored two students who worked on building a buddy allocator for PerspicuOS (Nested Kernel implementation in FreeBSD 9.0). Code.

Code

Nested Kernel

Main project site.

PerspicuOS

FreeBSD 9.0 Implementation of the Nested Kernel.

Memorizer

Fine-grained object tracing in Linux.

KCoFI + VirtualGhost

Managed runtime for FreeBSD, enforces kernel control-flow integrity and sandboxes malicious operating systems.

TorFA

Passive fingerprinting attack on Tor network.

News!


  • [2026] "Changes in the Guard: Detecting Authority-Exposing Patches with Differential Guarded Flow Paths" (with Sergey Bratus) accepted at SURE 2026, co-located with ACM CCS 2026 (pdf)
  • [2026] "Crypto-Ransomware and Their Defenses" published in ACM Computing Surveys (doi)
  • [2025] Co-taught the Linux rootkits component of Linux Malware Reverse Engineering at REcon 2025
  • [2025] "SoK: Software Compartmentalization" published at IEEE S&P 2025 (doi)
  • [2025] "Hi-Res: Precise Exploit Detection Using Object-Granular Memory Monitoring" published at IEEE SPW 2025 (pdf)
  • [2024] "Endokernel: A Thread Safe Monitor for Lightweight Subprocess Isolation" published at USENIX Security 2024 (pdf)
  • [2024] Founded Serenitix; joined the Dartmouth College Department of Computer Science as a Research Scientist
  • [2023] Joined Riverside Research as Principal Scientist, Secure and Resilient Systems
  • [2023] "Assessing the Impact of Interface Vulnerabilities in Compartmentalized Software" published at NDSS '23 (pdf)
  • [2023] "Whole-Program Privilege and Compartmentalization Analysis with the Object-Encapsulation Model" published at LangSec '23 (pdf)
  • [2022] Awarded NSF CAREER grant: Automatically Taming System Complexity with the Least-Authority Virtual Architecture (LAVA overview)
  • [2021] Serving on IEEE S&P 2022 PC       
  • [2020] Serving on USENIX Security and IEEE S&P 2021 PCs
  • [2020] Serving on RAID and USENIX Security 2020 PCs
  • [SP 2020] Awarded NSF Grant for CNS Core: Small: eXecution Graph Path Security (XGPS)
  • [FA 2019] Serving on the USENIX Security PC
  • [FA 2019] Awarded Intel Gift for exploring Automated Privilege Separation
  • [FA 2019] Three PhD Student joining the group!
  • [SP 2019] Serving and serving on USENIX ATC PC
  • [FA 2018] Serving on USENIX Security and ICDCS PCs
  • [August 2018] Beginning assistant professorship @ Rice!
  • [May 2018] Invited to serve on the ASPLOS '18 ERC.
  • [May 2018] Paper "MicroStache" has been accepted at RAID '18!
  • [May 2018] Accepted Assistant Professor position at Rice University!
  • [October 2017] Create a Mac OSX High Sierra iso image for XNU hacking.
  • [October 2017] Paper "BreakApp: Automated, Flexible Application Compartmentalization" has been accepted at NDSS '18!
  • [October 2017] Invited to serve on PCs: Usenix Security Symposium '18 and International Conference on Distributed Computing Systems '18.
  • [July 2017] Invited talk with LinuxKit Sig
  • [July 2017] Released alpha version of Memorizer, a fine-grained memory access tracing tool for internal Linux objects github.
  • [June 2017] Our ONR grant (BAA N00014-17-S-B010) "REVOLVER: Recurrent EVOLution and Verification of Encapsulated Rights" has been awarded. The grant is in collaboration with Steve Zdancewic and JMS at UPenn.
  • [March 2017] Deconstructing Xen has been covered by several bloggers (the morning paper, Trevor Jim, Computer Science Blog) and featured on Hacker News (https://goo.gl/C1b8Sf) thanks to Adrian Colyer with "the morning paper".
  • [January 2017] Apparently I've been Discovered: Article #46. Thanks to Stephen Ornes for the opportunity, it is truly an honor to comment.
  • [December 2016] Paper on "Deconstructing Xen" accepted at NDSS '17!
  • [December 2016] Served on Oakland Shadow PC
  • [November 2016] Workshop paper on DIRECT accepted at FEAST '16!
  • [August 2016] Officially graduated! Received my diploma from UIUC!
  • [May 2016] Postdoc at the department of Computer and Information Science, at the University of Pennsylvania working with Prof. Jonathan M. Smith
  • [May 2016] Successfully defended and deposited my thesis!  
  • [November, 17 2015] Nested Kernel Featured on Reddit: reddit OSDEV  
  • [June 2015] Latest work published at USENIX ATC '15: Slipstream  
  • [April 17, 2015] Nested Kernel Featured on Hacker News, and at the top for a second: hacker news  
  • [May 2015] Released the source code for the nested kernel: nestedkernel.org  
  • Let's Get In Touch!


    Ready to start the adventure? That's great! Give me a call or send me an email and I will get back to you as soon as possible!