Nathan Dautenhahn


Founder and CEO, Serenitix
Research Scientist, Computer Science, Dartmouth College
Previously Assistant Professor of Computer Science, Rice University

Abstractions, Mechanisms, and Policies for Trustworthy System Design and Implementation


"All problems in computer science can be solved by another level of indirection."
-- David Wheeler

Indeed. Consequently, an exponential number of security gaps have arisen!


Computing systems contain vast amounts of sensitive information, but most systems have numerous layers of abstractions, making it challenging to guarantee high-level security properties. My research identifies fundamental abstraction gaps and closes them with cross-layer solutions. I am most interested in building trustworthy systems software, where a single exploit could compromise the whole system. My research in operating system security has applied two key approaches: 1) decompose systems in ways that enhance trustworthiness and 2) harden them to attack.

Students at Rice (2018–2023)


PhD Graduates

  • Yi (Bruce) Chien, Ph.D. 2025 -- Toward Secure Compartmentalization. Now at Microsoft
  • Fangfei Yang, Ph.D. 2024 -- Towards Fine-Grained Isolation Mechanisms. Now at Microsoft
  • Bumjin Im, Ph.D. 2021 -- Automated Privilege Separation using Commodity Hardware. Now at Amazon

Alumni

  • Yudi Yang (PhD) -- Program Analysis for Automated Compartmentalization
  • Weijie Huang (PhD) -- Subprocess Isolation in Node
  • Lucas Atayde (UG) -- Privilege Analysis and Runtime Behavior Monitoring
  • Tiffany Xiang (UG) -- Biosec DNA Sequence Detection
  • Lily Gray (UG) -- Python Privilege Analysis
  • Peiru Yang (UG) -- Nested Kernel Instruction Rewriting and Privilege Analysis

Systems Security

Operating Systems

The foundation of virtual worlds, powerful, yet dangerous if exploited.

Virtualization

Abstractions at the right location to shift the culture of trust in overprivileged software.

Program Analysis

Static and dynamic analysis techniques to aid in policy derivation, transformation, and enforcement.

Microarchitecture

Hardware support for zero-cost safe execution.

Project Summaries


Secure Operating Systems

PerspicuOS (pdf) introduced the Nested Kernel architecture, and demonstrated how to virtualize the MMU at a single hardware privilege level without global memory safety or control-flow integrity. PerspicuOS combines read-only page table mappings with instruction capabilities. The base implementation mitigates code injection. Using the nested kernel we provided intra-kernel write-protection abstractions, and protected the system call vector table, a system call logging facility, and a tamper-evident process list data structure.

Nexen (pdf|analysis) deconstructed Xen into a Nested Kernel security monitor, a shared services domain, and per VM slices. The slice abstraction replicates code for each VM and sandboxes it. Nexen defends against 107 out of 144 Xen Security Advisories including denial-of-service attacks.

KCoFI (pdf) is the first kernel control-flow integrity system, demonstrating how to securely transition between threads during interrupts, signals, and bi-directionally between user space and system execution. KCoFI employs SVA memory protections to protect interrupt stacks.

Virtual Ghost (pdf) protects applications from untrusted operating systems by using ghost memory and controlled mappings, while enforcing control-flow integrity using KCoFI.

Endokernel (pdf) (USENIX Security 2024) is an intra-process security monitor that isolates memory at subprocess granularity and securely emulates system interfaces. An inside-out methodology identifies the operating-system primitives that allow a monitor to be bypassed and maps them back to the interfaces that depend on them, which exposed missing policies and guided a fine-grained locking design for multithreaded safety.

Endoprocess (pdf) (NSPW 2023) is a microkernel-based approach to protection within a process. It exports a low-level abstraction, the subprocess, overlaid on existing process interfaces such as mmap and mprotect, and adds extensibility through application-layer modules while staying compatible with existing OS abstractions.

Automated Decomposition

μSCOPE (RAID 2021) is a methodology and framework for analyzing least-privilege compartmentalization in large software artifacts. The core idea is to synthesize decompositions directly from program behavior and measure the opportunity for least-privilege separation.

BreakApp (pdf) automatically decomposes large-scale third-party applications along module boundaries and provides abstractions for developers to specify cross-module policies.

Object-Encapsulation Model (pdf) (LangSec 2023) translates a program into a Program Capability Graph that maps each operation to the code and data objects it may access, then groups them into encapsulated objects. The model quantifies privilege within a program automatically, so compartmentalization strategies can be measured and compared without deep knowledge of program internals.

CIVScope (pdf) (KISV 2023) builds on that measurement to model data flows through compartment interfaces and automatically measure the attack surface left by compartment-bypassing flows. Along diverse Linux kernel boundaries it finds 22,741 such paths across 395 of 4,394 driver interfaces, too many to correct by hand.

ConfFuzz (pdf|code) (NDSS 2023) studies Compartment Interface Vulnerabilities: when software is compartmentalized, a call that used to come from a trusted component can become an attack from a malicious one. ConfFuzz, an in-memory fuzzer for these interfaces, uncovered 629 vulnerabilities across 25 applications and 36 possible compartment APIs.

SoK: Software Compartmentalization (pdf) (IEEE S&P 2025) proposes a unified model for analyzing and comparing compartmentalization approaches, uses it to review 211 research efforts and 61 mainstream systems, and examines why compartmentalization is still not mainstream practice.

Exploit and Authority Detection

Changes in the Guard (pdf) (SURE 2026) introduces Guarded Flow Paths, in which every value flow carries the branch conditions under which it happens. Differential authority analysis runs one reachability query on both sides of a patch to name the check the patch removed and the authority it exposed.

Hi-Res (pdf) (IEEE SPW 2025) is a programmable detection framework that lifts high-level kernel exploit behaviors from low-level memory operations, building behavioral fingerprints from observed access patterns instead of hand-crafted monitors.

Crypto-Ransomware and Their Defenses (pdf) (ACM Computing Surveys 2026) characterizes ransomware behavior in depth and evaluates defenses by deployability as well as detection accuracy.

Hardware-Assisted Safe Execution

NFP + XGPS (on request) The Nested Flow Path (NFP) is a system wide control-flow abstraction that traces control-flows between subroutine invocations and specifies how to traverse back on subsequent return-like operations for all transfers within a thread and across thread boundaries. eXecution Graph Path Security (XGPS) is an implementation of the NFP that modifies micorarchitecture to automatically construct, protect, and enforce the NFP for the full software stack, including the operating system.

Microstache (pdf) is a new in-address space abstraction along with associated microachitectural mechanisms. MicroStache replaces coarse-grained, slow, and complex isolation abstractions with the simple stache memory segment, which is accessed through a special purpose CPU extension that isolates it from normal load and store operations, as well as cache based micro-architectural side-channels. MicroStache combines with static compiler analysis to protect a large class of safely accessed data significantly impairing code-reuse attacks.

Deterministic Record and Replay

QuickRec (pdf) is a hardware/software system that records non-deterministic events and then injects them for deterministic replay. QuickRec is the first x86 based FPGA system to track multithreaded micro-operation interleaving.

PinCap (patent 9501340) is the first relaxed consistency memory model replayer of Total Store Order interactions from a real Intel x86 CPU. PinCap uses Pin to virtualize main memory, and emulates micro-ops with internal CPU register and cache state.

Secure Web Browsers

Cocktail (pdf) is a web browser that replicates each user input to three diverse browsers and does simple 2 out of 3 voting before returning the data to the client. This is a form of opportunistic N-version programming. We found non-determinism and browser event models to create challenges for replicating state.

Xan (pdf) automatically transforms several web application behaviors to employ new browser security features for protection. We found that in many cases secure variants could be deterministically applied without loss of functionality.

Teaching


Doctoral Education Perspectives

What does it mean to obtain a doctorate? How can we do this systematically? Why is it that way? Will I know when I'm there? These are all core questions we analyze in the Doctoral Education Perspectives Seminar, characterizing the transition from dependent to independent scholar. I created this course while at UIUC to aid us in our journey. Please see the syllabus for a taste. I also created a panel on diverse and at-odds advising styles, which led to a great heated collegial debate.

Advanced Operating Systems

Operating Systems are a core element of computing. This syllabus was created as part of a course I took on research pedagogy, and included novel elements for peer reviewed grading and large systems design and implementation.

Operating Systems Seminar

As a teaching assistant for the OS seminar at UIUC I created a project where student proposals and final reports were shepherded by the instructors. I also devised a debate that gave students the opportunity to decide the best system organization: VMM, OS, or Microkernel? I hadn't created the Nested Kernel yet, otherwise it likely would have won.

Promoting Undergraduate Research in Engineering (PURE)

PURE is a program that pairs undergraduate freshman and sophomore students with research mentors. I mentored two students who worked on building a buddy allocator for PerspicuOS (Nested Kernel implementation in FreeBSD 9.0). Code.

Code

Nested Kernel

Main project site.

PerspicuOS

FreeBSD 9.0 Implementation of the Nested Kernel.

Memorizer

Fine-grained object tracing in Linux.

KCoFI + VirtualGhost

Managed runtime for FreeBSD, enforces kernel control-flow integrity and sandboxes malicious operating systems.

TorFA

Passive fingerprinting attack on Tor network.

ConfFuzz

Fuzzing for compartment interface vulnerabilities.

News!


  • [2026] "Changes in the Guard: Detecting Authority-Exposing Patches with Differential Guarded Flow Paths" (with Sergey Bratus) accepted at SURE 2026, co-located with ACM CCS 2026 (pdf)
  • [2026] "Crypto-Ransomware and Their Defenses" published in ACM Computing Surveys (pdf)
  • [2025] Co-taught the Linux rootkits component of Linux Malware Reverse Engineering at REcon 2025
  • [2025] "SoK: Software Compartmentalization" published at IEEE S&P 2025 (pdf)
  • [2025] "Hi-Res: Precise Exploit Detection Using Object-Granular Memory Monitoring" published at IEEE SPW 2025 (pdf)
  • [2024] "Endokernel: A Thread Safe Monitor for Lightweight Subprocess Isolation" published at USENIX Security 2024 (pdf)
  • [2024] Founded Serenitix; joined the Dartmouth College Department of Computer Science as a Research Scientist
  • [2023] Joined Riverside Research as Principal Scientist, Secure and Resilient Systems
  • [2023] "Assessing the Impact of Interface Vulnerabilities in Compartmentalized Software" published at NDSS '23 (pdf)
  • [2023] "Whole-Program Privilege and Compartmentalization Analysis with the Object-Encapsulation Model" published at LangSec '23 (pdf)
  • [2022] Awarded NSF CAREER grant: Automatically Taming System Complexity with the Least-Authority Virtual Architecture (LAVA overview)
  • [2021] Serving on IEEE S&P 2022 PC       
  • [2020] Serving on USENIX Security and IEEE S&P 2021 PCs
  • [2020] Serving on RAID and USENIX Security 2020 PCs
  • [SP 2020] Awarded NSF Grant for CNS Core: Small: eXecution Graph Path Security (XGPS)
  • [FA 2019] Serving on the USENIX Security PC
  • [FA 2019] Awarded Intel Gift for exploring Automated Privilege Separation
  • [FA 2019] Three PhD Student joining the group!
  • [SP 2019] Serving and serving on USENIX ATC PC
  • [FA 2018] Serving on USENIX Security and ICDCS PCs
  • [August 2018] Beginning assistant professorship @ Rice!
  • [May 2018] Invited to serve on the ASPLOS '18 ERC.
  • [May 2018] Paper "MicroStache" has been accepted at RAID '18!
  • [May 2018] Accepted Assistant Professor position at Rice University!
  • [October 2017] Create a Mac OSX High Sierra iso image for XNU hacking.
  • [October 2017] Paper "BreakApp: Automated, Flexible Application Compartmentalization" has been accepted at NDSS '18!
  • [October 2017] Invited to serve on PCs: Usenix Security Symposium '18 and International Conference on Distributed Computing Systems '18.
  • [July 2017] Invited talk with LinuxKit Sig
  • [July 2017] Released alpha version of Memorizer, a fine-grained memory access tracing tool for internal Linux objects github.
  • [June 2017] Our ONR grant (BAA N00014-17-S-B010) "REVOLVER: Recurrent EVOLution and Verification of Encapsulated Rights" has been awarded. The grant is in collaboration with Steve Zdancewic and JMS at UPenn.
  • [March 2017] Deconstructing Xen has been covered by several bloggers (the morning paper, Trevor Jim, Computer Science Blog) and featured on Hacker News (https://goo.gl/C1b8Sf) thanks to Adrian Colyer with "the morning paper".
  • [January 2017] Apparently I've been Discovered: Article #46. Thanks to Stephen Ornes for the opportunity, it is truly an honor to comment.
  • [December 2016] Paper on "Deconstructing Xen" accepted at NDSS '17!
  • [December 2016] Served on Oakland Shadow PC
  • [November 2016] Workshop paper on DIRECT accepted at FEAST '16!
  • [August 2016] Officially graduated! Received my diploma from UIUC!
  • [May 2016] Postdoc at the department of Computer and Information Science, at the University of Pennsylvania working with Prof. Jonathan M. Smith
  • [May 2016] Successfully defended and deposited my thesis!  
  • [November, 17 2015] Nested Kernel Featured on Reddit: reddit OSDEV  
  • [June 2015] Latest work published at USENIX ATC '15: Slipstream  
  • [April 17, 2015] Nested Kernel Featured on Hacker News, and at the top for a second: hacker news  
  • [May 2015] Released the source code for the nested kernel: nestedkernel.org  
  • Let's Get In Touch!


    Ready to start the adventure? That's great! Give me a call or send me an email and I will get back to you as soon as possible!