x86-64 OS Internals

Dojo Trailer

Course Schedule

May 14 to 15 or May 16 to 17.

Course Abstract

This class teaches you about the fundamental hardware mechanisms which all operating systems, virtualization systems, and firmware *must* interact with in order to run successfully on x86 hardware. For instance it teaches you what an OS, virtualization, or firmware must do to separate user code (ring 3) from kernel code (ring 0). Or how they can enforce non-executable memory to make vulnerability exploitation more difficult. This is taught in a *mostly* OS-agnostic way focusing on Intel-isms rather than OS-isms (albeit with using Windows as reinforcement, thanks to its excellent kernel-level debugging support.) This class also teaches you to be comfortable with Reading The Fun Manual (RTFM!) to give you self-sufficiency when seeking out the most accurate details of how things work.

Course Pre-requisites

You should have equivalent knowledge of x86-64 assembly, architecture, and specifically WinDbg as that provided in the x86-64 Assembly class, also offered at CanSecWest. If you don't have that background, or if you just need a refresher, you can sign up for the x86-64 All You Can Learn Buffet class to go through that material before proceeding to this class's material. And then you can even further proceed to the bundled  x86-64 Intel Firmware Attack & Defense class, if you complete all the material from this class.

Course Learning Objectives

• Understand how ring 0 (kernel) / ring 3 (userspace) privilege separation *really* works.

• Understanding how to use CPUID to query the features available on your system, and how to read the Model Specific Registers (MSRs) to check which ones your OS has actually enabled.

• Understand segmentation (and how it relates to privilege separation).

• Understand interrupts (and how they pertain to system calls and debugging).

• Understand system calls (and how they constitute a major attack surface of an OS).

• Understand virtual memory and page table setup (and how it interacts with security mechanisms like XD, SMAP, and SMEP).

• Understand how software and hardware breakpoints work.

• Understand how port IO allows communication to legacy peripherals and VMWare's control channel.

• Being comfortable with Reading The Fun Manual (RTFM!) to go seek out the most accurate details of how things work.

One-of-a-kind Class Format!

This class is run a little different from most classes. We provide you purpose-built recorded lectures instead of trapping you in realtime with live-lectures. But fear not, the instructor is always right there eagerly waiting to mingle with the students and answer any questions you have. (The instructor really likes being asked questions. It shows you're paying attention ;)). One of many benefits is that you can watch lectures at 2x speed and zoom ahead of the other students and get to the hands on labs quicker. Or if there's bits of material you already know, you can just skip them and move on to the bits you don't know! Another big benefit is that you get to take the full lectures and labs with you! That means if you forget stuff and then need it in 6 months, you can quickly re-bootstrap yourself! Or you can watch the class twice, to really grow those neural connections and cement it in your brain! And unlike live lectures, our lectures are always getting more factually accurate, by having any accidental errors edited out. 

Because we give you all the lecture and lab materials and videos after class, what you're really paying for is support from the instructor! So you'll be entitled to keep asking up to 20 questions after class, with 1-2 hour turnaround answers (after accounting for time-zone differences.) This lets you keep productively working through the material if you run out of time at the conference. If you'd like to learn more about the benefits of this style of class delivery, please read this blog post.

Course Agenda

• Introduction

• CPUID

  • CPU Feature Identification instruction

• Processor Execution Modes

  • Real mode, protected mode, IA-32e mode, system management mode

• Model Specific Registers (MSRs)

  •  Reading and writing MSRs (rdmsr, wrmsr)

• Privilege Rings & Segmentation

  •  Privilege rings start

  •  Segment selectors & segment registers

  •  Global Descriptor Table (GDT) & Local Descriptor Table (LDT)

  •  Segment descriptors

  •  Privilege rings finish

  •  Call gates

  •  Return to RTFM

  •  Implicit and explicit use of segmentation

• Interrupts

  • Interrupts vs. Exceptions

  • Tasks and the Task State Segment (TSS)

  • Interrupt Descriptor Table (IDT)

  • Interrupt Descriptors

  • Interrupt masking

  • Red Pill and virtualization detection

• System Calls

  • System Call instructions (syscall/sysret, sysenter/sysexit)

  • Syscall-adjacent techniques & instructions (swapgs, {rd,wr}{fs,gs}base)

• Read the Time Stamp Counter (RDTSC)

• Paging and Virtual Memory

  • Introduction

  • Paging and the Control Registers

  • Page Tables

    • MAXPHYADDR

    • 32 bit linear to 32 bit physical, 4KB pages

    • 32 bit linear to 32 bit physical, 4MB pages

    • 32 bit linear to 40 bit physical, Physical Address Extensions (PAE)

    • 48 bit linear to 52 bit physical, 4-level paging, 4KB, 2MB, 1GB pages

    • (Optional) 57 bit linear to 52 bit physical, 5-level paging

  • Page Table Entries

    •  CR3

    •  PML4E

    •  Exploit Mitigation Aside: XD, SMEP, SMAP

    •  PDPTE

    •  PDE

    •  PTE

  •  Canonical addresses

  •  Page faults

  •  Translation Lookaside Buffer & Shadow Walker rootkit

  •  Non-executable Memory (NX/XD bit)

• Interrupts & Debugging

  • Software breakpoints

  • Hardware breakpoints

  • Normal break on execute, write, read/write, port IO

  • Break on mov to debug registers

  • Trap Flag (TF)

  • Resume Flag (RF) and single step exceptions

• Port IO

  • In/out instruction and accessing VMWare "backdoor" IO port

• Conclusion

Hardware Requirements

•  A PC or an *x86* Mac (class won't work with an M1 Mac!) capable of running 1 VM at a time with ideally 4 GB of dedicated RAM.

•  Headphones for watching videos, (preferably over-ear so you're not disturbed as the instructor is walking around the class answering individuals' questions).

Software Requirements

•  Administrator privileges to install virtualization software on your machine.

•  A PC with VMWare Workstation or an *x86* Mac with VMWare Fusion (the free "Player" versions are fine).

•  ISO for installing 2 instances of Windows 10 x86-64 (30 day trial version is fine).

•  A link to a software setup guide will be sent before class, and the student should install before class to maximize time available for interaction with the instructor. (Other software includes Visual Studio 2019, the Windows Software Development Kit (SDK), the Windows Driver Development Kit (WDK), and WinDbg.)