What am I up to?
Long-time readers of this blog are probably aware that updates have been rare in the past few years, although I do try to keep time for some interesting articles from time to time. Most of my public research lately has been done through the Infosec Conference Circuit, so if you were not already aware, you can download slides from all my talks at the following URL:
http://www.alex-ionescu.com/publications
Additionally, I have a number of presentations I am looking forward to giving this year, including:
- Windows Container Internals (all about the technologies powering Centennial, Windows Defender Application Guard, Hyper-V Containers, Docker, and more — including the security model and bousyndaries of one of these) — SyScan 360 in Seattle (May)
- Surface Aggregator Module (SAM) Internals — a little chip on your Surface Pro/Laptop/Book that you probably didn’t know was there. If you liked my past talks on the Apple SMC, you’ll enjoy this as well — Recon in Montreal (June)
I also have a number of interesting design flaws I discovered this year in various Windows components — as these get patched (they are not Tavis-worthy wormable RCEs, not to worry), I have been mulling over a “Windows Design Flaw Garage Sale” talk similar to the famous one that Stefan Esser (i0nic) gave a few years ago about Apple/iOS — covering some past bugs (fixed and unfixed) and more recent ones.
However, this post is not about such small research updates — but rather about a much bigger piece of work that has taken up my time these last 12 months — the release of Windows Internals, 7th Edition (Part 1)!
Some history…
After the release of the 6th Edition of the book, which covered Windows 7, it’s fair to say that I was pretty burned out. The book incurred heavy delays due to my juggling of college, internships, and various relationships, while also requiring a massive amount of work due to the ambitious new sections, and coverage of the many, many changes that Windows 7 brought to the table (either fine-tuning many small things from Vista, or completely new kernel modules). Additionally, my co-authors also had new plans: David Solomon went on to retire and sunset his training business (David Solomon Expert Seminars), and Mark Russinovich was fully committed to his new role at Microsoft which eventually took him to Azure, where he is now the Chief Technology Officer (CTO), and kicking some major cloud/fabric butt with his extensive OS experience and security background. All of this to say — there was not much of an appetite to immediately begin writing a new book, with Windows 8 looming on the horizon (at that point still called Windows Blue).
Something else happened at that time: under leadership from Satya Nadella, Microsoft began delivering on its “Windows as a Service (WaaS)” model, furiously releasing a Windows 8.1 Update within a year of Windows 8 having shipped. Given that a single OS update had taken us years to cover, this release cycle was simply too rapid to successfully think about releasing a book in a timely fashion. I stopped thinking that a new edition of the book would ever be released, and I certainly didn’t think I’d be able to do one.
All gaps create opportunities, and two other authors decided that they could take on the 7th Edition and ship a successful update. They re-arranged the book in three parts, instead of two, with the first one focusing on Windows 8 User-Mode Metro (now UWP) Application Development, the second one on the Kernel, and the third one on Driver Development. I was not contacted or involved in these changes, and honestly, was not too happy about them. There are excellent driver programming books, just as there are application development books (even on Metro/UWP). This felt, to me, like an attempt to significantly cut down on the kernel portions of the book, and monetize on the Metro/Driver programming books, which obviously have a much wider audience.
Additionally, with Windows 8 having shipped, Part 1 was slated for that year, with Part 2 (Windows 8.1 would now be out) the year after, and finally, Part 3, a year after that (Windows 10 would now be in beta). By the time you’d get to the last part, the OS would’ve already moved two releases further — or, each part could cover that OS. Becoming a Windows 8 Metro App Development book, with Windows 8.1 Kernel Internals book, and Windows 10 Driver Development book. These were just my personal thoughts at the time — which I kept to myself, because every author needs a chance to be successful, and others may well have liked this model, and the book may have sold more copies than all previous combined – who was I to judge?
One year passed. Then another… then another. By now, given that my name was still on the cover — regardless of my lack of involvement — many people would come to me and ask me “What’s going on? Why are you taking so long? Do you need help?” on the friendly side… and of course, some not-so-friendly comments, from people that had pre-ordered on Day 1, paying anywhere between $30-90, and receiving nothing 3 years later, with an ever-delayed release date. I strongly considered putting out a statement that I had nothing to do with this book — but chose to simply ask Microsoft Press to remove my name from the cover and all marketing materials. I preferred losing my association with this Bible, rather than be responsible for its contents, and its delays.
A new hope
Around the time that I did that, however, I realized that yet –another– name had been added to the pool! It was that of Pavel Yosifovich, a Microsoft MVP whose blog I had followed a few times, and whom I had heard about doing some Windows Internals training in the past, mostly in Israel. I thought highly of Pavel — and he was an established author of previous books. Additionally, he now had a Microsoft e-mail address — suggesting that once again, the series would have a real “internal” presence, who would communicate with the developer team, read source code comments, and more — while Mark and I had only, and solely, been reverse engineering, we had always had help from David’s connections and insight into the developer team, which the new books would’ve lacked.
started writing the #Windows-Internals book 7th edition… almost done with chapter 1
— Pavel Yosifovich (@zodiacon) February 12, 2016
So I reached out, and to my pleasure, found out that Pavel had now become the sole co-author, the previous two having completely abandoned the project with no materials to show for it. Pavel was doing a herculean task of updating the entire book to now only cover Windows 8 and 8.1, but of course Windows 10 as well, which had reached its Threshold 2 (1511) Update, with Redstone 1 (1607) currently shipping to the Windows Insider Program (WIP). While having source access helps, this is still a task that I knew a single person would struggle with — and I really wanted the book to succeed for all of those that had placed their faith in it. I had also, over the last few years, had made lots of Windows reverse engineering, as many of you know, covering large parts of new Windows 8 and later components. This meant significantly reduced research time for me — all while having an amazing co-author. It seemed obvious that I should jump into the deep abyss of Windows Internals once again.
Pavel was extremely gracious in accepting an uninvited guest to the party, allowing me to make many changes to chapters that he had already completed (I don’t know if I would’ve done the same!). This started adding delays to the book, and Redstone 1 was about to ship — we decided to update the book to cover Redstone 1 from now on, and to go back to any places we knew there were changes. As we kept writing, I came up with new ideas and changes to the book — moving some things around, adding new kernel components, expanding on experiments, and the scope continued to increase. It was clear that I was once again, going to cause delays, which deeply bothered me.
Yet, Pavel was always there to pick up the slack, go beyond the call of duty, and spend nights on researching components as well as the more mundane parts of a book (screenshots and graphics). I could not have asked for a more humble host inside the world of his book. As we were wrapping up, I realized that Redstone 2 (1703) was nearing its feature complete date (around January of this year). I made yet another potentially delaying decision to go back, once again, and to hurriedly find any places where I knew changes had been made, and to update as much of the book as I could. I saw an opportunity — to release a Windows Internals book within weeks of a Windows release, covering that Windows release. A feat which had not happened in many, many releases.
And so, here we are today, a little over a month since Windows 10 Creators Update — Redstone 2 — 1703 has shipped, with the update slowly rolling out over the month of April to hundreds of millions of users, with Build 2017 right around the corner, and with a Windows Internals book in the midst of it all, covering the very same operating system. While I apologize for the additional six months this has cost your pre-orders, I do believe it was the right call.
What’s new in the book? What’s changed?
One of the first things that Pavel had changed (other than returning the book to its usual two-part focus on the kernel and related system components) is to better organize key Windows concepts into the first part of the book, instead of having them spread out over both parts — this way, people could get what would likely be 80% of the material that is relevant to 90% of people as soon as the first part was released, instead of having to wait for both. This meant making the following changes:
- Moving Memory Management and I/O Manager to Part 1
- Breaking out Processes, Jobs, and Thread Scheduling in two chapters — Processes & Jobs, and Thread Scheduling
- Moving System & Management Mechanisms to Part 2
- Removing Networking from Part 1
Once I joined, it made sense, with this new flow, to make a few additional changes:
- Processes and Jobs, now being its own chapter, became Processes, Jobs and Silos, which is the internal name for Windows Server Containers as well as Centennial/Desktop Bridge containers.
- It made little sense that we were covering the User-Mode Loader (a section I first added in the 5th Edition) as a System Mechanism, instead of an integral part of the Process section (which made constant references to Part 2). I moved this section to be part of the same chapter.
Outside of these broad strokes, a full list of all the changes would obviously be too complex. I would estimate the sheer amount of new pages to be around 150 — with probably 50 other pages that have received heavy modification and/or updating. You can definitely expect coverage of the following new features:
- Auto Boost [Scheduling]
- Directed Switch [Scheduling]
- Memory Partitions [Memory]
- Priority Donation/Inheritance [Scheduling]
- Security/Process Mitigations [Security]
- CPU Sets [Scheduling]
- Windows Containers [Processes]
- Store Manager [Memory]
- API Sets [Processes]
- AppContainer [Security]
- Token Attributes & Claims [Security]
- Protected Process Light [Security / Processes]
- Windows Subsystem for Linux [Architecture]
- Memory Compression [Memory]
- Virtual Trust Levels [Architecture]
- Device Guard & Credential Guard [Security]
- Processor Enclaves [Memory]
- Secure Kernel Mode / Isolated User Mode [Architecture]
- Pico Processes [Processes]
- Power Management Framework (PoFx) [I/O Manager]
- Power Availability Requests [I/O Manager]
- And a lot more
Thank You!
Finally, I’d like to thank many people, inside and outside of Microsoft, that helped with some of the content, ideas, experiments, etc. Especially Andrea Allievi, who helped with some very hairy parts of the Memory Management section!
I know both Pavel and I hope you’ll enjoy this flow a bit better, and that you’ll have lots of reading to do in this new Edition. Feel free to hit me up at @aionescu as usual.