Raymond Chen again;
Why do we even need to define a red zone? Can’t I just use my stack for anything? - https://devblogs.microsoft.com/oldnewthing/20190111-00/?p=10...
A closer look at the stack guard page - https://devblogs.microsoft.com/oldnewthing/20220203-00/?p=10...
According to this article, allocation in page sizes with implicit probing is used;
Stack clash mitigation in GCC, Part 3 (-fstack-clash-protection option) - https://developers.redhat.com/blog/2020/05/22/stack-clash-mi...
You have a desirable performance optimization feature —used in every Linux program— that happens to interfere with a lousy exploit mitigation.
No one should ever need more than 64kBs for a stack anyways.
Well, if people would stop storing anything except than return addresses on the stack, yeah, probably even 32 KiB of stack would be enough for anyone. It'd also single-handedly stop all kinds of stack-smashing attacks, too: can't overwrite a return address on the stack if nothing stores data on the stack except the CALL/RET instructions.
Unfortunately, the current zeitgeist is still to have "writeable stacks" which are only moderately less horrible for the security than "executable stacks".
With gcc, you get stack probing with -fstack-clash-protection, which is similar to _chkstk but GCC inlines the stack probes.
VLA got a bad name because of stack clash attacks, but without stack clash protection these attacks can appear also without VLAs (and the first such attacks actually exploited fixed-size arrays), and if you activate this protection there is IMHO not much reason to avoid VLAs.
If you need a small variably-sized buffers, VLAs are almost always superior to any alternative. alloca is worse in every way (see above), a regular array with worst-case bound increases stack use relative to a VLA and does not encode the correct dynamic size which makes bounds checking weaker, and moving the buffer to the heap is slower and complicates the code.
If you can not properly account for the sizes of the things you put on your stack and worry about VLAs exceeding the limit (but again, regular arrays with worst-case size increase stack usage compared to VLAs), on GCC you can use -Wvla-larger-than to make sure the size of each VLA stays bounded.
I'm also very impressed by part 2. I have my own lisp but I haven't managed to implement a compiler or code generation yet. Really enjoyed reading about the hashmap too. The textbook solution to collisions is probing and comparison. It never occurred to me that I could just resize the underlying array until the collisions disappear altogether.
He researched different methods. My favourite for elegance is to recursively hash the hash n times.
From UNIX 7th edition all the way up to C99, when VLAs where introduced, only to be made optional in C11, and the C23 update still doesn't support automatic VLAs, only for function parameters, thus the point stands.
I mean, Chen has decades of winternals in his head. Microsoft has been gutting their staff for years now. When the Petzold/Chen generation hang up their spurs, does Microsoft still have a critical mass of people who understand Windows from the metal up?
I haven’t had a student in two years that was even remotely interested in ring-0, internals, or really understanding a debugger.
I’m not being critical; they are just focused on higher level abstractions.
I wonder if it it is more that the percentage of people who choose to dedicate themselves to that type of work is miniscule. I work in graphics and performance, and it seems similar.
Few people really work on it specifically at any given company, and I've heard people warn others that there are few jobs in it.
But video game companies really want people for those roles and will pay well because they're hard to find. Still, few programmers show any interest in specializing in those skills. If you're passionate about it and willing to learn the details you'll eventually find a lot of job opportunities. I know people who want to work with this and do so - I know many more who have specifically said they want to stay away from it. I don't know anyone who wants to and can't.
There are a lot of great resources out there. The best modern beginner friendly resource I am aware of is Scratchapixel[0]. Back when I was first learning 3D I used to follow tutorials on places like NeHe Productions[1], which is probably a bit dated these days.
For more comprehensive information on all kinds of techniques, with examples from big games for each technique, the absolutely best resource is the book Real-Time Rendering[2].
If you're interested in ray-tracing rather than rasterization (i.e. more film than video games) a lot of people recommend "Ray Tracing in One Weekend"[3]. If you want to learn state of the art ray-tracing in depth, with all the math and and theory, the best resource is "Physically Based Rendering: From Theory to Implementation"[4], which is freely available online.
[0] https://www.scratchapixel.com/
[2] https://www.realtimerendering.com/
- The OS Dev wiki - Open Source Firmware Conference — TKey (shameless plug) - Tiny Tapeout - wafer.space
In rough hierarchical order from software to metal.
> I haven’t had a student in two years that was even remotely interested in ring-0, internals, or really understanding a debugger.
I know quite a lot of such people (even in student age) who are interested in such topics. I really have a feeling that you chose the wrong students at the wrong universities.
Evidence for my point: rather recently, No Starch Press published quite a lot of about such topics - I am rather certain that a publisher knows quite well which kinds of books do or don't sell well at a given time:
- The Book of Debugging https://nostarch.com/book-of-debugging
- The Linux Memory Manager https://nostarch.com/linux-memory-manager
- The Art of 64-Bit Assembly, Volume 2 https://nostarch.com/art-64-bit-assembly-v2
- The Ghidra Book, 2nd Edition https://nostarch.com/ghidra-book-2e
- Building a Debugger https://nostarch.com/building-a-debugger
- Microcontroller Exploits https://nostarch.com/microcontroller-exploits
- System Programming in Linux https://nostarch.com/system-programming-linux
- The Art of ARM Assembly, Volume 1 https://nostarch.com/art-arm-assembly-volume-1
- Getting Started with FPGAs https://nostarch.com/gettingstartedwithfpgas
- The Book of I²C https://nostarch.com/book-i%C2%B2c
I work with Duke University, the University of North Carolina, and Carnegie Mellon.
You don't know that "young engineers" are buying those books. I didn't claim that no one is interested in low-level development. My point is that most younger developers couldn't explain the difference between a mutex and a critical section, or how the OS handles a thread quantum, if their lives depended on it.
I could list a dozen new books on how to build an LLM from scratch. That doesn't mean that most developers understand LLM internals.
As an aside, I love your username. I have a tattoo of Aleph One. ;-)
To my knowledge this is taught in some "Operating System" course, and typically students have to do a hands-on implementation of at least some central parts of an operating system. So I guess these students simply did not pay attention in the respective course. :-(
It isn't that they didn't learn it: the issue is that most CS students graduate and work in areas that require zero OS knowledge. For example, when would I spin up a thread versus a fiber? Even ring-3 devs need to have some level of understanding if they want to create performant software.
First, I didn't work with the right universities. Now the students I work with "didn't pay attention".
/ignored
I can only say that my university-time experience was so much different, and I do observe a similar interest among younger students. So I am hypothesizing what could be the reasons.
But concerning your point, I would indeed claim that the hypotheses "wrong university" and "students did not pay attention" are positively correlated: I think a university where many students are not actually not very interested, excited and curious about the topics that are taught in the lectures does not form a good learning environment.
- Windows Internals, Parts I and II
- Windows 10 System Programming, Parts I and II
- Windows Kernel Programming, Second Edition
- Programming Windows, 5th and 6th Editions
Any ideas how this compares with Kerrisk's The Linux Programming Interface? I've only so much time to read one 1000+ page book...
Unluckily, I don't know, but comparing the Table of Contents for both books
> https://man7.org/tlpi/toc-short.html
> https://nostarch.com/system-programming-linux
I would claim that The Linux Programming Interface covers a broader range of topics, and I also think this book goes more in depth. On the other hand, System Programming in Linux seems to be more pedagogical, and is more targeted towards people who profit from doing exercises and programming projects to get their hands dirty.
I'm essentially arguing that unless you work at MSFT, there's next to no reason to learn that specific abstraction layer.
Really? Understanding the cost of ring transitions is incredibly useful. I recently consulted with a company that was having horrible performance issues, and it came down to the fact that the primary developer didn't know that certain Win32 calls forced ring transitions. The entire fix was switching from a mutex to a critical section (one causes a ring transition, the other doesn't).
Treating the OS like an impenetrable black box will bite upcoming engineers/companies... eventually.
I still twitch whenever someone says "use ddd" and they are not referring to Evans' seminal work.
:-D
A similar "brain drain" has occurred in macOS (formerly known as OS-X) over the years, as evident in man page documentation for "newer" daemons shipped. An easy way to verify this is to run:
ps -A | awk '{ print $4 }' | grep 'libexec/.*[a-z]d$'
While this exercise is illuminating, it is also depressing IMHO.
It was never OS-X, it was OS X, and originally Mac OS X, as in the one after Mac OS 9. The Mac prefix was dropped with Lion (10.7). The Mac OS lingo having itself been introduced with 7.6, before that the OS core was called System.
> It was never OS-X, it was OS X ...
If my worst sin is an introduction of a hyphen, then I can live with that.
Microsoft new blood has been educated on Macs and ChromeOS, even if they do games it is most likely consoles.
On WinUI community calls you usually would get puzzled faces when the Q&A touched when would WinUI be able to do "insert basic Win32/Forms/WPF" feature.
Management apparently doesn't care they actually understand Windows, or get the required trainings to meet the quality of their predecessors.
That is how you get Webview2 all over the place.
There have been some writings and posts here about Microsoft. Here is one from last spring, from a guy that was long time Windows core developer and moved to Azure group. It's well worth reading, what he writes about challenges they have had and most likely still have if not even worse now.
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
and the related HN thread
https://news.ycombinator.com/item?id=47616242
And from what I've understood old chaps like Dave Cutler are involved much less than they were for a very long time.
As per his interview on Dave's Garage, besides being of an age where he really doesn't need to work, at the time he was involved on getting Linux running on XBox on Azure, apparently Microsoft uses idle consoles from XBox Cloud for AI.
Oddly enough the open-source nature of the latter kind of takes away some of the thrill. Eerything is just... there, whereas with Windows there's always quite a bit of digging and investigation involved. Or maybe this is Stockholm syndrome; I dunno.
Stack Overflow was essential to figure out arcane flags that could solve my issues (when even MSDN, another great site with its examples, couldn't) and Raymond was very present in SO at that time, iirc he replied to one of my questions too. That's when I found his blog, always great reads!
Now I've been a 14-year Linux user and none of the toolkits and libraries give anything close to the winapi experience.
Surely you mean leanest and meanest :P
I'd also say that tooling on Windows is simultaneously better and easier to use than on Linux; the noob case of green play button in an IDE is taken care of, but if you want detailed performance and memory profiling, record-replay debugging, hot-reload, all of this is straightforwardly available on Windows.
But detecting objects in video also doesn't fall under the new definition of AI, which means LLMs and image diffusers.
To go even further off topic, being called a bot is certainly a wake up call for me that the internet is dying, and I'm not ready for it, and need to reposition myself asap somehow.
Probably enough to keep Windows going, at least.
> and want to
Not if the pay or location is uncompetitive.
On compiler generated x86 code, the base pointer register will quickly follow when entering the target function.