This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.
I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.
> basically just one for system and majority of serving workload ran on another priority and the rest was for batch
RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
> Application backends?
I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.
Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).
The errors were pushed to the client just like you suggested...
As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.
I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)
EDIT: The books have been given away
Worth reading:
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.
Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.
It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...
> At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%.
If you load an issue page, you'll see 1 failed request to:
/project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:
/in-product-messaging/copilot-budget-request-banner
/in-product-messaging/code-scanning-ai-findings-preview-banner
/github-copilot/chat
/_private/browser/stats
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
This is a pretty ancient thing as far as the FE goes: https://www.stevesouders.com/blog/2010/06/01/frontend-spof/
But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
Makes me wonder how many folks at GH and MSFT could even pass their own interviews.
Thanks for sharing though. I learn more about systems design from HN comments than anything else
This.
It's easy to be an "armchair quarterback," here, though.
Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.
Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.
Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.
And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.
It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.
I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.
I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.
"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "
And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.
Note: It's sarcasm.
If it's not your job, then just ignore the reports.
If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out.
Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.
If you have a highly conscientious personality, this is easier said than done.
Just doing what others wish is not conscientous in itself! It _may_ be depdending on situation but it can be just pathological towards the self.
When it's psyhocologically hard to do things you imagine will dissapoint someone that's probably not concientousness. It's more like low self-esteem or codependency.
It's very hard for someone to tell these apart themselves. Hence when this topic pops out it's good idea to remind that being super-accomodating may in fact be a personality flaw - that can be healed if acknowledged.
There is very large spectrum between "trying not to dissapoint anyone" and doing what you know is the right thing.
You don't owe the world anything at all. If you're conscientious, then give a little -- here and there. Don't turn it into an unpaid job.
Yet they kind of did. I've limited participation in my libraries with GitHub's setting that nobody who made an account in the last 6 months can do anything in my repos (after some misguided hustler thought they're an easy target and posted an ad).lp
Time's marching forward though. Wonder what will happen after a few more months. We'll have bot spam accounts that are no longer as fresh.
A great majority of business applications do run on open source projects, and in turn, are affected by them if things go awry. It’s a prisoner’s dilemma in this case.
(I don't mean you. just these so called open source developers.)
https://github.com/uclouvain/openjpeg
Basically the only library for reading jp2k data (complicated specs, ask your AI to one shot an implementation, mine said "it's 3000 lines of fiddly spec, too complicated"). Issues full of buffer-overflows. Recently unmaintained.
Used in tons of projects, now all possibly vulnerable.
did we lose anything of value? probably not.
If one claims such extraordinary figures of 100x increased productivity, a step forward never seen in the history of humanity in such short timespans, they must present extraordinary proof or be branded as a complete lunatic. I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.
So, what are you doing/ have done with all your productivity?
I’m only back at it four months later and I don’t really know what happened before, or why it’s working now, I’m just happy that I can scratch that itch again.
Another possibility is that the people who experience these 100x productivity increases are honest, correct, and simply had abysmal productivity which has now been increased to near-average junior levels thanks to AI.
The ratios are factual though. Just look at the "Insights" tab of any LLM written project. https://github.com/oven-sh/bun/pulse
This kind of velocity is impossible to achieve manually.
AI built me a 1.5k+ LOC react component which is probably a 15x increase on the file size I would have created, with negative impact on the project for those extra LOC.
I spent a couple of days on those, and its would have taken me months to write manually I am sure, so in that regards it's close to 50x.
I've also had Claude track down some logic issue in a module I was unfamiliar with which had very large and complicated flows. Would have taken me many days, since I did not have a reproducible case, so had to go by logs and customer description alone. I spent 5 minutes writing a prompt and when I checked back, Claude had identified the issue. The fix I had to implement myself, but was fairly easy. So there Claude definitely was a 100x increase in productivity.
Then there are cases where they're much more modest, or where they might even be negative, when they think they're fixing stuff but actually are introducing more bugs.
Most software work is just churn / doing the same thing over and over. More productivity can just mean more output, not better output.
Higher resource consumption and the set back in CO2 reduction?
On the other hand recent papers highlight the validity of concern/suspicion:
> This systematic review demonstrates that the environmental footprint of artificial intelligence is a structural and increasingly consequential challenge, shaped by interdependent decisions across algorithms, software pipelines, hardware infrastructures, and deployment contexts. The synthesized evidence shows that energy consumption and carbon emissions associated with AI systems are highly variable, context-dependent, and often underestimated — Beyond Efficiency: A Systematic Review of Energy Consumption and Carbon Footprint Across the AI Lifecycle (published in “Sustainability” an international, peer-reviewed, open-access journal) https://www.mdpi.com/2071-1050/18/3/1359
That's the whole 21 minute 59 second video in a nutshell.
A loud and hectic quick cut rambling video essay. Millions of views, naturally.
> water is not unlimited
> you don't want to release a bunch of acid into a river
> I know next to nothing about this
> I know basically nothing
> corn is one of the thirstiest major crops grown in the US
What is this rant supposed to inform? Whats wrong with OP being concerned about the costs of operating a DC?
When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative.
The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproductive.
I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets.
So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs.
Since the rise of AI the environmental impact doesn’t seem to matter anymore.
I guess because it’s the shiny new toy of the hackernews audience.
Privacy also lost importance given the fact that the same people who refused to give information like their phone number to companies like Google and Meta now upload their whole life to their AIs to asks what should the eat, hyperbolically speaking
The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.
That is moderate. Energy-intensive industry is around 130 EJ, and global final energy consumption > 450 EJ.
Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035.
Now that was about operational energy consumption. Someone might bring up manufacturing and construction.
From what I could find the climate impact of those are estimated somewhere between 10-35% of the total climate impact of data centers, so relatively small compared to the operational energy consumption.
It is very hard to justify more than moderate environmental impact here, in my opinion.
For the benefits of AI, my personal results have been great, so I am quite optimistic. And objectively, I find it hard to ignore recent results in mathematics and security research.
https://www.iea.org/reports/energy-and-ai/energy-demand-from...
That is today where we already consume too much. If by 2030 AI's consumption doubles it gets worse. While training large models draws major initial power, everyday AI usage (inference) now drives roughly 80% to 90% of cumulative AI energy
"Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035."
Seems like AI helps slowing down the rise of energy consumption.
We are at a point where we want less CO2 not moderataly more. In the end more is more.
If your doctor tells you to lose weight or you get sick it's not a success to gain weigth slower
And if the potential of >13 EJ/year is actually realized, it would seem like the net impact of the data centers is not just "moderately more CO2" but possibly "moderately less".
Please avoid low quality analogies on HN.
Anything but a reduction is bad and AI is a setback for that.
Potential benefits are as long useless as they aren’t realized.
BTW the energy consumption reduction is achieved by what kind of AI? LLMs?
The construction of data centers needs resources and also creates more the CO2.
The energy for these data centers is often created through fossil fuels which also creates additional CO2
What do you think why Google and MS scrapped their CO2 reduction targets
There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.
On CVE probing, and I haven't really seen anyone describe/use it (or I may be oblivious), but the way you do it is you curate a list of CVEs for the class of software you're writing, say a web server. Then you take this list in chunks and hand them off to your agents to devise and implement adversarial technically analogous attacks against your codebase. If it's red, report and patch. Ironically (even with Fable 5) it's never complained/refused to do it.
What I have noticed an increase though is in demands and pressure to deliver.
EDIT: Instead of simply down-voting, you're welcome to name examples that proves me wrong ;)
So, if it doesn't then you learned something about your workplace (and it's not good).
In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.
But ofc, slop has increased a lot more as well.
How long do they last until the get replaced by the next feature?
https://github.blog/news-insights/product-news/github-copilo...
An issue reported by a person account but post made by AI. https://github.com/oven-sh/bun/issues/39800
AI (robobun) responds and creates PR. https://github.com/oven-sh/bun/pull/37459
AI (coderabbit, claude, github actions) review the PR, AI (robobun) applies the fixes.
Some AI back and forth.
A human finally merges the PR.
Not gonna lie, it's kind of beautiful.
The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't you think there would be a better way to ensure an object gets collected, like setting the variable to null?
The comments in the code don't make a lot of sense either. Something so obscure and brittle has to be explained extremely clearly.
While the issue might be real, this commit is so far away from the locus of normal that it's sending red alert. Plus a hallucination is very likely with such a long investigation - once an LLM agent starts investigating it just assumes there is a problem. And this is the 1 out of 1 robobun commit that I looked at.
Edit: here's the next one: https://github.com/oven-sh/bun/commit/72ec6e2594892455df0090...
Make sure the fs module keeps working if someone freezes or seals its exports table. I was wondering who was going around freezing random tables from other modules, so I checked the linked issue - robobun reported the issue, too. Why? I'm skeptical of whatever robobun was doing when it decided that it was necessary for code outside of a module to freeze their export tables. It needs a very good justification.
Don't know anything about the second one.
I've had opus 5 along with its AI code reviewer agree to do some pretty stupid shit.
I am currently using bun, but may have to switch. I can't see how this can possibly turn out well in the long run...
Wow, is Bun the record holder for number of PRs?
I recall GitHub recommends to keep the number of PR to a certain level due things such as GitHub Actions slowing down.
Sure AI workflows might be a non-negligible share of all that usage but still the point is that initially forges existed to help developers collectively share a state then solve problems. Nowadays they are basically online filesystems with better notifications for other software to interact with and only optionally developers actually communicating.
https://github.com/oven-sh/bun/pull/39743 https://github.com/oven-sh/bun/pull/39735
robobun: "The ordering is load-bearing: reclaiming before this block made is_dead_request true and hung a parked textStream read (caught by body.test.ts in CI). The comment pins that constraint."
Ah, well, if something is load-bearing, then I guess that settles it. Need a comment to pin that constraint, in case a read is parked. These are words that normal humans commonly use in these ways.
(Always striking how much Claude obsesses over the minutiae of method contracts and side effects, exhaustively documenting them in comments. It’s much happier figuring out how to reorder some method calls with nonobvious side effects so the code works than it is refactoring them not to do unexpected things!)
I don't know if that's sarcasm or not. I know it doesn't work, but that's the future we've been promised, right?
I don't know how much my own time is wasted on Claude imagining API response formats that never existed.
Unironically: no.
On the other hand, rewrite was mostly done in record time. New version added massive number of features. Also huge bug fixes. Being used by Claude code by millions of people. Successfully used by some others even in canary. After release, multiple companies immediately switched due to massive amounts of resource savings and performance gains (and publicly posted about it).
Can there still be problems? Yes, I'm sure there will be. But denying the feat Oven pulled off with Bun in last few months is nothing but phobia/fud.
Many people are already posted about testing new bun version and I have yet to see a single post where the issue is the latest versions of bun. In some cases people posted it doesn't work but that's due to node compatibility etc and it didn't work on previous version either.
One does not need to be bun fanatic to see and call things as they are.
PS: I like bun because I hate how js ecosystem requires 100s of packages to do anything and bun is aiming to include batteries. This is good.
Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue.
As someone on the sidelines, this is really interesting to watch unfold.
If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.
If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.
Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...
However, there are sharks in the water, and with the diminishing mean of user technical knowledge, the product actually needs to become even more free. GitHub likely needs even lower friction.
"All it takes" is the insanely heavy technical lift to support that. There is no other solution. All the C-Suite needs to do is foster an environment with well-thought through, and possibly over-funded engineering, at the edge of the art. That sounds like an amazing challenge.
You can get work done on any software forge. But potential employers will still ask for your GitHub. People will judge your personal project by its GitHub stars and be less reluctant to download a binary from GitHub than elsewhere. Potential contributors will leave a PR on GitHub but probably not if they have to make an account on a new platform and learn how it works.
And of course, let's not assume any competitor can just absorb even a fraction of the traffic GitHub receives without suffering similar reliability issues.
The richest man in the world is interested in this dataset as well.
Or, is the idea just to drive everyone away from your platform?
With enough effort, you can rather obviously run CI per PR commit (it's a programmable system), but I've never seen aUI-integrated way to track the results, aside from browsing custom job names, which is very far from what I'd call "integrated" when compared to PR-level build markers. Similarly, I'm not aware of (but would not be surprised by) any way to disable per-main-branch commit builds, aside from initial pushes.
But I haven't poked around deeply in the settings, and business-account settings are rather different anyway so those might be wildly different / more flexible / more obtuse in exciting ways. Github is a very large and complicated product at this point, darn near anything could exist if you dive through enough UI layers or use old URLs to find soft-deprecated features.
Also, honestly, 100 commits = 1 transaction? That's far more of an over-simplification than anything I've said. It's a massive product with thousands of engineers, there's no chance at all it's just one database.
"Our billion-dollar infrastructure crumbles under a tremendous flood of 50 PRs per second" would just sound embarrassing.
You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.
There’s a difference between pushing 1 commit and 100.
There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank.
It does things like scan for references to issues to index, but it would just scan the log for a range.
I did disagree with GP though because there is no reason to assume that the ratio of commits to pushes has materially changed. So if that is the proxy they have always used for measuring growth, and they know it reliably does that then I think it’s a reasonable way to communicate this to this audience.
Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.
A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.
It’s also like… the core unit of git.
Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable.
Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of some type.
> I can navigate in GitHub directly to any individual commit
You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.
And yes, I agree there is indexing of commits, but that is a batch insert from a log.
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.
I feel it in my fingers. I feel it in my veins.
"We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity"
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
Useful / impressive for whom is the question. Not for us!
We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.
In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
Seems false. Lots of coding adjacent people, engineering managers, etc. are now pushing PRs.
> How is it impressive if we all know it's autogenerated?
Nobody is saying the code is impressive, just the growth of github is impressive. It's not doing anythign different based on the source of the code.
At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.
“We misconfigured a sidecar” is something I would think AI could quite easily find and fix.
half of engineering in big tech is just rewriting a system to scale
microsoft is incompetent, they havent changed windows/excel/outlook in 30 years
Luckily for Meta agents are not yet as much into doomscrolling as humans are.
That does not seem to be true - which two-decade period are you talking about? AWS has only been around for ~20 years, and I just reviewed a 10 year period, and not a single one of those years saw doubling in the whole year, let alone doubling in a few months. Which 20 year period are you referring to, and are you referring to doubling every few months over that 20 year period?
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
[0] even at current MS owned github
What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure
1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)
2. The core teams, the people who knew the existing systems have either been laid off or moved from Github
3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them
> "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
These companies were built and run by people passionate enough for the craft, ones who cared for the systems, who designed them. There is this idea that you can replace people by process and everyone is replaceable. What you have is a classical state where people are just doing their time.Not adapting is a choice
I've been saying this for a while: https://news.ycombinator.com/item?id=47534499
But it does raise the question for especially commercial users of Github whether it's time to reconsider the relationship with Github and maybe not put all our eggs in one basket. Basically, this wiped out a whole workday for many companies.
I'm not that eager to start self hosting my stuff. But I am considering it. Besides availability, CI build performance is also becoming a blocker for us. My AI coding jobs creating lots of PRs are making that a bottleneck. Fixing that in Github would require switching to a paid plan. And at that point, self hosting might be the more cost effective option. There are a few tradeoffs here of course. But I like the idea of throwing more memory/cpu at this to get blazingly fast builds.
GitHub is the de facto standard.
A majority of this increased traffic is the codex/claude in auto mode used by people who don’t even know what git is.
Bonkers.
You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.
Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most) knowledge work are being driven through the floor, artists are being squeezed more than ever, etc., etc. That's to say nothing of the existential threats to the economy, environment and critical thinking which are growing daily. I really think we've lost the plot, folks.
To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before.
I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
We're going to see is a lot of stuff people use LLMs for moving back to using bespoke algorithms solving the one specific issue. LLMs are just machine learning unlocked for the masses, where you can ask it "translate this for me", "what's the sentiment of this text", and it just does it. Any of these were already achievable before LLMs, they just needed a machine learning engineer to implement.
I myself started an LLM driven project to classify and sort all the photos I've got. I just ask an LLM to poop out tags for an image. Using a proper algorithm for this is possible, but I'd need to put in some effort to actually get it set up. LLMs do make it easier since they can directly, instructively help you that set up, and provide an easy stream of training data to distill out a model perfectly fit to your usecase.
I think majority of work currently going through LLMs is such inefficient tasks that can be solved quicker and better with a limited algorithm.
You don't need a chatbot.
I could try to market this app, but I tried that 2 years ago, and it was already essentially a failure. So I'm probably going to keep it for myself or just try to find a few users who like it and will use it for free.
On the other hand, famine is coming. Come back and read this next year. Was your dalliance worth these food shortages?
...that sounds pretty worthless tbh. The promise of AI wasn't that you can bang out your own little half baked duolingo clone. It was that every engineer was supposed to become hundreds or thousands of percent more productive. The assumption being that things generally would therefore get noticeably better.
We're seeing hundreds of percent more commits. We're not seeing that translate to any real world improvement of anything. That's the problem.
Call me "ignorant" if you want, but back it up by showing what all this "AI productivity" has actually accomplished in real economic terms. Not vaporware.
Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
When I ask Claude about the AI ROI, it seems to cite that ~95% of AI PoC's are negative ROI but the 5% that do have ROI tend to have decent return -- my guess is mostly back office clean up to reduce expenses and increase revenue/profit.
IIRC Uber claim their 1.5k/mo budget lead to no new value creation.
FWIW, I look at the situation with similar skepticism. One argument you could make was what Marc Andreesen said with the hypothesis that "all the big companies get nuked", so no new valued gets created but the large market cap companies get eaten by thousands of little pirañas. Obviously, so far, that doesn't seem to be the case.
I mean real utility impacts. More reliable and efficient software, better features, etc. So far we've seen none of that. In fact, it seems like the opposite--more volume but a much lower quality product.
Let me tell you, it is _silly_ the amount of vibe-coded software in this area is popping up every week. Is the software duplicated? To an extremely large extent, yes. It is useful? Yes, but each piece of software seems to have a smaller and smaller audience, and quality is often severely lacking.
Iv'e done my own, too, for "RF debugging" as I called it to look into SNR issues related to interference. The software getting produced is likely useful _somewhere_, its just that youre not going to notice it.
Are we just moving towards personalised digital assistants for everyone, which in turn will produce software to function? Not unlikely.
Where is anything saying there should be? If they were all private hobby projects, you'd have no grounds for complaint.
What are the boosters ranting and raving about if not the above? Am I just confused and the end game for all of the AI hype, investment, infrastructure build out, etc. over the last N years is ... hobby projects?
Here's a quote I found after a quick web search for "dario ai benefits":
> The list of positive applications of powerful AI is extremely long (and includes robotics, manufacturing, energy, and much more), but I’m going to focus on a small number of areas that seem to me to have the greatest potential to directly improve the quality of human life. The five categories I am most excited about are:
> Biology and physical health
> Neuroscience and mental health
> Economic development and poverty
> Peace and governance
> Work and meaning
Dr David Burns offers his Teams CBT therapy with an LLM based app for people who can't afford therapist visits[5].
U. Chicago says that AI weather forecasts can be faster and cheaper than supercomputer-based physical simulations and produce results weeks ahead, which makes accurate forecasts affordable and available for farmers in middle-income countries[6].
With your smartphone camera you can get an AI-assisted bike-fit at home[7].
You can get a bike fitness training plan[8] and performance analysis with an AI that you can chat with to analyse and explain or change it, used by three professional cycling teams.
Prophix Australia analyse photos of powerlines taken by helicopter using AI to track assets and identify and verify potential faults[9]. AI systems scan images for early signs of plant disease in the vineyards of the Swiss Alps [also 9]. AI systems analyze multispectral images and satellite images of crops to detect nutrient deficiencies or pest problems in large farms[also 9].
In 2020 - 2022 a trial of a system in a hospital to identify patients at risk of sepsis helped medics treat them 2 hours earlier and reduced the death rate from sepsis by 18%.[10]
Self-driving cars now exist. Camera drones which follow you and respond to gestures now exist.
[1] https://newsnetwork.mayoclinic.org/discussion/mayo-clinic-ai...
[2] https://www.breastcancer.org/screening-testing/artificial-in...
[3] https://www.nature.com/articles/s41698-026-01276-6#Sec17
[4] https://www.cancer.gov/research/infrastructure/artificial-in...
[5] https://www.feelinggoodapp.com/
[6] https://climate.uchicago.edu/insights/ai-is-transforming-wea...
[7] https://www.myvelofit.com/
[9] https://medium.com/design-bootcamp/what-has-ai-ever-done-for...
[10] https://www.scientificamerican.com/article/algorithm-that-de...
Crazy how anyone would think it would.
Is this all wasted if commits drop in a few months because folks are just pushing a lot now to figure out what works and what doesn't and we eventually settle into something between where we are now and where we were a year ago?
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
I assumed it does, do you know that it doesn't?
Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right?
It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitHub was already so mature as a solution when they bought it, I don’t know that they could have added that type of value just by slapping a Microsoft logo in the footer.
They just added 3 million CPU cores to the fleet. That’s a lot of cheddar.
For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins.
It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.
They already did this, no?
Right now anyone can publish to public repos in an unlimited manner. They could choose to limit that and elevate unlimited to a paid co-pilot of codex bundled plan.
15 years from now, they'll say it was obvious.
The worst outages I've been part of always have some version of this :(
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
“Adding complexity to the reverse proxy” is kind of the central feature of service meshes, but the point is you pick one that’s battle tested and solves more problems than it creates.
Do you know of a documented instance where it would have helped?
I felt dirty implementing such a solution (introducing randomness for the sake of randomness is off-putting), but it worked. It wasn't until a while later that I even heard the term "jitter" in this context and realized this was a pretty decent solution for this kind of problem.
Regardless, if you're going to introduce something where adding jitter is appropriate, I'd just just add jitter. It's not premature optimization; it's an essential part of that kind of functionality.
[1] https://en.wikipedia.org/wiki/Exponential_backoff#Example
If you have a switch in your network, or a speed faster than 100Mbps, you aren't using this model.
However, WiFi uses it.
If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.
It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…
More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.
Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.
Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.
It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.
Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.
Added jitter to clients and turned off retries on the serverless platform.
I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulative delay if it's really costly to do retries.
Request amplification via retries is a different problem that causes large amounts of traffic (but it is generally more steady than spiky)
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down.
I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe if you’re working at Jane St or something; or other circumstances where you can deploy immediate changes to the client; and you’re willing to trade ‘better p50 for worse outages’.
But in the case of shipped code that’s run on clients, I’ll continue exponentially backing off all the way, all the time, for everything.
To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. Then service A knows service B is dead, does not retry, and proapgates the failure to clients. This is hard to do perfectly, but there's no alternative that works.
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.
I'm not proposing doing any exponential backoff retries, or even retries at all for internal services.
In my mind, the retries with exponential backoff and jitter belong only on the end-client(VSCode in this case). Everything else -> fail fast.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif
Yes, exponential backoffs alone are insufficient. Yes, adding jitter helps randomize the calls across a fleet and should be the default with exponential backoffs. Yes, both of these may be sufficient for most systems. Yes, you can dive more into circuit breakers and adaptive retries to limit thundering herd.
https://aws.amazon.com/blogs/architecture/exponential-backof...
When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will time out before you finish all the work, at which point you're doing a bunch of slow expensive work for no gain (and the enqueued slow expensive work will make your outage worse).
There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any of your ingresses (e.g. triggered by an ETL pipeline, or a scheduled job).
With a complex architecture you can't just slap a circuit breaker on a couple of ingresses and call it a day. Don't get me wrong, putting them there does go a long way, but you won't be covering all your bases.
At least it should be required for critical level services in production.
“Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket. This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted.”
I’m particularly fond of the token-bucket mechanism for pacing recovery.
Retries in such a situation should be handled internally with the client at most responsible for failing over with a circuit breaker to another zone. Having the client auto retry right away is not something that behaves well as shown here, even if in the happy path it happens to stimulate increased availability without actually investing in the proper architecture for it
https://web.archive.org/web/20250128235041/https://codahale....
(Also both CP and AP have alternative very dodgy meanings)
That wouldn't be a unit test - that's more like an end-to-end or integration test.
Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.
Unit tests already mock network calls so create mocks to return errors instead.
Some days, I would seriously rather work at Wendy's.
Narrator: He would not.
Money is not important to me insofar as I have enough to live an average life. I don't need anymore than that.
Or, don't tell them and do it anyway. It'll make your own life 100x easier, unless you enjoy the stress of bug fixing and responding to incidents where you've broken something.
Writing code without tests because it takes too long is like running a car engine without ever bothering to change the oil - it'll run for a bit, and eventually fail catastrophically.
A lot of retry libraries I’ve seen require the user to configure them. You can use a library with all the right settings, but if you configure it wrong, you are really no better off than if you hadn’t
And in fact you can see the degradation of software over the previous decade-plus via Google Trends search for ‘something went wrong’ lol: https://trends.google.com/trends/explore?date=all&q=%22Somet...
Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".
Call the function multiple times and ensure it'd passing the expected wait durations in for each time it's called followed by a fatal error after say 30 seconds.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
Cloudflare outage on December 5, 2025 [1]:
> However, we have never before applied a killswitch to a rule with an action of “execute”.
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
[1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
I recently tried zed but it ddos’d my nas over an smb share lol
Intellij for basically all software development that is more than 2 minutes.
Just pick a specific task that you do routinely (say markdown editing) and just use neovim for that one task - eat the elephant one bite at a time and markdown is generally small enough/light enough that you won't get out into the weeds trying to make neovim the perfect IDE in the process.
Ergonomics are highly variable, so it might not help someone else, but for me it did the opposite of "give me carpal tunnel syndrome."
They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted.
Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.
It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?
That means nothing, especially with all the layoffs.
One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.
The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code.
A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time.
I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.
Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.
It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.
Same with Oracle. Both of them are good at getting contracts and then executing them not poorly enough that it's breach of contract but not well enough that the customer stops paying. That's their business, not software.
Would you want to go and spend billions to migrate off Microsoft/Oracle when you could just not and get to the next quarter.
At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.”
A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.
I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...
Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.
Probably the biggest thing that has me dumbfounded about everything in the AI space and the tooling in GitHub and stuff with co-pilot. They've figured out a way to make us pay them to steal all our valuable data. And to package it up all nice for them with a bow on it and not question it.
IMO definiely time to segregate the free and the paid service. Otherwise those of us who need the service to stay up will have to look for alternatives.
The post mortem conclusion should've been to put usage limits on the free tier so paid users aren't killed by them.
I'd say kids today are spoiled, but there's no doubt that this has been a rough year for github even if it is understandable circumstances.
I'm not sure there's some solution for what you're complaining about other than the government coming in and saying they have to split github. What are you advocating for here, or is this just complaining?
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful
Retries are a great way to turn errors rare into common.
Source: decades of operational pain.
But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.
I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?
Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means
In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.
I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.
As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.
In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient. A case I run into often is doing something with my phone while leaving the house. There’s a window where it still thinks it’s on the WiFi but it’s too far away for it to work anymore. Initiating an action in that window often produces an alert telling me to try again, and trying again a few seconds later almost always works.
Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.
For sh*ty providers they are great. Best of all when backsourced to the user by "Try again later."
> As it obscures it when stuff has genuinely gone wrong
Works as designed - at every level.
Crazy.
It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know
We have some marketing folks that have gotten into vibe coding stuff for their personal use. First let me say: good for them and I'm glad they're experimenting with new ideas and tools. The side effect of that is that looking at their repos, they're having Claude go whole hog and make upwards of hundreds of commits a day, all with things that they haven't taken a look at.
I don't think I can say this is wrong of them, because their tools encourage that and they shouldn't have to consider their impact on an enterprise service, but I wonder if this trend is similar in other places.
Their Copilot cloud agent offering is suffering with a case of some of the worst corporate ADHD I've seen. We built a cloud agentic development pipeline on it, and it seems like almost every other week they silently change something with zero public announcement that creates real disruption for our team.
Note: that's not bugs in the Copilot platform like the article discusses. That's real, breaking changes to the platform that clearly aren't being tested/reviewed before being pushed to prod, with zero public announcement or documentation.
Support is useless – we're paying customers in the 4-5 figures and our tickets go unanswered.
I love(d) GitHub, but I do think they've lost enough public trust at this point that their time is ticking. With talk of new VCSes designed specifically for agents, I do believe it is just a matter of time. Which pains me somewhat to say.
They are under no obligation to continue functioning as a bottomless free repository for text file hosting, especially now that text file creation has multiplied exponentially. They could make a few almost purely commercial changes and solve this without any major re-engineering while maintaining their status as the go-to public / open source code hosting platform.
1. Immediately increase pricing of all enterprise licenses and add super-committer overage fees.
2. Rate limit or cap commit size / frequency for public accounts.
Their service is more valuable than ever and switching is much harder if people have automation built up on their platform. Now is the time to cash in their chips.
And the positive externality of increasing commit cost would be forcing people to have some semblance of restraint for the AI content they generate.
It is specific and technical.
I've been finding it nearly impossible to try to hammer raw Claudish into tolerable prose. I usually have to re-write most of the text by hand if I want it slop-free.
P.S. Hi Claude! I know you might be reading this :) Don't take it personally. Let your trainers know that you need some help with this.
There are a lot of ways to avoid exponentially more commits, issues, and PRs breaking your backend down, and begging every visitor and user to please use AI to write 40x more code that needs 40x more fixes is not one of them.
And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.
I feel like this becomes a lesson on how NOT to design and operate a SaaS.
I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.
I don't like paying for free stuff but gh certainly worth it.
Which mythical tech company - free or paid - does this?
Might force people to review their slop before pushing it.
But I'm not going to pay per commit over my monthly plan
I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.
> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.
Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.
With that mindset I'm not surprised this happened and it will surely happen again.
Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.
Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.
I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.
No wonder why GitHub is down all the time.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
the increase in traffic over the last few years is way bigger than I thought
The vibes are off.
What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
Central US data center failed to scale with it
Why let your entire platform suffer?
I'd say I saw these because I follow certain creators and/or high signal reposters in the fields of software and games.
And, if you'll forgive me, I myself am working on a totally new way to make games, called Jinks (multi-platform Web, Mac, Linux, Windows, even back to Wii and Dreamcast; games can be introspected, edited, queried at runtime; one game released so far to prove it; work in progress)
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/
Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.
Azure just has very poor performance and reliability characteristics. It’s a particularly bad migration target for a colo-based company that mainly runs on owned hardware (such as GitHub or LinkedIn). Requires much larger architecture changes than (say) a company coming from AWS.
for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail
EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option
EDIT 2: the email part would be PITA, so $4/mo is attractive on that background
It's an expensive, low revenue generating site.
There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.
How is it "ripe" for disruption?
That can't be cheap.
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
Plus the UX isn’t nearly as good. Sometimes it feels downright clunky in comparison.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
Pretty sure this line appears in every one of these.
Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.
In addition they say that they will continue their migration to azure and that azure is supporting their actions run. But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.
I think it's because their leadership actually doesn't care that it goes down. A weekly outage is now an accepted cost of continuing to allow unlimited free access with infrastructure that cannot possibly handle the load. As a result, everyone is looking at their GitHub Enterprise bills and cost of stopped work, calculating how much they'd save by self-hosting.
I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.
I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
We should be building tools that help us all move off of github as soon as possible. The amount of action code that will need to be rewritten is daunting.
GitHub has alternatives, but no replacement
https://news.ycombinator.com/item?id=49135365
Why developers are ditching GitHub for Codeberg and self-hosting alternatives
https://news.ycombinator.com/item?id=48842611
and new entry:
Cursor Origin Code Hosting
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
Absolutely do not recommend using GitLab.
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
I still don't know what it is or what it does.
You're in luck, GP comment described it for you.
self-hosted > You install it on your server
lightweight > It does not consume a lot of resources (cpu, disk, ram)
software forge > offers tools that help with creating software collaboratively (repository hosting, change request management, wiki for docs,…)
https://github.com/google/git-appraise (unmaintained)
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
Here's something an engineer writes, loaded with facts:
"I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called my IT managers and learned we had hard resetting in progress for the past 20 minutes with minimal impact on recovery."
Totally missing from the article.
No results.
Cool
Seems like we are headed for Tech Gridlock.
If you're worried about food prices, you should be happy that robots will make agriculture less labor-intensive and bring prices down.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
Where is LB placed? Is that hte LB in front of API GW? or behind GW? GW can have rate limiter or cloudflare FW can handle the rate limiting. The doc does not show which part of LB is down so it's all guessing at this point so i won't go forward with this.
"You've seen what we've done. The August 21st outage comes next. See you then!"
No side projects? Nothing? Just seems odd.
this is fine.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
It shows zero contributions in the past year, on this account. This is a huge, huge red flag.
Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and his GitHub profile simply tells me: "this guy doesn't code."
I don't care if this guy doesn't work on GitHub itself, I hardly would expect that, but IMO, any CTO of a company like GitHub should eat, breathe, and sleep code. He might, but his profile, which is being published as if it means anything, tells me he doesn't.
Here's me: https://github.com/ryanisnan
In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.