Every large co runs into this problem. New cos have fewer obstacles to paying more… until they get big.
Oh, and yes the preferred language in my browser is set to English.
This solution with WAL and compact resembled the database internals as taught by Martin Kleppmann.
I asked Claude about this observation, and learned that the author (Vicent Martí) wasn't only core Github developer but also worked on Vitess from PlanetScale.
So no surprise these two contexts resulted in the design of Continuity as it is.
https://vitess.io/ - Vitess is a database clustering system for horizontal scaling of MySQL.
however whether or not this improves their team - who knows! team dynamics is a hard thing to achieve, not just throwing good people in the same bucket ..
Again reminded of what an amaizing piece of engineering S3 is (99.999999999% - 11 nines of durability) [1]
1: https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu...
How many startups and corporations built on top of S3 semantics?
Snowflake and the big lakehouses are one of them. They use S3 as giant disk in the sky.
Of course S3 goes beyond just Amazon these days. It's a proven design and architecture for doing object storage that you find across different cloud providers that you can also self host if you need to.
This article will no doubt inspire people working on projects like forgejo, gitea, tangled, etc. that are all facing the same scaling challenges.
Ears wide open for a model that does better for the same parameters. Pareto efficiency is important, but Anthropic doesn't seem to care about it. They desperately need a Haiku 5, IMO.
It can't be any worse than current github, and since we've heard plenty of people express their dissatisfaction with github recently, this is actually a good direction for Cursor. A product people may want. We'll see.
Yeah, and the grass is always greener somewhere else.
The only way to finally end up in a happy state about your git hosting and automated workflows running, is by running the whole shebang yourself. Saves you so much sanity in the long-term, even if starting and scaling is slower.
This is one of the best technical articles I’ve read. And I came away a lot more knowledgable about a technology I have used daily for over a decade and consider myself adept at. Kudos to the author.
[0]: https://aws.amazon.com/s3/ (Click on Durability and availability)
Which is not to say that TFA’s approach is invalid or uninteresting, but the mechanics of how “an S3” works are really obscure to those of us not in the know.
I've got no dog in this fight and no specific rationale, but it smells like a shortcut that's going to make some assumptions that might surprise later (maybe not at the global level, but a local level)
Last time I looked many of the alternatives didn't have as good support for some of the transaction-friendly things like conditional writes.
Hopefully if you're trying to build something complex on top of S3 and have any expectation of potentially needing to run on top of other infra in the future, you know which magic features you need and can see which of them are supported by what vendors/OSS projects. :)
But it's not really that different than doing cool stuff by pushing a bunch of things into Postgres or Kafka or [other thing] and not explaining how those things work.
> Packfiles are the fundamental building block of Git storage and Git networking.
(emphasis mine)
> His approach was storing the objects in a distributed hash table. This was only possible thanks to JGit, a custom Git implementation in Java. Like any good ol' Java library, JGit provides enough interfaces and factories and interface factories to abstract all the details of a normal Git repository, including replacing its on-disk packfiles with a DHT. Although the system worked and results were good enough for normal Git operations, the limitations of the Git protocol (which again, require packfiles to be sent over the network regardless of how you store data on the server) made the git clone performance bad enough to discard the design altogether.
Looks like Java enterprise design patterns aren't all bad after all :-) and the git ecosystem would have profited from a bit of abstraction and separation of concerns here, where network protocol, git domain model, and storage layer are evolved somewhat independently. The domain model is what everyone in the ecosystem needs to agree on, the network protocol is what at least the given local & remote host need to agree on, but storage is mainly a local concern.
Of course the question is whether git would have today's market and mind share if they had gone down that path. The ecosystem would be a lot more heterogeneous, evolving network protocol would probably take much longer, etc.
When I look at one of the recent blog posts (https://github.blog/news-insights/company-news/an-update-on-...) I see crazy charts of PR merged, Commits, Repositories created. These are all git operations and not just some slow Rails app.
> Short term, we had to resolve a variety of bottlenecks that appeared faster than expected from moving webhooks to a different backend (out of MySQL), redesigning user session cache to redoing authentication and authorization flows to substantially reduce database load. We also leveraged our migration to Azure to stand up a lot more compute.
The scaling issues they have don’t have git as the bottleneck from what I understand
> The number of repositories on GitHub is growing faster than ever, but a much harder scaling challenge is the rise of large monorepos. For the last three months, we’ve been investing heavily in response to this trend both within git system and in the pull request experience.
They also mention large monorepos and git read operations as a core issue in the other incident post that's on the HN front page right now:
https://github.blog/news-insights/company-news/the-august-17...
I don't care about what they say because of who they sold out to
I feel like the article is a bit light on the design of the WAL and maintenance operations, but maybe it's me.
Anyway, I'd love to have a friendly chat with the author.
Doesn't 3PC require all nodes to agree, not just a majority?
IE - the interactive diagram and note about latency being bounded by the slowest replica
Ah, and then it looks like the losing nodes get marked unhealthy:
> In essence, every write operation goes through a voting protocol, and any replicas on the losing side of the vote are marked as unhealthy—unavailable for reads or writes—until they can be repaired. Repairs are automatic and quick. Because a majority agreed either to accept or to roll back the update, there are still at least two replicas available to continue accepting both reads and writes while the unhealthy replica is repaired.
But not sure how unhealthy nodes are discovered. Maybe the coordinator marks them, and if the coordinator crashes it allows stale reads and detects invalid writes?
The first step is to separate the problems. The content addressable object storage and storing the refs. libgit2 has two backends for those already.
Then use appropriate distributed dbs for the two backends. With large scale something like Cassandra for the objects (AP) and FoundationDB (CP) for the refs (or the equivalent Amazon services).
For performance the objects need to be cached locally and requests routed to nodes where the objects are cached locally.
A few scaling trends, all triggered by coding agents, make it unsustainable to keep stacking agents on the interfaces designed for human uses — Git, PRs, CI, review:
1. Time to produce a given feature dropped by orders of magnitude. Review, CI, and branch workflows were calibrated to human writing speed. The gates did not get faster.
2. Users will not consume 100x more features. They expect 100x more value from each interaction. The interesting artifact is no longer “the code that implements a feature.” It’s the mapping from intent to outcome.
3. Intermediate work exploded while durable output shrank. Agents leave scratch repos, traces, failed attempts, discarded implementations. The thing the user actually wanted got smaller and more valuable — closer to “satisfy this intent” than “ship this feature.” That is a lure to build much more complex software, and we are versioning the scrap heap with the same tool we use for the product.
4. The primary goal of software engineering till today is to share actions between users, so their intent are served through the same routines. Such software only satisfy commonality. It's now possible to build based on peculiarity. With that, a whole new space of personalization.
This looks like the history of machine tools: first powerful and imprecise, then more powerful and enormously precise. Software has to make the same jump — much more useful, and much more personalized.
I don’t know what the end state looks like. I doubt Git should remain the central mechanism.
Edit: Revised with help of AI.
Will free Git hosting continue to be sustainable, though, without architectural improvements like this?
But I believe it would be easy to fix with limitation for the free accounts (CI time limit, total size of repo limit etc.)
Cursor Origin Code Hosting
Not sure the last month GH had fewer than five minutes of downtime - but I think it's been a while
Unfortunately, Git is not as distributed as it claims to be.
I've been working on another project where I have binary files checked in, assets that move or are moved and then changed that git "loses track of" due to the significant change. This just may be me doing silly things like moving a file and then editing it and then committing it and git thinks I deleted it and created a new file.
They should have just failed to scale git and leave place for the less idiosyncratic bazaar or mercurial to take the lead. These were slower, but would probably have scaled much better in the long term.
Github's core innovation was re-centralizing a decentralized SCM (as per article) by giving it nice web UI and features such as PRs. Oh and also free code hosting space. None of which relied on git's peculiarities.
When I've worked at big companies they had bespoke SCM that resolved this. But i'm curious about mid size teams that are big enough to overwhelm git but not so that they can rewrite it.
The problems start when there is a lot of changes on one branch and usually the first band-aid is to put a commit queue in place. The few very big companies that need even more than that probably aren't going to put their code on a new git hosting provider tomorrow.
> Those down-voting don't understand why any sacrifice of convenience or tech is worth it.
FWIW my comment was actually highly upvoted but it looks like X employees and/or Musk fans showed up and downvoted all criticism in this thread.