When the original storage index was designed, jsDelivr tracked roughly 350,000 unique files. By the time we replaced it, that number had grown to more than 600 million. We moved the index from Redis to MariaDB, migrated storage from Amazon S3 to Wasabi, compressed and cleaned up the stored data, and reduced the GitHub API work needed for uncached requests.
Most jsDelivr requests are served directly from CDN edge caches. The origin handles the less visible path behind them: resolving a package and version, retrieving or generating a file that has not been cached yet, storing the result, and returning it to the CDN.
This path matters most for newly published versions, less frequently requested files, GitHub branches and tags, and dynamically generated resources such as /+esm bundles. An origin problem can therefore affect cold or unusual requests while popular files continue to be served normally from cache.
Over the past few months, we completed two major groups of changes. The first rebuilt the metadata and storage layer used for cached files. The second reduced our dependence on GitHub API capacity when resolving GitHub-backed URLs.
A companion post covers the package-level /+esm improvements. This one is about the origin infrastructure underneath them.
From 350,000 to more than 600 million files
The original storage index was designed when jsDelivr had around 350,000 unique files.
The design anticipated substantial growth and was expected to remain practical at tens of millions of files – about one hundred times the initial scale. For a long time, it did.
The object-storage bucket itself was the source of truth. Redis kept a derived index for fast origin lookups. Once per day, the origin listed every stored file and rebuilt the complete index, while incremental updates kept it current between rebuilds.
That made consistency straightforward. If Redis lost data or missed an update, the bucket still represented the actual state, and the next complete listing reconstructed the index. At the scale we had then, both the listing load and the Redis memory requirements were reasonable.
By the time the index passed 50 million files, a full listing already took several hours. Other work took priority and the migration was postponed. When we returned to finish it, the bucket contained more than 600 million files.
Each rebuild now meant scanning hundreds of millions of files that had not changed, consuming substantial object-storage bandwidth, and keeping the complete index in memory on the origin side.
Replacing the in-memory index with MariaDB
The new implementation stores file metadata in one central MariaDB database.
For package files, records are keyed by the package source – npm or GitHub – together with the package or repository name, file path, and version. Additional fields identify generated variants such as minified or ESM files and keep the upload metadata needed to locate the physical file.
Normal origin requests use direct indexed queries. MariaDB can keep hundreds of millions of rows primarily on disk instead of requiring the entire working index to remain in memory.
We initially considered using a central database for writes and a local read replica on every origin. That would provide slightly lower read latency and keep local lookups available during a central outage, but it would also introduce replication lag, more components to operate, and periods when different origins could see different states. We ultimately chose a single central database and handle availability through the fallback path described below.
Keeping storage and metadata consistent
With MariaDB, the index becomes the source of truth for which files the origin considers available in storage.
The upload path writes the file first and creates the database record only after the upload succeeds. If a row exists, the corresponding file was successfully stored.
The reverse case is acceptable. A file upload may succeed and the following database insert may fail. Without the database row, the origin treats the file as uncached. The next request retrieves or generates it again and performs another upload.
The upload process attempts to remove the unindexed file after a database failure. If that cleanup also fails, the result is an orphaned file using some additional storage – not an index entry pointing to content that was never uploaded.
Database primary keys also prevent concurrent origins from indexing several copies of the same logical package file and version. If another origin completes the same write first, the later duplicate can be removed.
Falling back when the database is unavailable
If the origin cannot query the metadata database, it skips the object-storage lookup and fetches a fresh copy from the original npm or GitHub source.
The request may take longer and may repeat work for a file that is already stored, but the content can still be returned. This is the same path used for a package file jsDelivr has not cached before.
Migrating and shrinking the object store
Changing the index and storage layout required migrating the existing files, so we also used the opportunity to reduce storage volume and provider costs.
We moved the object store from Amazon S3 to Wasabi and Brotli-compressed every compressible file for which compression produced a smaller result. Existing files were recompressed during the migration, and new files now go through the same check before upload.
The migration also found around 40 million duplicate files created by problems in the previous origin upload process. They were additional physical copies of content already stored elsewhere and could be removed while the new index was being built.
Brotli compression and duplicate cleanup reduced the stored volume from roughly 32 TB to around 15 TB. Together with the move to Wasabi, these changes cut our storage and bandwidth costs by about 85%.
Using the GitHub API more carefully
A GitHub-backed jsDelivr URL may contain a semantic version, an exact tag, a branch name, a full commit hash, a shortened hash, or no explicit version at all. Before downloading the requested file or repository archive, the origin has to determine what that value represents.
The older resolution path fetched a repository’s complete tag list in more cases than necessary. On repositories with large histories, resolving one file could require many paginated GitHub API requests.
Full commit hashes, likely shortened hashes, well-known branches, gh-pages, and other values that clearly look like branches are now fetched directly. They do not require a tag listing first.
Requests that need semantic-version matching still use repository tags, but the listing is capped at 1,000 entries and cached for later requests. One cached list can resolve many versions of the same repository.
If the list is truncated and the requested value looks like an exact tag, the origin performs a targeted reference lookup. Exact older releases therefore remain accessible without downloading an unbounded tag history for every request.
When a request for latest falls back to a branch rather than resolving to a release, the origin follows GitHub’s HEAD reference instead of assuming that the default branch is named master.
Caching useful metadata
The order of GitHub lookups affects how much of the work can be reused.
A cached tag list can answer many later requests for the same repository. A sequence of exact-reference probes returning 404 provides much less useful information.
For ambiguous versions, the origin starts with the reusable tag metadata and only then falls back to targeted lookups. Missing versions in an existing repository can often be answered from the cached list instead of repeatedly probing individual references.
We also added monitoring and automated abuse detection for clients requesting unusually large numbers of unrelated repositories. The origin tracks distinct repositories by IP address and network range, with additional weight for repeated requests to repositories that do not exist.
These controls protect the GitHub API capacity shared by all jsDelivr users without changing normal use of GitHub-backed URLs.
Since deploying the combined resolution, caching, and monitoring changes, we have no longer seen the GitHub rate-limit errors that previously affected origin requests. We are also making fewer API calls overall, which reduces the load jsDelivr places on GitHub.
What this means for the origin
The origin no longer rebuilds an in-memory index by scanning more than 600 million files. It now uses MariaDB for indexed metadata, can fall back to the original source when the database is unavailable, and stores less than half the previous amount of data at a fraction of the cost.
The GitHub path also performs less work for common references and makes better use of metadata already in cache. For users, the practical effect should be better reliability on the requests where the origin matters most – new releases, uncommon files, and GitHub references that have not yet reached the CDN edge.