Examples Templates Store Pricing Docs Turbo CSS Blog Introducing Turbo UI Named buckets Turbo CSS is Tailwind CSS on steroids Introducing Turbo CSS, the most advanced web-design language Calculate how much storage you use Better login system Collaboration settings Filesystem and Database are not cutting the problem space right What I'm working on 64bit File node IDs New how-to videos Creating buttons gets easier Introducing reusable components HTTPS by default Introducing the Boomla Theme CDN for faster pageloads Write your own website builder on top of Boomla On On composition Shared admin access A fresh config editor Building a multi-purpose theme A filesystem to replace your CMS New file link type: scope Mobile editing support Inline file wrapping changed Package sandboxing New PHP-like JavaScript engine [sjs-4e] Send emails to the website owner New JavaScript engine [sjs-4] A better editing experience New email service provider Glossary and other changes New panel changes Improved registration flow Boomla goes multiplayer Using local dev tools Why Boomla doesn't need Git File Panel Let's build a community Automatic updates Improved sjs-3 API New Frontend CSS modules Work offline with Boomla Faster page loads via caching Drag & drop supercharged Supporting CommonJS modulesĀ  Paranoid about loosing data IDE usability improvements Simple App install flow Meetups in Budapest Goodbye broken links Flow control from user space Customizing apps Contextmenu support for apps Deprecating the .Class file Hello Changelog Embedding 3rd party plugins Introducing Tools Installing apps just got amazing Public beta Host on our servers Simple deploy with push/pull Version Control for the Web 350M files on a 1TB disk 2 weeks in review
Control Panel

64bit File node IDs

2020-10-19

This is a very technical update on the Boomla Platform, aimed at curious developers.

EDIT: originally, I went overboard and rolled out 128bit File node IDs. As it may have other performance implications and may in fact not be required, I decided to do a slow ramp-up instead and only use 64bit IDs for now. If that turns out to be not enough, we can still increase it later.

Old File node IDs

Until today, Boomla has been using 32bit incremental File node IDs, for example i102FD. This is just a special formatting of the number 66301. It starts with an i to avoid ambiguity, followed by the number expressed in hexadecimal form because it is somewhat shorter.

I intentionally said File node ID not File ID, because the two are different. File node IDs identify file nodes within volumes while File IDs identify files in filesystems. A filesystem may contain nested volumes so a File ID may contain one or more nested File node IDs.

An example (old) file ID could be f102FD.158A9. It starts with an f, followed by file node IDs in hex format, separated by dots.

Merging branches

This worked super well but Boomla has evolved to a point where it became necessary to merge branches. The problem is, with incremental file node IDs, merging is almost guaranteed to fail.

The reason is that File IDs are super important in Boomla, so they must be tracked throughout the version history. (For example, they are used for automatically redirecting visitors after a page was renamed.) But with incremental file node IDs, one will use the same IDs on every branch.

Let's look at an example. Say you have the website example.com with the biggest file node ID of i20000. You want to work on a new feature so you create the branch beta.example.com. You keep working on both branches, you create new files on each. The problem is, the first new file will get the ID i20001 on both branches! Because of this, merging will fail.

We need to guarantee that file nodes will get unique IDs on each branch, thereby avoiding merge conflicts.

Random File node IDs

To avoid this, we are going to use random file node IDs. Unfortunately, at this point using 32bit file node IDs will become too tight. At 32bit, the we can allocate ~4 billion IDs. On the other hand, it is said that one can only use the square root of it without a high probability of conflicts, which is only 65536 files. That's clearly a bit too tight, even when considering this applies to a single volume only, so an entire website having a tree of volumes could have way more than that.

Because of this, the size of File node IDs has been expanded to 64 bits. That's an insanely large number: 18.446.744.073.709.551.616. Even't its square root is 4.294.967.296. This means you can create billions of new files on separate branches without the likelihood of them conflicting. (Note: that's not a limit on the number of files but the number of new files on feature branches!) Thereby we have practically eliminated the chance of any file node ID conflicts.

An example File node ID (FnodeId) now looks like this:

iF78B0BDEC6305840

The equavalent File ID looks like this (note the f at the start):

fF78B0BDEC6305840

And an example nested File ID now looks like this:

fF78B0BDEC6305840.EC64D53884F1744D

Rolling out

We want to keep all existing file node IDs so the roll-out did not require any migration. New file nodes will get these large, random IDs while existing files will keep their old ones.

Merging is not publicly available yet but will be some time in the future.


Cheers,

you can follow me on Twitter