Replacing an Asset Pipeline

At Onfido, a lot of our major applications are built with Rails. For our assets, therefore, we have been using the Rails asset pipeline (Sprockets).

Sprockets is great for getting going very quickly — without much configuration you can start writing semi-modular Javascript and SCSS code, the compilation step taken care of. You can include images, and minification comes for free. Even asset fingerprinting occurs magically, with no more effort than enabling a flag.

Unfortunately, Sprockets also has some hidden drawbacks. As our projects grew in size, we began to really feel the following issues:

  • speed — Sprockets is very slow.
  • datedness — Sprockets has not incorporated CommonJS, leaving lots of packages in npm as out of reach. Equivalent gems (for example jQuery) tend to be very dated.

There are Rails solutions for supporting CommonJS code, but nothing that truly addresses the speed issue. We began to consider dropping Sprockets, but what else is there? Tools like Gulp are great, but very project-agnostic. While agnosticism works for the wider community, creating a gulpfile.js for our Rails-oriented applications to manually recreate Sprockets in every one of them would be duplicative and unnecessary. So what else is there?

Enter Pinion

We decided to build our own tool to simulate a lot of the core functionality of Sprockets. It leverages Gulp to define all of its tasks and keeps it quick and easy to extend.

How much faster are we talking? I'll go further into this a little later, but we're currently seeing our pipeline roughly doubling in speed.

It’s entirely open-source, and you can check it out on GitHub. Take a look through the README to get an idea of the tasks that it offers.

The Tablecloth Trick

Unfortunately, swapping pipelines mid-project isn’t as simple as changing one or two configuration flags. There are several issues that you might hit when trying to swap over to a more modern pipeline

Javascript require

As mentioned previously, Rails does not support CommonJS. Instead to load in other files, it offers a //= require method. This method inlines the relevant file in place of the comment, which is subtly (but significantly) different to how a CommonJS require() call works.

CommonJS enforces modularity of each file, meaning that globals do not persist outside of a file, without an explicit window.x = or global.x = call. Rails, on the other hand, encourages global-happy scripting. Hunting these globals and replacing them with modular exports can be quite tedious.

Javascript and SCSS strictness

Sprockets is very lax with its adherence to Javascript and SCSS warnings. The biggest offenders we discovered in our codebase after swapping to Pinion’s much stricter environment were missing var declarations in Javascript, and missing _ prefixes for partial SCSS file names.

Seamless dev swap

As mentioned in the Pinion README, we included some helpers to mimic the default behaviour of asset_path etc.

But on top of this, we needed a way for developers focusing on Rails work to not even have to notice that they had swapped asset pipeline. For us, this involved adding in some dusty scripts to automatically boot up a $(npm bin)/pinion instance whenever a container wakes up.

This will obviously change depending on your workspace and requirements.

Gems

Lots of gems provide assets and some extra functionality for assets. These can’t just be pulled out, and instead need to be replaced with similar code performing the same functionality. This has been the hardest step for us.

Goodbye Sprockets?

Getting rid of these gems has been such a fiddly task that, in fact, we haven’t entirely finished yet. So, how are we using Pinion with these gems? Well, it’s a little messy.

First, it’s useful to look at a diagram of what an asset pipeline is from a high level.

In this case, we take our static assets and run them through the pipeline. The assets will either be raw source, or references to external sources that it wants the pipeline to bring in (through some form of package manager).

But there’s no reason to only have one pipe. At Onfido, we currently have two isolated pipes that make up our asset pipeline.

As well as Pinion, we are using Sprockets to build a subset of our assets folder. This subset contains gems that run through Sprockets, providing certain assets to the application. We strictly enforce that new source code written should be run through the Pinion pipeline, rather than Sprockets.

As we replace these gems with equivalent code, we slowly bring ourselves to where we want to be:

Speed

So how much faster actually is Pinion? It's hard for us to give an exact number, being split between Pinion and Sprockets at the moment — but we can compare our double-pipeline speed, to our previous Sprockets speed.

The old Sprockets pipeline yielded us this speed...

time RAILS_ENV=production bundle exec rake assets:precompile  
# ...
real    2m54.564s  
user    2m21.140s  
sys     0m17.420s  

The new Pinion+Sprockets pipeline currently yields us this speed...

(time NODE_ENV=production $(npm bin)/pinion build rev &) && (time RAILS_ENV=production bundle exec rake assets:precompile &)
# ...
# Pinion finishes
real    0m52.285s  
user    0m38.620s  
sys    0m5.120s  
# ...
# Sprockets finishes
real    1m30.985s  
user    0m43.770s  
sys    0m18.100s  

So already we're seeing our pipeline running at 192% of its previous speed. And we hope to see that number continue to rise as we deprecate Sprockets fully!

Conclusions

We’re still in a transition period at the moment, but this is certainly something worth thinking about quite closely before making the switch.

Unless you’re starting a green project, swapping over from Sprockets to Pinion can be quite a serious undertaking — whether or not it’s worth the end result will depend on your workspace requirements.

But please check out Pinion if you’re interested, and contribute if you like it!

Author image
Stephen is a front-end software engineer at Onfido. When not at the office, Stephen can be found neglecting friends and family to focus on his Mario Kart career.
top