Reduce memory consumption with a custom ActiveRecord attribute

Posted: Apr 11, 2021

In the team I work we try to see what can be improved in our projects before it hits us, performance is one of those topics. In general, I like to play with different profilers. Especially, we are on Heroku, thus if we can optimize something we can stay longer on a Performance-M plan (who really need 14 Gb of RAM on a Performance-L plan? I would like to see something in between, but Heroku's marketing team has another opinion).

Our project has one endpoint which gets called quite often. So, I profiled it with memory profiler and saw a line pointing to hstore.rb.

allocated objects by file
-----------------------------------
      2694  /usr/local/bundle/gems/activerecord-6.0.3.5/lib/active_record/connection_adapters/postgresql/oid/hstore.rb

Actually, it is a first line in this section, the total allocated memory is 1.49 Mb.

That seemed interesting. Whenever an hstore value is fetched from the DB, it gets parsed there, ActiveRecord gets rid of quoting. In our project Hstore is used mainly for keeping translations. So, every record keeps identical keys in the Hstore column, something like this:

"en"=>"English", "de"=>"Deutsch"

After scrolling down to an Allocated String Report section, I saw this

432  "de"

The de string was allocated 432 times.

If locale keys can be cached, we save some bytes in RAM. The first thought was to monkey patch Rails, but after exploring internals of ActiveRecord, I found that ActiveRecord supports custom types, I didn't know this feature of ActiveRecord. According to commits, this feature was added 6 years ago! So, I copy-pasted internals of ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Hstore and adjusted it to cache keys.

module Types
  class LocaleHstore < ActiveRecord::ConnectionAdapters::PostgreSQL::OID::Hstore
    def deserialize(value)
      if value.is_a?(::String)
        ::Hash[value.scan(HstorePair).map { |k, v|
          v = v.upcase == "NULL" ? nil : v.gsub(/\A"(.*)"\Z/m, '\1').gsub(/\\(.)/, '\1')
          k = parse_key(k)
          [k, v]
        }]
      else
        value
      end
    end

    def parse_key(key)
      @cache_key ||= {}

      return @cache_key[key] if @cache_key[key]

      parsed = key.gsub(/\A"(.*)"\Z/m, '\1').gsub(/\\(.)/, '\1')

      @cache_key[key] = parsed

      parsed
    end
  end
end

ActiveRecord::Type.register(:locale_hstore, Types::LocaleHstore)

Then added it to models:

class Product < ApplicationRecord
  attribute :title_i18n, :locale_hstore
end

There are more than one model keeping translations and some models have a few attributes with translations. So, the scope of this change is more than one model.

After this simple optimization "de" string was allocated 128 times, yeah, there are more parts to be reviewed, this number still looks too high. Although, the total memory consumption per request was reduced on 75,98 Kb.

Might be interesting too: