chris.heald.me

Domains, DNS, and SSL, oh my!

Thu, 20 Jan 2022 00:00:00 -0600

I recently helped a friend navigate the process of untangling himself from his registrar’s expensive SSL offerings, and wrote up the following to help illustrate what the pieces are and how they fit together. He asked that I publish this somewhere that others could benefit from it, so here it is!

Registrars, DNS, Servers, and SSL.

Registrars are gatekeepers for registering “names”, which are subdomains on top-level domains (TLDs). There’s something called the “internet root nameservers” which are the bottom of the turtle stack. They say “this so and so can answer queries for this particular TLD” like .com and .net and .mobile and whatever. There are 13 root servers on the internet (actually served by hundreds of machines, but logically, there are 13), and they currently serve about 730 TLDs. All the software in the world that uses DNS knows about these 13 root servers (or delegates that knowledge). This is known as the DNS Root Zone.

The root servers are administrated by ICANN, and Domain Name Registrars (like GoDaddy and Namecheap) contract with ICANN to provide domain registration services, so Namecheap says “I want to be able to sell domains on the .com TLD”, they pay ICANN, ICANN lets the root servers know about Namecheap as a valid resolver of .com domains.

Registrars’ primary function is to sell registration of domains on the TLDs they contract on. They may provide other services, but fundamentally you pay them money, they agree to resolve DNS lookups for NS (nameserver) queries for yourname.com to a nameserver of your choosing to figure out where it should actually go.

This is the point at which your NECESSARY engagement with them is at an end. Most registrars sell a bunch of other services, like hosting, SSL, and mail. None of this is required (but may be convenient, if often overpriced).
Once I, as a requesting client, have discovered the nameserver for a domain, I send a query to that nameserver for the record I want. If you want to discover the webserver IP address for a name, you send a query for an A record. If you want the mailserver, you send a query for the MX record. So on and so forth.

The nameserver responds with an IP address to the querying client.
The client can make a request of the kind and shape of its choosing to the IP address. If you know the IP address, you don’t have to go through the whole rigamarole, but nobody wants to hand out “please visit 152.43.20.221/~mysite/public/ for more information” on their business cards.
So, SSL. SSL is used during step 3, usually by clients making HTTP requests (which are used for serving web pages). It’s a way to validate that the IP you’re talking to actually is authorized to respond to requests for yourname.com, and it allows for the encryption of traffic in a way that both ends can understand. Clients send a request to an IP, but they also include a Host: yourname.com header which indicates “this request was made because a client wanted yourname.com”, which lets the server do some routing (in case multiple domains are hosted on one IP) and lets it decide what cert to respond with (the one that matches the request for that domain)
To validate an SSL certificate, your client has to know about a certificate authority that has signed the certificate you receive from the server. CAs can be thought of as notaries public. Browsers come with a list of trusted CAs (LetsEncrypt is one). If a random website hands you a cert, you have no idea if they made it up or not. But if the cert you receive for a given domain is cryptographically signed as valid by a CA (notary) you trust, then you can trust that the certificate was issued by someone who owns that domain, because the CA trusts that they own the domain. Many domain name registrars also provide CA services (they get themselves registered as trusted notaries for signing certs).
To get an SSL cert that’s signed as valid by a CA, I therefore have to prove to the CA operator that I do, in fact, have permission to serve traffic on yourname.com. How this proof is established is up to the CA. Some registars do that by requiring that you host your domain registration with them, so they know by lazy virtue of the fact that you’re logged in and that you registered yourname.com that you do in fact have permissions on that name, but there are other ways, too.

By contrast, LetsEncrypt lets you prove ownership by either
1. making your website answer to a random challenge they provide, or
2. making your DNS answer to a random challenge they provide
Both prove you have control over the domain. This is why it works anywhere with any host, though, because as long as you can provide those challenges you can prove ownership and get your certs signed by them.
The SSL certificate is basically just a binary blob which can be read by computers as a cryptographically-signed claim of “this certificate is for this domain and is good through this date”. Once you HAVE the cert you can install and use it anywhere you want (as along as the name you’re serving under matches what’s on the cert). You don’t have to use a registrar’s hosting services, you don’t have to use their mail services, you can serve that binary blob from any server you want. As long as the client’s expectation (I requested yourname.com and got a certificate for yourname.com) is met, and the certificate is signed as valid by a CA the client trusts, you’re good to go.

A Practical Example

I register my domains through Namecheap, and I tell Namecheap “use these DigitalOcean nameservers for my names”.

$ whois heald.me | grep Name
Domain Name: HEALD.ME
Registrar: NameCheap, Inc.
Name Server: NS1.DIGITALOCEAN.COM
Name Server: NS2.DIGITALOCEAN.COM
Name Server: NS3.DIGITALOCEAN.COM

I configure my DNS at DigitalOcean, which publishes the records I specify on their nameservers. These records include things like:

heald.me.        300  IN      TXT     "v=spf1 include:_spf.protonmail.ch mx ~all"
chris.heald.me.  300  IN      A       178.128.3.212

My DNS is configured to point my names to my DigitalOcean servers (178.128.3.212).

When my client goes to look up my DNS, it goes to the root servers (or to a server that has information from the root servers cached) and figures out where we’re going. Here’s a full verbose trace of a query for “the A record for chris.heald.me”.

dig IN A heald.me +trace

; <<>> DiG 9.16.15-Ubuntu <<>> IN A heald.me +trace
;; global options: +cmd
.                       12681   IN      NS      f.root-servers.net.
.                       12681   IN      NS      g.root-servers.net.
.                       12681   IN      NS      h.root-servers.net.
.                       12681   IN      NS      i.root-servers.net.
.                       12681   IN      NS      j.root-servers.net.
.                       12681   IN      NS      k.root-servers.net.
.                       12681   IN      NS      l.root-servers.net.
.                       12681   IN      NS      m.root-servers.net.
.                       12681   IN      NS      a.root-servers.net.
.                       12681   IN      NS      b.root-servers.net.
.                       12681   IN      NS      c.root-servers.net.
.                       12681   IN      NS      d.root-servers.net.
.                       12681   IN      NS      e.root-servers.net.
;; Received 525 bytes from 192.168.4.248#53(192.168.4.248) in 63 ms

me.                     172800  IN      NS      a0.nic.me.
me.                     172800  IN      NS      a2.nic.me.
me.                     172800  IN      NS      b0.nic.me.
me.                     172800  IN      NS      b2.nic.me.
me.                     172800  IN      NS      c0.nic.me.
me.                     86400   IN      DS      45352 8 2
;; Received 681 bytes from 192.203.230.10#53(e.root-servers.net) in 9 ms

Here’s the root server query. My computer talks to my upstream DNS servers, which are preconfigured to know about the root servers. It went to the root servers and said “I need to know where to find information about .me domains”. It learns that this a set of servers at nic.me.

These records exist because nic.me is contracted with ICANN to provide .me registrations.

heald.me.               86400   IN      NS      ns2.digitalocean.com.
heald.me.               86400   IN      NS      ns3.digitalocean.com.
heald.me.               86400   IN      NS      ns1.digitalocean.com.
;; Received 606 bytes from 199.253.60.1#53(b0.nic.me) in 206 ms

It went to b0.nic.me and asked for where to find nameservers for heald.me. b0.nic.me said “Oh, that’s these DigitalOcean servers”.

These records exist because Namecheap is contracted with nic.me and was permitted to write the records for heald.me after I bought the domain through them.

chris.heald.me.         300     IN      A       178.128.3.212
;; Received 53 bytes from 173.245.59.41#53(ns2.digitalocean.com) in 31 ms

Finally, it went to ns.digitalocean.com and said “I need the A record for chris.heald.me” and ns2.digitalocean.com responded “That’s 178.128.3.212.

These records exist because I instructed DigitalOcean to serve them for the domain heald.me via their nameservers.

My DigitalOcean servers are running nginx, which I’ve configured to automatically provision SSL certificates for my names from LetsEncrypt, and which respond to challenges by LetsEncrypt by creating an arbitrary challenge file on demand to be fetched via an HTTP request.

Requests for chris.heald.me go to the root name servers to find out who can serve .me, then delegates down the chain to find who registered heald.me. Namecheap says “I’ve got that one! Go to ns1.digitalocean.com to find out more”.

Your client goes to ns1.digitalocean.com and says “I want the A record for chris.heald.me”, and DO says “that’s 178.128.3.212”.

Your client connects to 178.128.3.212 on port 443 (which is the port for HTTPS by convention) and says “This is an HTTPS request for Host: chris.heald.me”.

My server responds with “Here’s my certificate for *.heald.me”. Your client says “Yup, it’s signed by LetsEncrypt, and I trust LetsEncrypt, so I trust you. And chris.heald.me matches what I asked for, so we can go ahead. Give me what you have for Path: /”

My server responds with HTML content and your browser displays it. Et voila!

Secure DNS-based ad filtering with Pihole and Stubby via Docker

Thu, 24 Oct 2019 00:00:00 -0500

I’ve been running a Pihole DNS server internally for a while. It’s great! It’s like adblock, except for your whole network - all your devices, TVs, phones, tablets, and computers - get ad filtering.

Recently I wanted to see if I could get my Pihole to play nice with Stubby. With the imminent advent of DNS-over-HTTPS and the subsequent ISP freakout over the loss of their sweet, sweet data mining, I figured now was as good a time as any to add an extra layer to my network.

I use and love Docker. I’ve worked up this simple Bash script which will wire up Stubby (pointed to Cloudflare’s DNS-over-TLS servers), and then point the Pihole to Stubby. This means that the Pihole does all its name resolution away from the prying eyes of my ISP (yes, it gives it to Cloudflare, though Cloudflare has a better stance on privacy than most ISPs), and I get DNS filtered. Best of both worlds!

Here’s the script. I run it on an Ubuntu 16.04 server, but you should be able to run it on anything that can run Docker.

#!/bin/bash
if [ "$#" -ne 1 ]; then
    echo "Usage: $0 123.123.123.123 password"
    echo "Specify your host's LAN IP and the desired Pihole password"
    exit 1
fi

IP=$1
PASSWORD=$2

docker run -d -it --restart always --name stubby -p 8053:8053/udp mvance/stubby:latest
# We need a network for the pihole to be able to talk to stubby
docker network create --subnet 172.18.0.0/16 dns
# Attach stubby to the network, specifying the IP it should use
docker network connect dns stubby --ip 172.18.0.3

docker run -d \
    --name pihole \
    -p $IP:53:53/tcp -p $IP:53:53/udp \
    -p 67:67/udp \
    -p 80:80 \
    -p 443:443 \
    -v pihole:/etc/pihole/ \
    -v dnsmasq.d:/etc/dnsmasq.d/ \
    -e ServerIP="${IP}" \
    -e DNSMASQ_LISTENING=${IP} \
    -e WEBPASSWORD="${PASSWORD}" \
    --restart=unless-stopped \
    --cap-add=NET_ADMIN \
    --dns=172.18.0.3 \
    pihole/pihole:latest

echo -n "Your password for https://${IP}/admin/ is "
docker logs pihole 2> /dev/null | grep 'password:'
# Attach pihole to the network, specifying the IP it should use
docker network connect dns pihole --ip 172.18.0.2

You can go to http://your-lan-ip/admin to log into the Pihole dashboard (and you should, it’s neat), or just resolve a request against it. Try this:

export LAN_IP=123.123.123.123 # replace 123.123.123.123 with your server's LAN IP

dig @$LAN_IP google.com                   # normal request - should succeed
dig @$LAN_IP analytics.google.com         # pihole-filtered request - should fail
dig @$LAN_IP sigok.verteiltesysteme.net   # dnssec test - should succeed
dig @$LAN_IP sigfail.verteiltesysteme.net # dnssec test - should fail

You could do this with a docker-compose file pretty easily, too, but I appreciate the straightforwardness of Bash scripts. :)

And that’s it! Just update your DHCP server (probably your router) to hand out your LAN IP as the DNS server for devices on your network, and DNS on your network is secured and you’ll notice a lot of ads, trackers, and other annoyances no longer exist.

Docker & Default Routes

Sat, 29 Sep 2018 00:00:00 -0500

Docker has some surprising behavior when you have a container attached to multiple networks. If you’re not aware, it can cause some really bizarre network issues.

TL;DR

If you have multiple networks attached to your container, make sure your first network lexically sorts before the others you attach, or your default gateway will change, which breaks all kinds of things.

The Long Version

Lately I’ve been working on blue-green deployment processes with Docker. It’s a simple enough idea: bring up a container, make sure it’s healthy, and then turn on network routing for it. This is especially important when “container up” doesn’t mean “application in the container is ready” (basically any Rails or Java app, for example).

The basic structure of my setup is:

nginx-proxy on each Docker host runs on a known port, and the LB proxies to it
Containers are brought up with a VIRTUAL_HOST environment variable, which instructs nginx-proxy to rewrite its nginx config and reload, so it can now route to the new container.

This is a problem when the container isn’t quite ready, because nginx can be reloaded before the application is answering. This results in at least 1 request stalling failing, and depending on upstream nginx configs, can end up blacklisting the entire backend (or docker host, in the LB’s case) for a timeout period.

So, to solve this, I’m using the typical back/front network setup:

nginx-proxy is attached to the docker bridge network.
nginx-proxy is also attached to each application’s front network, app_stage.
Each container is brought up in its back network, app_stage_internal and health checked.
Once the application is healthy, the container is attached to the front network, app_stage.
We kick off the nginx config scan/rewrite/reload, and nginx should start routing traffic.

Not that hard, right? Except…well, it didn’t work. The first few requests after attaching the front network to the container would stall. If I had open long-running connections (Redis and Websockets were the biggest offenders), then they just remained open but never received any data. They were connected, but effectively dead. What on earth was going on? If I used just one network, it wasn’t a problem. I’d begun to wonder if Docker’s network connect mechanism was broken in some way that either everyone knew about and didn’t talk about, or if I just had a broken setup. I couldn’t find any reference to anyone else having any problem like this.

I finally stumbled into the solution using netshoot to inspect every metric I could think of. I noticed that when I’d bring up my containers with this mechanism, I’d get something like:

 [1] 🐳  → route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.24.34.1     0.0.0.0         UG    0      0        0 eth0
172.24.34.0     *               255.255.255.0   U     0      0        0 eth0

And then when the second network was attached:

 [1] 🐳  → route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         172.24.35.1     0.0.0.0         UG    0      0        0 eth1
172.24.34.0     *               255.255.255.0   U     0      0        0 eth0
172.24.35.0     *               255.255.255.0   U     0      0        0 eth1

Wait, what? Why was the default gateway changing after connecting a second NIC?

It turns out the answer is right there in the documentation:

You can connect and disconnect running containers from networks without restarting the container. When a container is connected to multiple networks, its external connectivity is provided via the first non-internal network, in lexical order.

Wow, okay. Your default gateway becomes whichever network appears first in a list sorted by name. Yikes.

The fix was easy enough. Rather than using app_stage and app_stage_internal, I just switched my network naming convention to BACK_app_stage and FRONT_app_stage. That way, the front networks always sort after the back networks and don’t take over as the default gateway when a new network is attached. It’d also work to make the back network internal, but since my containers do occassionally depend on external network connectivity to come up (I’m looking at you, Passenger), my team needs back networks to be able to get to internet.

I did find this issue which describes the problem (and asks for a fix), but it’s been open for about 2.5 years now. A better fix would be some way to specify the metric of the new interface during docker network connect, but I can’t find any way to do that, either. A third option would be to just use ip in the container to rejigger the default route, but that’s also a no-go since the ip tool may not be available in every container.

Upgrading legacy Ruby & Rails installs to support TLS 1.2

Mon, 25 Jun 2018 00:00:00 -0500

I recently had a legacy machine running an ancient Ruby 1.8.7 app which I needed to upgrade to support TLS 1.2. A few bespoke requirements made it infeasible to upgrade the core OS to a more modern version, but fortunately, it was relatively straightforward to get it back to a servicable state. While it would be vastly preferable to rebuild the application for a more modern stack, it wasn’t feasible in this case, so this is the next-best option.

The biggest issue with upgrading these kinds of machines is the ancient OpenSSL versions linked against tools like curl and wget, which can make it difficult to retrieve files from remote locations for upgrading! Our path forward is SCP.

The tl;dr here is that we’re going to:

scp tarballs of the openssl and curl sources to the target machine
Build and install them, with Curl pointed to our newly-installed OpenSSL
Use the upgraded Curl to bootstrap (or ugprade RVM)
Use RVM to install an RVM-specific OpenSSL
Use RVM to install Ruby with our new RVM OpenSSL install
Custom-compile the Passenger agent.

Let’s get started.

Replacing Curl

Curl’s a fundamental tool, one that we take for granted far too often. Old versions which only support TLS 1.0 can’t communicate with large parts of the web now, so our first order of business is to get a working communications line out to the rest of the web.

I grabbed the OpenSSL 1.0.2o tarball and the curl 7.6.0 tarball locally, then scp’d them up to the target machine:

scp curl-7.60.0.tar.gz user@target:~
scp openssl-1.0.2o.tar.gz user@target:~

Then on my target machine, as root:

mv /home/user/curl-7.60.0.tar.gz /usr/local/src
mv /home/user/openssl-1.0.2o.tar.gz /usr/local/src
cd /usr/local/src
tar -xzf curl-7.60.0.tar.gz
tar -xzf openssl-1.0.2o.tar.gz

Time to compile OpenSSL:

cd openssl-1.0.2o
./configure --prefix=/opt/openssl-1.0.2o
make && make install

And Curl:

cd ../curl-7.60.0
PKG_CONFIG_PATH=/opt/openssl-1.0.2o/lib/pkgconfig ./configure
make && make install

Verify that we have the right version:

$ curl -V
curl 7.60.0 (i686-pc-linux-gnu) libcurl/7.60.0 OpenSSL/1.0.2o zlib/1.2.3
Release-Date: 2018-05-16
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy

Great! We have a working lifeline to the outside world. Let’s make sure we can speak TLS 1.2:

$ curl --tlsv1.2 -I https://google.com/
HTTP/1.1 301 Moved Permanently

Success!

Getting and installing RVM

Now that we have Curl, we can install RVM via the usual mechanism:

\curl -sSL https://get.rvm.io | bash

Relog your session, and then you have RVM ready:

rvm pkg install openssl

This will custom-compile OpenSSL for RVM to use with the Ruby versions it installs. Then we just install Ruby:

rvm install ree --with-openssl-dir=/usr/local/rvm/usr

And test it:

rvm use ree
ruby -ropenssl -e 'puts OpenSSL::OPENSSL_VERSION'
OpenSSL 1.0.1i 6 Aug 2014

Passenger

Finally, we needed a updated passenger install. This is slightly fiddly because:

Newer gem versions won’t work on Ruby 1.8.7
Ruby 1.8.7 ships with a version of Rubygems that doesn’t realize that
Passenger on Ruby 1.8.7 won’t boot up apps with newer versions of Rubygems.

We’ll work around this by doing the install with a newer Rubygems, then downgrading it for compatibility.

First, upgrade Rubygems:

gem install rubygems-update
update_rubygems

Then install Passenger:

gem install passenger -v 5.3.2

We’ll need to custom-compile Passenger’s agent against our custom OpenSSL version:

EXTRA_PRE_LDFLAGS="-L/usr/local/rvm/lib" EXTRA_CXXFLAGS="-I/usr/local/rvm/include" EXTRA_CFLAGS="-I/usr/local/rvm/include" passenger-install-nginx-module

Finally, downgrade Rubygems:

rvm rubygems latest-1.8

Restart nginx and you should be up and running.

faster_pathname: Making Sprockets faster

Sun, 09 Feb 2014 00:00:00 -0600

Late last year, I was working on replacing the Sprockets pipeline for our internal developers with a Guard-based solution, as a means of improving the speed of change/reload/test cycles we’re all so familiar with. Using Guard to do our asset rebuilds was substantially faster, but I found that when I used Sprockets to do asset lookups, things got a lot slower. This led to me digging around a bit and finding that Pathname is excrutiatingly slow, and Sprockets leans on it really heavily.

The Sprockets team decided to not fix the issue, but to instead just rely on Pathname less in the future. That’s good, but it doesn’t solve the problem that Sprockets is costing us many thousands of developer-hours right now. So I threw together a gem to fix it, instead.

faster_pathname is a gem that just monkeypatches Pathname with faster-than-default behavior. It’s primarily targeted at alleviating the issues that Sprockets tends to have, but it should result in a general performance improvement for anything that uses Pathname. I would have preferred that Sockets use a solution that subclassed Pathname with customized behavior, but since Sprockets doesn’t seem to be changing any time soon, this is a quick path to a solution.

You can get the source at GitHub. It’s very simple - just a couple of monkeypatches, and an isolated copy of the Ruby 1.9 and 2.0 pathname test suites. In my tests, it sped up Sprockets asset lookups by around 25%, and has a very noticable impact on page load times in development environments.

Please note that it does not currently pass the 1.8 pathname tests, but I’ve not yet investigated that as 1.8 is officially deprecated, and I’m not a big fan of writing new software for deprecated platforms. The build currently fails on JRuby, but its test failures are consistent with stock JRuby’s test failures, so I’m confident in saying that it’s equivalently functional there, too. We’ve been running these patches (in non-gem form) across our dev team for a few months now, and have been very happy with the results.

Try it out and let me know how it works for you.

No, Rails' CookieStore isn't broken

Thu, 26 Sep 2013 00:00:00 -0500

A post recently hit the Full Disclosure seclist titled “Move away from CookieStore if you care about your users and their security”. The post discusses a property of session cookies - notably, that hitting “logout” doesn’t prevent a cookie from being reused to regain that session later, since if someone manages to jack one of your users’ cookies, they can just replay that cookie again at any time and gain access to the users’ account.

It’s worth first noting that this vulnerability requires your user’s session cookies to be compromised in the first place, so the whole vulnerability hinges on with “if your user is already owned, then…”

That said, yes, if you use sessions naively, then a compromised cookie may be used to gain access to a user’s account so long as the application’s session secret hasn’t changed (thereby invalidating the cookie signature). Note that this is true for all session stores, though in the case of serverside sessions, this only holds until the session gets swept (which may happen on explicit logout, but does not necessarily happen at a defined time otherwise); presuming you have some kind of session TTL in play, an attacker could keep their jacked session ID active indefinitely there, as well. A hijacked session cookie is Bad News (which is why you should be using HTTPS and HTTPS-only cookies!) no matter how you slice it.

Fortunately, if you’re worried about this class of attack, mitigating it is Pretty Darn Simple.

If you just want parity with serverside session stores that just perform expired session sweeps, then you can enforce a TTL on a session by just providing a TTL value in the session, and validating that when the session is read, then updating it when the session is written. You could do this trivially with a Rack middleware, or if you just want it in your app:

class ApplicationController
  before_filter :validate_session_timestamp
  after_filter  :persist_session_timestamp

  SESSION_TTL = 48.hours
  def validate_session_timestamp
    if user? && session.key?(:ttl) && session[:ttl] < SESSION_TTL.ago
      reset_session
      current_user = nil
      redirect_to login_path
    end
  end

  def persist_session_timestamp
    session[:ttl] = Time.now if user?
  end
end

That’s it. Any session that hasn’t been touched in 48 hours won’t validate and will get tossed out, same as serverside sessions (and as a bonus, you don’t have to do any session sweeping yourself! Hooray!) This does leave the cookie vulerable to TTL refreshes, so perhaps you want something more robust.

Something that neither CookieStore or server-side stores can do by default is maintain a list of sessions associated with a given user, and provide the user a means to revoke access granted to previously-granted sessions. Consider the case where you walk away from a public computer having forgotten to hit “log out” - you have no means of invalidating that session from another computer. This is a problem!

Fortunately, it’s trivial enough to just save a list of active sessions if desired:

class User
  # Presume an active_sessions field on the model that is large enough to hold some list of sessions:
  serialize :active_sessions, Array

  def activate_session(id)
    active_sessions.push id unless active_sessions.include? id
    save
  end

  def deactivate_session(id)
    active_sessions.delete id
    save
  end
end

class SessionsController
  def login
    # ...
    current_user.activate_session session[:session_id]
  end

  def logout
    current_user.deactivate_session session[:session_id]
    reset_session
    # ...
  end
end

class ApplicationController
  before_filter :validate_active_session

  def validate_active_session
    if user? && !current_user.active_sessions.include? session[:session_id]
      reset_session
      redirect_to login_path
    end
  end
end

You could get more complex with this and save things like the last IP and geolocation that each session was active from, and present that to the user, GMail-style. You could enforce a maximum number of sessions that can be active at any given time. This makes it easy to let users log out other sessions:

class User
  def expire_sessions!(active)
    self.active_sessions = [active]
    save
  end
end

class UserController
  def logout_other_sessions
    current_user.expire_sessions! session[:session_id]
    # Redirect or whatever
  end
end

This technique is applicable to all session stores, not just CookieStore (you won’t be able to get a list of sessions for a given user ID by default in ActiveRecordStore or whatever other serverside store you might want to use). You can give your users proactive control over their account security, and keep using CookieStore with all its benefits (like being invulnerable to session fixation!)

jquery-migrate and XSS

Mon, 26 Aug 2013 00:00:00 -0500

We were recently flagged by a security researcher for an active XSS hole in our site. After bisecting the origin of the hole to the introduction of jquery-migrate, I put together a minimal proof-of-concept for it and spoke to Dave Methvin on the jQuery team about it. He told me that this was not, in fact, a bug, but was working as intended. To that end, I’m publishing this to warn people about the danger of jquery-migrate’s divergent approach to this issue, so that you can be extra sure to sanitize your jQuery selectors.

The proof of concept is as follows:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="http://code.jquery.com/jquery-migrate-1.2.1.js"></script>
<script>
$(function() {
  x = $("a[href='" + window.location.hash + "']")
  console.log(x)
})
</script>

(This particular example is a simplification of how we were persisting in-page tab selection state in a few places; I suspect it’s a semi-common use case)

You can invoke it via something like:

http://example.com/xss-poc.html#<img src=x onerror=prompt(1)>

Or you can just invoke, from a console:

node = $("a[href='#<img src=x onerror=prompt(1) />']")

With just bare jquery, this returns an empty set [] - no nodes matched the selector. When jquery-migrate is present, it constructs nodes in a documentFragment (thus letting the img’s onerror fire, even if you never attach the node to the DOM), then returns the nodes, resulting in:

[<img src="x" onerror="prompt(1)">]

The Issue

The basic problem here is that jQuery has this ambiguous syntax wherein you can actually select nodes in a document:

$("foo")

Or you can create nodes:

$("<div id='foo'></div>")

In both cases, the $ function is used, and the operation is inferred by the data passed to it. jquery-migrate is doing something that breaks this inference and causes jQuery to attempt to construct a node rather than applying it as a selector in the example given:

$("a[href='<img src=x onerror=prompt(1) />']")

I was unable to reproduce this exploit using only jQuery 2.0.3, 1.10.2, 1.9.1, 1.8.3, 1.7.2, or 1.6.4. However, adding jquery-migrate to any of these immediately resulted in the application becoming vulnerable to XSS, except under 1.6.4 and 1.7.2. As far as I can tell, this is because the jQuery team wants something like this to work:

var node = $("bare text and node <img src=x />")

and jquery-migrate somehow breaks something that causes it to misinterpret the a selector as a node construction. Given that this is divergent from the current jQuery core behavior, I’m not quite sure how it’s working as intended, but I’ve been assured that it is, so I feel that the best course of action is to educate people on the divergent behavior.

Anyhow, all this is to say “jquery-migrate might punch XSS holes that you thought were closed”. Use with caution. And, as always, sanitize your inputs; assuming that a library that is not specifically responsible for sanitization is doing the right thing may end up leaving you in the lurch when they change something and stop sanitizing things that were previously sanitized.

Instrumenting Rails applications with ruby-prof

Fri, 02 Aug 2013 00:00:00 -0500

I recently read a blog post talking about a performance tuning tool called tracer_bullet.

There’s already a great tool for doing this work, though, called ruby-prof. It’s a C extension, so it isn’t going to work in JRuby (but JRuby already has great profiling tools available), but for people on MRI, it’ll work flawlessly.

Arbitrary Profiling

Profiling arbitrary Ruby code is as easy as something like this:

result = RubyProf.profile {
  # Some slow code here
}
open("callgrind.profile", "w") do |f|
  RubyProf::CallTreePrinter.new(result).print(f, :min_percent => 1)
end

This means that it’s pretty easy to abstract into a helper:

class ApplicationController
  def profile(prefix = "profile")
    result = RubyProf.profile { yield }

    dir = File.join(Rails.root, "tmp", "performance", params[:controller].parameterize
    FileUtils.mkdir_p(dir)
    file = File.join(dir, "callgrind.%s.%s.%s" % [prefix.parameterize, params[:action].parameterize, Time.now.to_s.parameterize] )
    open(file, "w") {|f| RubyProf::CallTreePrinter.new(result).print(f, :min_percent => 1) }
  end

  helper_method :profile
end

And then call it from your code (anywhere, but let’s demo a view)

<% profile "myview" do %>
  # Some code here
<% end %>

This will generate the appropriate callgrind file in your tmp/performance directory.

Request Profiling

So, extending this to full requests should be pretty straightforward. You could just do it in an around_filter, but there’s already a solution for you. This pattern is already wrapped up in request_profiler as a rack middleware. All you have to do is include it in your Gemfile:

gem 'request_profiler', :git => "git://github.com/justinweiss/request_profiler.git"

Then you can profile a request just by adding a query parameter:

http://foo.com/bar/baz?profile_request=true

This will generate a callgrind dump in tmp/performance, which you can then pull up with kcachegrind/qcachegrind.

Reading a dump

So now that you have your traces, what do you do with them? You probably want kcachegrind (for Windows/Linux) or qcachegrind (for OS X, install via homebrew).

To demo, I’m using publify, because it’s the first open source Rails app I could find. I know nothing about it. I just cloned and installed it:

git clone https://github.com/fdv/publify
cd publify
mv config/database.yml.sqlite config/database.yml

Added request_profile and installed:

echo "gem 'request_profiler', :git => 'git://github.com/cheald/request_profiler.git'" >> Gemfile
bundle install

Add the middleware to config/environments/development.rb:

config.middleware.insert 0, "Rack::RequestProfiler", :printer => ::RubyProf::CallTreePrinter

And start the server:

rails s

Logged in, and it’s time to profile!

http://192.168.4.139:3000/admin?profile_request=true

This generates a callgrind file for us. Opening it in KCachegrind gives us something like this:

First things first, in the bottom, you’ll notice Total process_time cost: 420 198. This is in microseconds, which means that overall, this page took 420 milliseconds to run.

Second, I’ve sorted by the Self column. This shows me which functions the action spent the most time in. Incl. means the function plus its children. You’ll use both to track down slow bits.

So, at the top here, you see REXML::Attributes#get_attribute. It took almost 26ms, and its total runtime was 134ms, or about 32% of the total page runtime. I happen to know that REXML is slow and that you should always use Nokogiri instead because it is fast and awesome rather than slow and terrible (REXML was responsible for me almost completely writing off Ruby as a language, but that’s a story for another day).

Clicking on the offending method gives me two intersting things here. I’m looking at the All Callees view, sorted by Distance. This is kind of like a stack trace, except that it includes all possible paths to this function. This is useful because it lets us figure out where in the application code this library code was invoked from. As you can see, I’ve highlighted the nearest application code to the method we’re looking at, which is Admin::DashboardController#parse_rss.

I can find that in the source code easily. It’s pretty standard REXML:

def parse_rss(body)
  xml = REXML::Document.new(body)

  items        = []
  link         = REXML::XPath.match(xml, "//channel/link/text()").first.value rescue ""
  title        = REXML::XPath.match(xml, "//channel/title/text()").first.value rescue ""

  REXML::XPath.each(xml, "//item/") do |elem|
    item = RssItem.new
    item.title       = REXML::XPath.match(elem, "title/text()").first.value rescue ""
    item.link        = REXML::XPath.match(elem, "link/text()").first.value rescue ""
    item.description = REXML::XPath.match(elem, "description/text()").first.value rescue ""
    item.author      = REXML::XPath.match(elem, "dc:publisher/text()").first.value rescue ""
    item.date        = Time.mktime(*ParseDate.parsedate(REXML::XPath.match(elem, "dc:date/text()").first.value)) rescue Date.parse(REXML::XPath.match(elem, "pubDate/text()").first.value) rescue Time.now

    item.description_link = item.description
    item.description.gsub!(/<\/?a\b.*?>/, "") # remove all <a> tags
    items << item
  end

  items.sort_by { |item| item.date }
end

I’m just going to replace that with a Nokogiri equivalent. After adding nokogiri to the Gemfile, I just added a parse_rss_nokogiri method:

def parse_rss_nokogiri(body)
  xml = Nokogiri::XML(body)

  items        = []
  link         = xml.xpath("//channel/link").first.text rescue ""
  title        = xml.xpath("//channel/title").first.text rescue ""

  xml.xpath("//item").each do |elem|
    item = RssItem.new
    item.title       = elem.xpath("title").first.text rescue ""
    item.link        = elem.xpath("link").first.text rescue ""
    item.description = elem.xpath("description").first.text rescue ""
    item.author      = elem.xpath("dc:publisher").first.text rescue ""
    item.date        = Time.mktime(*ParseDate.parsedate(elem.xpath("dc:date").first.text)) rescue Date.parse(elem.xpath("pubDate").first.text) rescue Time.now

    item.description_link = item.description
    item.description.gsub!(/<\/?a\b.*?>/, "") # remove all <a> tags
    items << item
  end

  items.sort_by { |item| item.date }
end

And then updated the parse method to call it:

  def parse(url)
    open(url) do |http|
      return parse_rss_nokogiri(http.read)
    end
  rescue
    []
  end

Let’s re-run the profiler and then check out the results:

REXML is no longer in the callgrind trace, and because Nokogiri is so fast, it isn’t either. In fact, our overall page runtime has dropped from 430ms to 164ms - an improvement of 262%.

Conclusion

Using ruby-prof (and request_profiler), I was able to take an app I know literally nothing about, profile it, identify a slow spot in its default page, and improve its runtime by 262%. ruby-prof and kcachegrind rock. Use them.

Seppuqu: Self-terminating Sidekiqs

Thu, 01 Aug 2013 00:00:00 -0500

We’ve been having an issue with Sidekiq occassionally not shutting down properly during a deployment. This ends up causing issues, since it can mean that we get Sidekiq workers that end up running different code than what we expect that they’re running.

I whipped up a quick little middleware that causes Sidekiq to self-terminate if it determines that it’s running a version of code that is older than the latest release. We use Capistrano, so my Sidekiq::current_release_version code just parses out the current version from the path, but you could use whatever versioning system you want as long as it compares < and > cleanly.

At some point I’ll probably wrap this up into a little gem.

# In an initializer
module Sidekiq
  def self.current_release_version
    @current_release_version ||= File.expand_path(__FILE__).scan(/\d{10,}/).map(&:to_i)[0] || -1
  end

  def self.latest_release_version
    Sidekiq.redis do |conn|
      conn.get("release_version") || -1
    end.to_i
  end

  module Middleware
    class VersionEnforcerMiddleware
      def call(worker, msg, queue)
        lrv, crv = Sidekiq.latest_release_version, Sidekiq.current_release_version
        if lrv <= crv
          yield
        elsif
          Sidekiq.logger.info "My version (#{crv}) mismatches latest version (#{lrv}). Shutting down..."
          Thread.main.raise Interrupt
        end
      end
    end
  end
end

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add Sidekiq::Middleware::VersionEnforcerMiddleware
  end
end

Sidekiq.redis {|c| c.set "release_version", Sidekiq.current_release_version }

Making MongoMapper 9x faster.

Mon, 29 Jul 2013 00:00:00 -0500

I’ve been doing some heavy work on MongoMapper lately. It all started with a StackOverflow question, which led to a Rails developer user asking me what I thought of Mongoid vs MongoMapper. I’ve been using MM for ages, and was happy enough to offer a favorable opinion of it. But, I wanted to back up my assertions. I wrote a benchmark. It…was disappointing.

This touched off a huge amount of work which resulted in some extremely large bugfixes, massive improvements to MM’s document read speeds. Here’s how it happened.

Know Where You Come From

Initially, my goal was to prove the MongoMapper was approximately as fast at Mongoid at common use cases. I wrote a quick little benchmark, along the lines of this:

class MMUser
  include MongoMapper::Document
  plugin MongoMapper::Plugins::IdentityMap

  key :name,      String
  key :age,       Integer
  key :birthday,  Date
  key :timestamp, Time
end

class MongoidUser
  include Mongoid::Document

  field :name,      type: String
  field :age,       type: Integer
  field :birthday,  type: Date
  field :timestamp, type: Time
end

LOOPS = 150

Benchmark.bm do |x|
  x.report "Mongomapper" do
    profile "mm" do
      LOOPS.times { MMUser.limit(200).all }
    end
  end

  x.report "Mongoid" do
    profile "mongoid" do
      LOOPS.times { MongoidUser.limit(200).to_a }
    end
  end
end

Nothing too fancy. But, imagine my dismay when I got the results:

MongoMapper Read 150000     batches of 1000       64.250000   5.950000  70.200000 ( 70.786035)
    Mongoid Read 150000     batches of 1000        5.810000   0.580000   6.390000 (  6.499160)

Ouch. Ouch. MongoMapper isn’t just slower, it’s an order of magnitude slower. Time to dig in. What’s going on?

How I Learned To Stop Worrying And Love The Benchmark

One of my favorite tools for performance analysis is kcachegrind. I use Windows as my desktop, and do my development on a headless Linux server. Fortunately, kcachegrind is cross-compiled for Windows. There’s also a OS X equivalent called qcachegrind for the Mac users out there.

I cut the benchmark down a bit for the purposes of iteration, and added ruby-prof profiling. This gives us something like:

Mongomapper 30.030000  21.560000  51.590000 ( 51.948948)

So now we know where we start - about 1730 microseconds per document load.

ruby-prof comes with support for cachegrind dumps baked in. I’ve written about it before, so I won’t cover it here. Instead, let’s just look at what kcachegrind says.

I’m performing 150 loops and reading 200 documents apiece. Each document contains 5 keys (the 4 named plus _id), so I’d expect 5 * 150 * 200 = 150,000 calls to write_key. But instead, the first things I noticed were:

read_key is called 600,000 times (4x per document load)
write_key is called 300,000 times (2x per document load)

Surely we can improve that. That sounds like a good place to start.

MongoMapper uses a plugin architecture that starts with a basic implementation of a set of methods on each document, then allows plugins to override them and call super to add additional functionality. This is nice because it helps encourage good separation of concerns, and it makes it easy to treat functionality as separate modules for the purposes of debugging. Unfortunately, this also means that a given module might not only be running the code that it thinks it is.

MM does its document loads in keys.rb. This is pretty straightforward - it takes a hash of attributes, and for each attribute, calls a setter that sets the value and allows MM to do its typecasting magic. Easy, right?

Well, almost. Because of the plugin architecture, our “private” method write_key turns out to do an awful lot of work!

As you can see, Keys calls []= which calls Dirty#write_key, which then super back to Keys::write_key. Let’s go look at Dirty#write_key:

Uh-oh. I think we found the source of our read_key calls. What’s happening here is that MongoMapper’s dirty attributes functionality overwrites write_key so that whenever a key is written, its old value is read (that’s the read_key call), and is then compared to the value being written. If the values mismatch, then the field is marked dirty. But this is a database load! We don’t need plugins to be able to modify and act on the data - we just want our data to be set on the document.

The solution I went with was to split this into an internal_write_key method and an overridable write_key method. When loading from the database, Keys is just concerned about getting the object populated. So, load_from_database now calls internal_write_key, and we changed write_key to do the same, so we have a consistent plugin interface, but bypass letting plugins get their hooks into the data as it’s being read out of the DB.

--- a/lib/mongo_mapper/plugins/keys.rb
+++ b/lib/mongo_mapper/plugins/keys.rb
@@ -284,7 +284,7 @@ module MongoMapper
             if respond_to?(:"#{key}=") && !self.class.key?(key)
               self.send(:"#{key}=", value)
             else
-              self[key] = value
+              internal_write_key key, value
             end
           end
         end
@@ -300,6 +300,10 @@ module MongoMapper
         end

         def write_key(name, value)
+          internal_write_key(name, value)
+        end
+
+        def internal_write_key(name, value)
           key         = keys[name.to_s]
           as_mongo    = key.set(value)
           as_typecast = key.get(as_mongo)

Not too much, right? Let’s see what it does to performance:

Mongomapper 15.420000  10.530000  25.950000 ( 26.153595)

Not bad! Just cutting out the excessive reads and extra overhead due to Dirty#write_key improved our load time from 1730usec to 872usec - cutting our runtime in half. But we can do better!

Plucky 0.5.2 had a bug in it that would cause cursors to be iterated twice when reading. This basically meant that MM had to parse twice the data per document load. Let’s relax the Plucky reference:

--- a/mongo_mapper.gemspec
+++ b/mongo_mapper.gemspec
@@ -15,5 +15,5 @@ Gem::Specification.new do |s|

   s.add_dependency 'activemodel',   '~> 3.0'
   s.add_dependency 'activesupport', '~> 3.0'
-  s.add_dependency 'plucky',        '~> 0.5.2'
+  s.add_dependency 'plucky',        '~> 0.5'

That brings us to:

Mongomapper  8.080000   5.230000  13.310000 ( 13.426457)

Down to 443 usec per document load. Getting much better now!

Looking at a fresh callgrind trace, we see that we’re still spending a LOT of time in write_key (which we expect; this is a read-from-the-database test after all), but perhaps that can be optimized?

write_key is spending a lot of time in Keys#set. However, the reason that Keys#write_key calls Key#set is to cast the incoming value to a Mongo-friendly type. But, we know that we’re already coming into this with a Mongo-friendly type! We’ll add a parameter to internal_write_key that prevents it from trying to perform a cast in this case:

--- a/lib/mongo_mapper/plugins/keys.rb
+++ b/lib/mongo_mapper/plugins/keys.rb
@@ -274,7 +274,7 @@ module MongoMapper
             if respond_to?(:"#{key}=") && !self.class.key?(key)
               self.send(:"#{key}=", value)
             else
-              internal_write_key key, value
+              internal_write_key key, value, false
             end
           end
         end
@@ -305,11 +305,11 @@ module MongoMapper
           internal_write_key(name, value)
         end

-        def internal_write_key(name, value)
+        def internal_write_key(name, value, cast = true)
           key = keys[name.to_s]
           set_parent_document(key, value)
           instance_variable_set :"@#{name}_before_type_cast", value
-          instance_variable_set :"@#{name}", key.set(value)
+          instance_variable_set :"@#{name}", cast ? key.set(value) : value
         end
     end
   end

Our reward:

Mongomapper  4.890000   3.560000   8.450000 (  8.531254)

Down to 284 usec/document!

We’re out of obviously slow stuff, but let’s keep looking. Keys::ClassMethods#keys, Keys::#keys, and Keys::ClassMethods#key? are cumulatively taking about 57 usec/document - about 20% of the total runtime!

Looking in Keys::ClassMethods::key?

def key?(key)
  keys.keys.include?(key.to_s)
end

Well, that won’t do; Ruby provides a key? method on Hash already.

--- a/lib/mongo_mapper/plugins/keys.rb
+++ b/lib/mongo_mapper/plugins/keys.rb
@@ -32,7 +32,7 @@ module MongoMapper
         end

         def key?(key)
-          keys.keys.include?(key.to_s)
+          keys.key? key.to_s
         end

Mongomapper  4.720000   3.190000   7.910000 (  7.987447)

Another 22 usec/document gain.

We’re making a lot of calls to the instance and class-level #keys methods. Perhaps those can be streamlined away.

(This was actually a much more involved change, but it’s simplified for purposes here)

--- a/lib/mongo_mapper/plugins/keys.rb
+++ b/lib/mongo_mapper/plugins/keys.rb
@@ -270,8 +270,9 @@ module MongoMapper
       private
         def load_from_database(attrs)
           return if attrs.blank?
+          @__keys = self.class.keys
           attrs.each do |key, value|
-            if respond_to?(:"#{key}=") && !self.class.key?(key)
+            if respond_to?(:"#{key}=") && !@__keys.key?(key)
               self.send(:"#{key}=", value)
             else
               internal_write_key key, value, false
@@ -302,11 +303,12 @@ module MongoMapper
         end

         def write_key(name, value)
+          @__keys = self.class.keys
           internal_write_key(name, value)
         end

         def internal_write_key(name, value, cast = true)
-          key = keys[name.to_s]
+          key = @__keys[name.to_s]
           set_parent_document(key, value)
           instance_variable_set :"@#{name}_before_type_cast", value
           instance_variable_set :"@#{name}", cast ? key.set(value) : value

And results:

Mongomapper  4.200000   2.700000   6.900000 (  7.009927)

Yet another 29 usec/document gain.

In Closing

There were a number of other changes, refactorings, and improvements made, and even more tested and discarded, but the fundamental procedure remains the same - measure, use those measurements to find a line of attack, and improve. The test suites will help you discover if you’ve messed something up, and your benchmarking tools can help you improve your runtimes by finding and eliminating unnecessary code paths.

In the end, we ended up with something like this:

Mongomapper  3.290000   2.410000   5.700000 (  5.761945)

Down from 1730 usec/document to 192 usec/document, a gain of ~9x, making it competitive with Mongoid. At some point it becomes difficult to do an apples-to-apples comparison, as Mongoid uses the Moped driver to talk to MongoDB, while MongoMapper still uses the 10gen Mongo driver, so there is a divergence in terms of what each ODM is doing under the covers.

This is just the tip of the iceberg, but it’s the core of the optimization pass. In the process of making these changes, I converted the test suite to rspec, made embedded associations lazy-loaded, added finer-grained control over document marshalling, added ActiveRecord-style block syntax to document creation, added key aliases, and more.

All of this work and more has gone into getting MongoMapper back into the race in terms of speed, and I’m proud to say that the 0.13.0.beta1 release has been published and is looking great so far. If all goes well, then we’ll be pushing 0.13.0 soon, which will be the last release in the 0.x series. Once that happens, we’ll be merging in Rails 4 compatibility, making some backwards-incompatible improvements to embedded associations, and more. Onward, to the future!