WebSockets from Ember, Rails, Nginx with AWS ElastiCache Redis

When we finally upgraded our Rails application to v5, plans were put in place to take advantage of its WebSockets capabilities.

Locally on development machines, things seem to chug along smoothly until it all broke down on our staging stack.

Here’s the basic setup:

* Rails 5.0.x
* Ember 2.x (using Ember Cable for WebSockets)
* Puma
* Nginx (for reverse proxying Ember calls to Rails + serving Ember app)
* AWS ElastiCache Redis
* Postgres hosted on AWS RDS
* EC2 instance + ELB + custom security group… etc.
* Entire stack is orchestrated with Chef running on AWS OpsWorks

I didn’t think the set up was anything out of the ordinary per se. The only exception is probably my choice of using ElastiCache instead of having an EC2 server running our own Redis instance (or having it hosted at Redis Labs or something). But other than that, it’s pretty vanilla.

Immediately we rant into trouble having Ember making a proper connection. We kept seeing this error:

WebSocket connection to 'ws://staging.domain.name/cable' failed: Connection closed before receiving a handshake response

First thing first, we had to make sure our /cable endpoint was proxied properly so that Rails can answer the calls. So in our main Nginx config file, we added:

1
2
3
4
5
6
7
8
9
10
location /cable {
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   Host $http_host;
    proxy_set_header   Upgrade $http_upgrade;
    proxy_set_header   Connection "Upgrade";
    proxy_redirect     off;
    proxy_http_version 1.1;
    proxy_pass         http://app_server;
}

It still didn’t quite work.

1
2
3
4
5
I, [2017-03-22T18:46:39.166623 #4175]  INFO -- : Started GET "/cable" for 127.0.0.1 at 2017-03-22 18:46:39 +0000
I, [2017-03-22T18:46:39.167699 #4175]  INFO -- : Started GET "/cable/" [WebSocket] for 127.0.0.1 at 2017-03-22 18:46:39 +0000
E, [2017-03-22T18:46:39.167837 #4175] ERROR -- : Request origin not allowed: https://staging.domain.name
E, [2017-03-22T18:46:39.167950 #4175] ERROR -- : Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE:)
I, [2017-03-22T18:46:39.168064 #4175]  INFO -- : Finished "/cable/" [WebSocket] for 127.0.0.1 at 2017-03-22 18:46:39 +0000

Obviously we need to specify in Rails that our Staging subdomain, along with its https protocol should be allowed. So we added that to our Rails /config/environments/staging.rb file.

config.action_cable.allowed_request_origins = %w(https://staging.domain.name staging.domain.name)

Once we squared that away, we were still left with that dreadful error on line #4 above.

Failed to upgrade to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE:)

Somehow, the $http_upgrade variable in Nginx is simply not being set despite browser’s request. As it turns out, WebSocket traffic isn’t quite HTTP/HTTPS. The devil is in the details of the AWS ELB (load balancer).

I simply swapped out HTTPS with SSL. And BOOM! Like that, it worked:

Successfully upgraded to WebSocket (REQUEST_METHOD: GET, HTTP_CONNECTION: Upgrade, HTTP_UPGRADE: websocket)

Now connection was being made properly. Except for this one little thing: The browser had to keep reinitialize the connection every few seconds. And we noticed the UI simply wasn’t being updated despite changes to data. Sigh…

Next stop was to test if the EC2 instance itself was actually talking to AWS ElastiCache. There are a couple of ways to try this:

In Rails:

1
2
3
4
5
irb(main):001:0> redis = Redis.new(:host => 'my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com', :port => 6379)
=> #< redis client v3.3.3 for redis://my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com:6379/0>
irb(main):002:0> redis.ping
Redis::CannoConnectError: Error connecting to Redis on my-elasticache-instance-identifier.001.usw2.cache.amazonaws.com:6379 (Redis::TimeoutError)
...

Using Telnet:

1
2
3
$ telnet my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com 6379
Trying /ip ADDRESS/...
# eventually times out

OMFG. WHAT ELSE IS WRONG NOW??!!

As it turns out, the devil, again, is in the details. This time, it’s in AWS Security Groups.

AWS resources are only allowed to talk to each other if said resources are within the same security groups (among other things). In this case, AWS ElastiCache has a default security group, which we’re not allowed to change (not to my knowledge anyway). So the trick was to make sure that our OpsWorks Layers also have been assigned the default security group. Now, OpsWorks documentation says that you must reboot each instance within the layer for the new security group settings to kick in. But the truth is I had to spawn brand new instances for the new security group to stick. But once that was done, everything finally started to gel:

1
2
3
4
5
6
irb(main):001:0> redis = Redis.new(:host => 'my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com', :port => 6379)
=> #<redis client v3.3.3 for redis://my-elasticache-instance-identifier.0001.usw2.cache.amazonaws.com:6379/0>
irb(main):002:0> redis.ping
=> "PONG"
...
</redis>

I hate technology sometimes.

References:
* https://blog.jverkamp.com/2015/07/20/configuring-websockets-behind-an-aws-elb/ (cached in PDF)

Changing AWS RDS PostreSQL database name


Working with AWS RDS isn’t always the easiest. That’s the limitations that come with hosted solutions. /shrugs/

For example, you can’t change master username once an instance has been spawned, not even on an instance restored from a snapshot.

But I did figure out how to change a Postgres RDS instance’s database name. To do this, we’ll need terminal connection to the database:

1
2
$ psql -U james_holden -h my-database-identifier.abc123.us-west-2.rds.amazonaws.com -d donnager
$ password for james_holden: # provide your db password here

At this point, if you attempted to change the database name, Postgres will complain immediately:

1
2
donnager=> ALTER DATABASE donnager RENAME TO rocinate;
donnager=> ERROR:  current database cannot be renamed

However, if you get out of the connected db, switch to the standard “postgres” db, it’s all fair game:

1
2
3
4
5
6
7
donnager=> \c postgres
psql (9.5.5, server 9.5.4)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
You are now connected to database "postgres" as user "james_holden".
postgres=> ALTER DATABASE donnager RENAME TO rocinate;
ALTER DATABASE
postgres=> \q

Peace is restored again.

Temporarily Stopping McAfee AntiMalware Agent on the Mac

McAfee’s AntiMalware product can be a real resource hog for some people. Fortunately there’s a way to simply disable it via the command line. It can then be easily re-enabled the same way.

I made an alias in my profile to take care of that:

alias kill-mcafee='sudo /usr/local/McAfee/AntiMalware/VSControl stopoas'

Re-enabling it is just as easy:

alias unkill-mcafee='sudo /usr/local/McAfee/AntiMalware/VSControl startoas'

Downgrade Node.js to a Specific Version Using Homebrew

It’s a pain in the ass to have to look this up every time I do this. So here it goes:

Install Homebrew versions:

$ brew tap homebrew/versions

Look for the package you need:

$ brew search node

That is going to return a list of packages with keywords matching “node”

$ leafnode node node010 node04 node06 node08 nodebrew nodenv

Install the package with the version you need:

$ brew install node010

Then…

$ brew link --overwrite node010

Done.

Mac OSX Yosemite Automation with Javascript

In Apple’s latest Mac OSX update, a new programming language support was added on the OS level without much advertising — Javascript. But Apple did provide documentation about it. A couple of nice folks published great resources on how to leverage them.

It’s really amazing seeing how far Javascript has come in the past few years, penetrating practically the entire stack of web application development cycle. It’s an exciting time to be a software developer.

Ember.js Lessons Learned Retrospective

Ember.js LogoWhile googling for some syntax on Ember.js, I came across this incredibly helpful list of “lessons learned” from Landon Noss aptly named “Things I wish someone had told me when I was learning Ember.js“. He documented 28 tips learned from the battlefield. I love lists like that: Concise lessons distilled down to digestible bullet points, enough for others to go through the same troubles and speed up the learning curve. Kudos to Landon for the awesome list.

Just in case the article, the link or the site disappears, here’s a PDF version I generated for safekeeping.

Debugging Ruby 2.1.x and Rails 4.1.x with RubyMine

One of RubyMine’s strongest features is its debugging tool. But it’s always been finicky to get it working for me. I finally got it to work since upgrading to Ruby 2.1.x. The situation was complicated by Ruby 2.x not yet supported by debugger gem… etc.

To get everything to sing in harmony, here’s what I used:

And that was it!

Javascript Structs and ImmutableStructs

This is a pretty cool article on implementing structs and immutable properties & objects in Javascript. Data structure like this makes it possible to construct and serialize user roles and capabilities objects on the client-side user Javascript that prevents users from tempering with the view. Bad ass!

Here’s a PDF backup of the article just in case the site or URL goes blind. You can’t be too safe!

Upgrading to Mac OSX Yosemite

Mac OSX Yosemite became publicly available today. Since my Mac is in-between projects, I decided to take a chance and upgrade.

I ran into three problems for my Ruby on Rails set up:

Other than those 3 things, pretty much everything else has worked as-is out of the box. I enjoy the crisp look of the fonts and icons in the new OS. And hopefully nothing crazy will happen going forward.

Deploying Chef without a Chef Server

chef-logo

Chef is a great tool. I really love it. But to take full advantage of it, you need a Chef server (either build one yourself or have it hosted with Opscode (which we did at my last job). For small businesses and/or personal server(s) of very small scale (I’d suggest up to 3 to 5 servers at most), a nifty tool like LittleChef can really be quite useful. But if servers you manage ever grows more than a handful, I’d highly recommend hosted service like Chef to keep your sanity.

SuperHero.js for Javascript Best Practices

Javascript has been evolving way more quickly than I have been able to keep up. I’m still behind on some of the recommended best practices and modern techniques. Luckily someone had the good sense of putting up and documenting these things. I hope s/he/they keep up the good work to spare the rest of us.

This wonderful resource is SuperHero.js. It keeps up-to-date a list of resources on best practices and latest techniques in the world of Javascript for the rest of us peasant coders. Kudos to the site maintainer(s).