Rails webapp bundle (idea)

September 14th, 2012 · · Programming One comment

Having created many web application in Ruby, I always feel pain in my ass when I have to deploy a new release. Usually I have problems with the environment, last time for instance I had to reconfigure my appication to use postgresql instead of sqlite just to work on heroku. I’m paranoid and I want to be sure the deployed application works perfectly. I use TDD, write so many unit and functional tests under developing which pass on my computer, but it doesn’t mean my software works on the production environment too. The hosting services provide me only a black box, just for example heroku where the deploying process installs and configures everything after pushing to the master branch and I just hope it will work by end of deployment.

I was thinking on a unique solution for releasing web applications. My goal is a web application would run like a simple application, just start with a client tool and it runs everything it needs. I describe my idea and give you a little proof of concept to show it works!

Idea

The idea came from Google AppEngine where the applications must be read only, combining it with the LiveCD solution where you can use an operating system from a cd without any IO problems. What would happen if my whole application and its dependencies (web server, log service, database engine) moved to a readonly image and I just run it like the operating system from LiveCD.

Later I can test this readonly image with a writeable partition like a black box testing, so when all of the functional (or end-to-end) tests pass through the web server will mean that the application in the image works well with the installed services. The deploying process will copy the image to the production environment, stops and detaches the currently running app version and start the the new application version with the new environment.

In this blog entry I want to show you a way how you can create an image that contains everything you need.

Release process

For creating the bundle I will use Debian squeeze Linux distribution. If you use Mac or Windows I suggest you to install a virtual machine to VirtualBox or to whatever you like, I have a special deployment server for this process.

When I bumped the version of the application, a git commit hook activates the release process to create a fresh bundle.

Alright, so I suppose you have installed the Debian Squeeze system. Now let’s create the bundle.

First stage – Create the chroot environment

First switch to root user, because the commands will need root priviliges:

1

su
su

To create a chroot environment for the application stack we will use the debootstrap tool.

1
2
3

apt-get install debootstrap
mkdir -p /root/bundle
debootstrap squeezie /root/bundle
apt-get install debootstrap
mkdir -p /root/bundle
debootstrap squeezie /root/bundle

Mount the proc file system is necessary to reach some kernel services from the chroot.

1

mount -t proc /proc /root/bundle/proc
mount -t proc /proc /root/bundle/proc

Second stage – Install the dependencies

Now we have a chroot environment, we will install some additional packages in it. Curl will be necessary the fetch Ruby and build-essential contains g++, gcc and other important packages to compile nginx and gem files which contain C code.

1
2
3

chroot /root/bundle
apt-get install curl
apt-get install build-essential
chroot /root/bundle
apt-get install curl
apt-get install build-essential

Install the current rvm version that will be useful to setup the required Ruby version.

1
2

curl -L https://get.rvm.io | bash -s stable --ruby
echo "source /etc/profile.d/rvm.sh" >> /etc/bash.bashrc
curl -L https://get.rvm.io | bash -s stable --ruby
echo "source /etc/profile.d/rvm.sh" >> /etc/bash.bashrc

Before installing Ruby, some zlib and openssl libraries should be installed.

1
2
3
4

apt-get install zlib1g-dev
apt-get install libssl-dev
apt-get install libpcre3-dev
apt-get install libcurl4-openssl-dev
apt-get install zlib1g-dev
apt-get install libssl-dev
apt-get install libpcre3-dev
apt-get install libcurl4-openssl-dev

Now we have everything to install the environment for the application, so fetch the application first. I will show you the following steps on my Check.it project.

1
2
3

apt-get install git-core
mkdir /var/www
git clone https://github.com/Nucc/check.it.git /var/www
apt-get install git-core
mkdir /var/www
git clone https://github.com/Nucc/check.it.git /var/www

Install the Ruby version you need. Usually the project’s root directory contains an .rvmrc file with Ruby version information.

1

rvm install ruby-1.9.3
rvm install ruby-1.9.3

Install the system dependencies of your application. I will install sqlite3, but you may need mysql or postgresql in this step:

1
2
3
4
5

apt-get install libsqlite3-dev
# apt-get install mysql-client
# apt-get install postgresql-client
 
// install dev packages that your application requires
apt-get install libsqlite3-dev
# apt-get install mysql-client
# apt-get install postgresql-client

// install dev packages that your application requires

Install the dependencies of the application (you can use Gemfile.lock later with bundle install --deployment)

1
2
3

gem install bundler
cd /var/www
bundle install
gem install bundler
cd /var/www
bundle install

I use Nginx with Passenger to balance and proxy the requests to the application.

1
2
3

gem install passenger
passenger-install-nginx-module
# Choose number 1 in setup process
gem install passenger
passenger-install-nginx-module
# Choose number 1 in setup process

Make some security steps. Change to owner of /var/www to www-data, we will run nginx with www-data permissions.

1

chown -R www-data /var/www
chown -R www-data /var/www

The last step is to configure the nginx server. Insert the following configuration to /opt/nginx/conf/nginx.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

user  www-data;
worker_processes  5;
 
error_log  /var/www/log/error.log;
pid        /var/www/log/nginx.pid;
 
events {
    worker_connections  1024;
}
 
http {
    passenger_root /usr/local/rvm/gems/ruby-1.9.3-p194/gems/passenger-3.0.17;
    passenger_ruby /usr/local/rvm/wrappers/ruby-1.9.3-p194/ruby;
 
    include       mime.types;
    default_type  application/octet-stream;
 
    gzip  on;
 
    server {
        listen       80;
        server_name  localhost;
 
    root /var/www/public;
        passenger_enabled on;
        passenger_use_global_queue on;
    }
}
user  www-data;
worker_processes  5;

error_log  /var/www/log/error.log;
pid        /var/www/log/nginx.pid;

events {
    worker_connections  1024;
}

http {
    passenger_root /usr/local/rvm/gems/ruby-1.9.3-p194/gems/passenger-3.0.17;
    passenger_ruby /usr/local/rvm/wrappers/ruby-1.9.3-p194/ruby;

    include       mime.types;
    default_type  application/octet-stream;

    gzip  on;

    server {
        listen       80;
        server_name  localhost;

	root /var/www/public;
        passenger_enabled on;
        passenger_use_global_queue on;
    }
}

Alright, we have everything we need, exit from the chroot environment and umount the /proc directory!

1
2

exit
umount /root/bundle/proc
exit
umount /root/bundle/proc

Third stage – Create image file

Now we create a readonly image file that contains everything that we have created in the previous stage. We will use Squashfs to pack it to one file:

1
2
3
4

apt-get install squashfs-tools
 
# Be sure you unmounted the /root/bundle/proc directory!
mksquashfs /root/bundle /root/release.webapp
apt-get install squashfs-tools

# Be sure you unmounted the /root/bundle/proc directory!
mksquashfs /root/bundle /root/release.webapp

/root/release.webapp will be the bundle that we will run on the hosting machine.

Launch process

We will create a very simple environment to be able to run this readonly image. I use a Debian squeeze machine for this purpose too, but if you need a machine to host the application I suggest you IntoVPS virtual hosting. The following picture shows the architecture of the running application environment:

Aufs is a union fs tool running in kernel space that unions a readable and a writeable directory together. Every changes on the readable directory will appear on the writeable directory. We have 3 directories the run the app, one for the readonly (/root/app/readonly), one for the writeable (/root/app/writeable), and one for the union directory (/root/app/active). We will use loop device to mount the .webapp image content to the readonly directory.

1
2
3

mkdir -p /root/app/writeable
mkdir /root/app/readonly
mkdir /root/app/active
mkdir -p /root/app/writeable
mkdir /root/app/readonly
mkdir /root/app/active

Create the mounts:

1
2

mount /root/release.webapp -o loop /root/app/readonly
mount -t aufs -o br=/root/app/writable=rw:/root/app/readonly=ro none /root/app/active
mount /root/release.webapp -o loop /root/app/readonly
mount -t aufs -o br=/root/app/writable=rw:/root/app/readonly=ro none /root/app/active

Mount the /proc and /var/share if you have any resource (database) that uses any application communicates on unix-socket.

1

mount -t proc /proc /root/app/active/proc
mount -t proc /proc /root/app/active/proc

Try out your environment:

1
2
3
4

chroot /root/app/active
cd /var/www
rake db:migrate
/opt/nginx/bin/nginx
chroot /root/app/active
cd /var/www
rake db:migrate
/opt/nginx/bin/nginx

Goals

My goal is to automate the release and launch process in Ruby and Bash. So I’m looking for volunteers to implement the server and client side. The code will be hosting under my Github account.

1. The release process should recognize the used Ruby version from .rvmrc file of rackup application.
2. The release process should parse the .webapp directory of the rackup application and install additional tools and configuration files from this directory (like memcached).
3. The .webapp directory of the rackup application may contain a startup file that describes what services should start and stop when attaching or detaching it.
4. The client application should work out of the box, so like

1
2

apt-get install webapp-client
webapp-client --start /root/app-1.2.webapp --port 8080
apt-get install webapp-client
webapp-client --start /root/app-1.2.webapp --port 8080

The goal is it works like a simple GUI application, so just launch and stop it and the tool controls everything else.

Participate

If you feel free to join send me a message on Twitter to http://twitter.com/nucc!