Use Amazon Cloudfront when appropriate.
Every project that starts growing needs to track down performance issues and bottlenecks, audiobox.fm is no exception.
However there are cases when the cure is worst than the disease, this is one.
We do streaming, a lot of it, the entire concept of audiobox.fm revolves around streaming your media over the Internet and it should be fast as possible.
A solution was needed to store and stream this content privately, after some evaluations we eventually settled on Amazon S3 and things shaped up pretty nicely from then, instant streaming even from Europe by fetching data contained in an US bucket.
I then received an email from Amazon, in which they explained they opened up CloudFront, a content delivery network working over HTTP and RTMP.
The advantages over the traditional fetch-from-bucket system is that the content is served from the nearest cloud to the requesting user, thus greatly reducing latency.
It's all fun and games until I noticed that the streaming was actually slower then fetching from the US bucket. It may be that the servers are suffering a heavy load, but I think there's more.
I started digging around and I think I have some explanation to that, making audiobox a non-use case.
A bit of background
The doubts in using CloudFront started when I was investigating the possibility to use this CDN as an asset server, serving css/javascript to the end user; however I did noticed that many users on Amazon forums started to ask why their assets were not in sync with the actual content of the bucket after an update.
The answer is simple, while CloudFront fetches from the S3 bucket, it does cache the file, which is in fact the purpose of a CDN.
It's not possible, at the moment, to manually expiry a file to be re-fetched from the bucket, instead the developer is asked to either:
- wait 24 hours
- rename the asset
So?
While many developers try to find a way to expire their assets we have the opposite problem, we would like to see them there stored forever (or at least for a long time).
When making an initial request to CloudFront, the system checks the existance of the file on the bucket and then it gets transferred using the internal Amazon resources to the nearest point.
If I ask a file when I'm in Europe, CloudFront will fetch the file from the US before serving it to me, thus adding a overhead in our request.
A 25MB file needs to travel through Amazon internal network, getting stored on CloudFront server and then served to me.
Now, this solution is super-fine when streaming content to the general public, let's say a video, because it will be requested multiple times and chances are that the file is already in the geographically located CloudFront distribution.
But for a private collection of audio, say 1000 files, this is impratical because records will expire in 24 hours. Files don't get stored on the CDN for more. There will always be a "fetch-again" from the US bucket, adding the extra overhead.
We will continue monitoring and testing CloudFront, but for our use there are disadvantages:
- slower streaming for a private, single-user cloud
- sometimes stream starts when the file has been full downloaded
- copy on CDN is useless because chances are that if the user wants to listen to a file again (in the 24 hours range) browser cache will help there, thus making no request to the CDN, rendering useless its purpose
- effective cost (CloudFront is not free)
- coupled with the fact that Safari/Webkit suffers a HTML 5 bug where the audio and video tag src gets requested twice (even three times sometimes) it's killer
The ideal solution would be that CloudFront proactively mirrors buckets in every of its geographic location, but that will never happen for many reasons.
Installing MySQL Ruby Gem in Snow Leopard Server
Snow Leopard Server doesn't come with MySQL client and library files.
In order to get them we need to download a package, for Snow Leopard Server 10.6.2:
http://www.opensource.apple.com/other/MySQL-49.binaries.tar.gz (71.2MB)
The main entry page should be http://www.opensource.apple.com/release/mac-os-x-1062/ but I cannot find the package there, getting the url was guess work from reading an Apple support page: http://support.apple.com/kb/TA25017
What we are doing is to uncompress a archive directly in our root.
Once you get the file you should decompress it unless your browser already did for you.
Since Safari did it for me I'm having MySQL-49.binaries.tar in my Downloads folder.
Open up a terminal.
cd Downloads sudo tar -xvf MySQL-49.binaries.tar -C / # use -zxvf if you still have the .tar.gz file
We should have the required libraries in order to install the MySQL Ruby Gem.
You may have already read the previous articles about MySQL and Ruby, however the procedure is different:
sudo env ARCHFLAGS="-arch x86_64" gem install mysql
This should give you a working installation of MySQL Gem.
Hosting Git repositories through WebDAV on Snow Leopard Server
This will be quick and dirty; the necessity comes from a particular source code that I can't trust to host online to a third party vendor/provider.
I have a Snow Leopard Server, so I will make use of the tools provided, there's little information about the subject, so here it is.
Also I choose WebDAV since my SSH is blinded to the outside world, in order to access it I require a VPN connection, but since I want to share some repo to the outside world, WebDAV comes handy.
I host my repositories in a subdomain, so before anything, open up your DNS Server Admin section and start editing by adding your new shiny hostname as you please by adding a new record there.
I assume you are using a subdomain along the lines of repositories.yourdomain.com; you don't have to do this step if using an ip address, this is up to you.
Login into your server through ssh or the way you like and create a repositories index directory somewhere.
In my particular setup I have this structure for my websites:
/Sites/kain/mydomain.com/[www, macruby, etc]
So go where you like and create a directory, I'll use repositories
mkdir /Sites/kain/mydomain.com/repositories
Assign this directory to the _www user
chown -R _www:_www /Sites/kain/mydomain.com/repositories cd /Sites/kain/mydomain.com/repositories
Start creating a new directory that will host the git repository:
mkdir myproject.gitAfter that go in the Web section, and create (or copy) a new website.
Relevant General setup:
Host name: repositories.yourdomain.com
Web folder: /Sites/kain/mydomain.com/repositories
Relevant Options setup:
Folder listing: enabled (not really necessary but useful for testing what you see in index later)
WebDAV: enabled
Rest disabled
Relevant Realm setup:
create a new realm by clicking the plus sign
Realm name: myproject
Authentication: Basic (will try Digest/Kerberos in future)
Select Location for the select widget instead of Folder and write
/myproject.gitIn the right section now add your users that will have access, after that change their permissions.
Since my repos are private I only keep the users that have write access there by setting their permissions to Browse and Read/Write WebDAV, None for Everyone.
Relevant Logging setup:
enable logging as you please
Relevant Security setup:
I choose to use SSL for this tutorial, so you are going to use a certificate and port 443.
Save and close Server Admin.
Let's go back to our SSH server session.
Enter your project directory and exec those commands:
cd /Sites/kain/mydomain.com/repositories/myproject.git sudo git init --bare sudo chown -R _www:_www . sudo mv hooks/post-update.sample hooks/post-update sudo git update-server-info
Now switch to your client.
Edit your ~/.netrc file with those informations:
machine repositories.mydomain.com
login myusername
password mypasswordEnter your project dir and exec those commands:
cd myproject git config --global http.sslVerify false # If you don't own a valid SSL certificate on your server, skip --global to enable this option only for this project git init # gitify the directory git add . # add files to git repo git commit -m "hi" # first commit git remote add origin https://repositories.mydomain.com/myproject.git/ # end slash is important git push origin master --force -v
Now edit this file, always in your project root dir:
mate .git/configAnd add
[branch "master"] remote = origin merge = refs/heads/master
This way you can do git push and git pull normally.
Done.
AudioBox.fm
Hi readers,
I'm putting together a new project, dubbed audiobox.fm.
It's in active development; if you feel you can leave your email or follow the project on Twitter to be notified when doors open for beta stage.
I'm going to limit the initial access to the app, so hurry :)
Cheers.
My ToolBox
Being a computer junkie I usually use a certain number of tools, ranging from productivity to my amusement. Here's a generic list.
Hardware
MacBookPro5,1 Intel Core 2 Duo 2,53 GHz 4 GB RAM
Time Capsule 1TB
Random Logitech Mouse
Development
TextMate
Safari
Firefox
Xcode and Interface Builder
Terminal.app
Navicat Premium
phpMyAdmin
phpPgAdmin
Sequel Pro
Parallels 5
GitX
System Utilities
Server Admin and Workgroup Manager
Angry IP Scanner
SuperDuper!
DiskWarrior
WhatSize
Imaging
Adobe Photoshop CS3
Skitch
Video
VLC
ReduxEncoder
IShowU HD
HandBrake
Networking
ForkLift
Tweetie
Skype
Transmission
X-Chat Aqua
Adium
Music
iTunes
CoverSutra
XLD
Mp3/Tag Studio (on Parallels) plus various home-made tools
Everyday Apps
Mail.app
Disco
AppCleaner
RAR Expander
Still something is missing, but you get the picture.
About ripping articles
My apologies, the Cappuccino series articles has been suspended indefinitely. I won't go into details, so don't ask.
Also sorry if it sounds cryptic to you.
Install id3lib-ruby on Snow Leopard
We are going to use MacPorts for this.
Install the prerequisite:
sudo port install id3lib
And install the 64-bit gem:
env ARCHFLAGS="-arch x86_64" gem install id3lib-ruby
Test:
16:53 ~ $ irb >> require 'id3lib' => true
Cappuccino: the jake branch

Those steps assumes you have already setup your Cappuccino source similarly to my environment.
If not, please read previous articles.
Keep in mind that the "jake" branch will be merged into master at some point, and the switch should happen sometime in the upcoming weeks; at least I've been told this way so take this information with a grain of salt.
Therefore: those instructions can turn obsolete at any time and probably are.
If you read my previous article you already know that I'm running Snow Leopard.
At any point do not close the Terminal window until you did every step.
Download Narwhal from GitHub.
cd ~/src/ git clone git://github.com/tlrobinson/narwhal.git
Activate paths and use tusk to install required packages.
source ~/src/narwhal/bin/activate tusk install jake browserjs narwhal-jsc jack
Build Narwhal JSC:
cd ~/src/narwhal/packages/narwhal-jsc make webkit export NARWHAL_ENGINE=jsc
Build Cappuccino:
# if you don't already have a cappuccino source remember to have a CAPP_BUILD defined and # git clone git://github.com/280north/cappuccino.git ~/src/cappuccino_source cd ~/src/cappuccino_source git checkout -b jake origin/jake rm -rf $CAPP_BUILD jake jake debug # if you want also the Debug build jake docs # if you want up to date local xml/html documentation, requires doxygen
Building Cappuccino with jake instead of rake will take some minutes to compile.
I read about some people reporting that the process gets stuck, this didn't happened to me (well it did, but because I was compiling against master by mistake); the jake building process took about 4 minutes to complete on a 2,53 GHz Unibody Intel Core 2 Duo Macbook Pro.
We are going to tweak our $PATH in order to find the new Cappuccino Tools and Narwhal binaries like jake.
Keep in mind that you probably already have them in /usr/local/bin so we must adjust our PATH with some precedence.
Old utilities in /usr/local/bin aren't modified by this procedure, since the jake task that should install them is currently commented.
Here is my .bash_profile related code:
# CAPP_BUILD, directory for built Cappuccino framework. export CAPP_BUILD="$HOME/src/cappuccino_framework" # Cappuccino jake branch. PATH="$CAPP_BUILD/Release/CommonJS/objective-j/bin:$PATH" PATH="$HOME/src/narwhal/bin:$PATH" # Finalize PATH export PATH=$PATH
This will make capp, nib2cib and press available on command line.
Reopen Terminal and test:
03:52 ~ $ which capp /Users/kain/src/cappuccino_framework/Release/CommonJS/objective-j/bin/capp 22:42 ~ $ cd Sites/ 22:42 ~/Sites $ capp gen MySecondApp -t NibApplication /Users/kain/src/cappuccino_framework/Release/CommonJS/objective-j/lib/capp/Resources/Templates/NibApplication -> MySecondApp # ...
Have fun!
On Cappuccino spriting and why it matters, part 1

The first working builds of sprited Cappuccino were released around 14th October.
If you're familiar with Cappuccino this means frameworks that include images/themes and also entire applications can be sprited.
This will be the default behaviour for Cappuccino applications, we'll talk about advantages in a bit, first, a word about spriting if you're wondering what is it.
Developed around mid-1970, spriting is a technique still used today; if you're into CSS you might have used it.
If you're old enough and curious you may have also dug into a videogame's resource files and you may have seen images composed of multiple little images in a grid.
In fact spriting mostly consists in placing images into a larger image grid, and display the image we are requesting by supplying its correct position in the grid.
What that means in Cappuccino?
Simply put, by using spriting, Cappuccino can squeeze all your project's images in a single file, thus reducing the number of requested/served files.
This is a big gainer, since Cappuccino also makes intensive use of themes; Aristo, the now default theme, counts around 111 files.
They will be only 2 after spriting kicks in for all users in master branch.
The wonderful thing is that it's built-in, you are not required to do this kind of stuff. Couple that with the fact that will be using Narwhal-JSC soon, Cappuccino will be *really* fast.
I will discuss with Francisco Tolmasky (tolmasky) of 280 North about how spriting works in Cappuccino and as usual will report my findings here, in a second part of the article.
Foundation.j and AppKit.j 404
During some Rails integration with Cappuccino I noticed that my Rails' logs were filling fast with 404s.
At first I thought I did something wrong in including my Frameworks, however it turns out that's a normal behaviour by design.
I won't go into the details here on why it's happening, but here's a solution:
Ruby on Rails
To avoid the stack trace generation overhead in Rails simply put down those routes:
map.connect '/Frameworks/Foundation/Foundation.j', :controller => 'application', :action => 'handle_cappuccino_404', :conditions => { :method => :get } map.connect '/Frameworks/AppKit/AppKit.j', :controller => 'application', :action => 'handle_cappuccino_404', :conditions => { :method => :get }
Then in your handle_cappuccino_404 method you can simply render :nothing => true:
class ApplicationController < ActionController::Base def handle_cappuccino_404 render :nothing => true end end
I think we can't rescue_from ActionController::RoutingError at this point.
Please also read another solution involving Rails Metal below.
Rails Metal
I was thinking about using Rails Metal to avoid even the extra bits:
script/generate metal cappuccinoThen in app/metal/cappuccino.rb:
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails) class Cappuccino def self.call(env) if env["PATH_INFO"] =~ /^\/Frameworks\/Foundation|AppKit\/Foundation|AppKit.j/ [200, {"Content-Type" => "text/html"}, []] else [404, {"Content-Type" => "text/html"}, ["Not Found"]] end end end
No need for the method described above: you can safely remove routes and the 404 handling method from your controller.
Sinatra
To avoid it in Sinatra:
# from http://github.com/mchung/cappuccino_sandbox get "/Frameworks/AppKit/AppKit.j" do [404, {}, "Look elsewhere"] end get "/Frameworks/AppKit/Foundation.j" do [404, {}, "Look elsewhere"] end
