iCoreTech Research Labs Just a placeholder

8Jan/101

Use Amazon Cloudfront when appropriate.

Every project that starts growing needs to track down performance issues and bottlenecks, audiobox.fm is no exception.

However there are cases when the cure is worst than the disease, this is one.

We do streaming, a lot of it, the entire concept of audiobox.fm revolves around streaming your media over the Internet and it should be fast as possible.

A solution was needed to store and stream this content privately, after some evaluations we eventually settled on Amazon S3 and things shaped up pretty nicely from then, instant streaming even from Europe by fetching data contained in an US bucket.

I then received an email from Amazon, in which they explained they opened up CloudFront, a content delivery network working over HTTP and RTMP.

The advantages over the traditional fetch-from-bucket system is that the content is served from the nearest cloud to the requesting user, thus greatly reducing latency.

It's all fun and games until I noticed that the streaming was actually slower then fetching from the US bucket. It may be that the servers are suffering a heavy load, but I think there's more.

I started digging around and I think I have some explanation to that, making audiobox a non-use case.

A bit of background

The doubts in using CloudFront started when I was investigating the possibility to use this CDN as an asset server, serving css/javascript to the end user; however I did noticed that many users on Amazon forums started to ask why their assets were not in sync with the actual content of the bucket after an update.

The answer is simple, while CloudFront fetches from the S3 bucket, it does cache the file, which is in fact the purpose of a CDN.

It's not possible, at the moment, to manually expiry a file to be re-fetched from the bucket, instead the developer is asked to either:
- wait 24 hours
- rename the asset

So?

While many developers try to find a way to expire their assets we have the opposite problem, we would like to see them there stored forever (or at least for a long time).

When making an initial request to CloudFront, the system checks the existance of the file on the bucket and then it gets transferred using the internal Amazon resources to the nearest point.

If I ask a file when I'm in Europe, CloudFront will fetch the file from the US before serving it to me, thus adding a overhead in our request.

A 25MB file needs to travel through Amazon internal network, getting stored on CloudFront server and then served to me.

Now, this solution is super-fine when streaming content to the general public, let's say a video, because it will be requested multiple times and chances are that the file is already in the geographically located CloudFront distribution.

But for a private collection of audio, say 1000 files, this is impratical because records will expire in 24 hours. Files don't get stored on the CDN for more. There will always be a "fetch-again" from the US bucket, adding the extra overhead.

We will continue monitoring and testing CloudFront, but for our use there are disadvantages:

- slower streaming for a private, single-user cloud
- sometimes stream starts when the file has been full downloaded
- copy on CDN is useless because chances are that if the user wants to listen to a file again (in the 24 hours range) browser cache will help there, thus making no request to the CDN, rendering useless its purpose
- effective cost (CloudFront is not free)
- coupled with the fact that Safari/Webkit suffers a HTML 5 bug where the audio and video tag src gets requested twice (even three times sometimes) it's killer

The ideal solution would be that CloudFront proactively mirrors buckets in every of its geographic location, but that will never happen for many reasons.

Tagged as: , , 1 Comment
23Nov/097

Installing MySQL Ruby Gem in Snow Leopard Server

Snow Leopard Server doesn't come with MySQL client and library files.
In order to get them we need to download a package, for Snow Leopard Server 10.6.2:
http://www.opensource.apple.com/other/MySQL-49.binaries.tar.gz (71.2MB)

The main entry page should be http://www.opensource.apple.com/release/mac-os-x-1062/ but I cannot find the package there, getting the url was guess work from reading an Apple support page: http://support.apple.com/kb/TA25017

What we are doing is to uncompress a archive directly in our root.
Once you get the file you should decompress it unless your browser already did for you.
Since Safari did it for me I'm having MySQL-49.binaries.tar in my Downloads folder.

Open up a terminal.

cd Downloads
sudo tar -xvf MySQL-49.binaries.tar -C / # use -zxvf if you still have the .tar.gz file

We should have the required libraries in order to install the MySQL Ruby Gem.

You may have already read the previous articles about MySQL and Ruby, however the procedure is different:

sudo env ARCHFLAGS="-arch x86_64" gem install mysql

This should give you a working installation of MySQL Gem.

23Nov/090

Hosting Git repositories through WebDAV on Snow Leopard Server

This will be quick and dirty; the necessity comes from a particular source code that I can't trust to host online to a third party vendor/provider.

I have a Snow Leopard Server, so I will make use of the tools provided, there's little information about the subject, so here it is.

Also I choose WebDAV since my SSH is blinded to the outside world, in order to access it I require a VPN connection, but since I want to share some repo to the outside world, WebDAV comes handy.

I host my repositories in a subdomain, so before anything, open up your DNS Server Admin section and start editing by adding your new shiny hostname as you please by adding a new record there.
I assume you are using a subdomain along the lines of repositories.yourdomain.com; you don't have to do this step if using an ip address, this is up to you.

Login into your server through ssh or the way you like and create a repositories index directory somewhere.
In my particular setup I have this structure for my websites:

/Sites/kain/mydomain.com/[www, macruby, etc]

So go where you like and create a directory, I'll use repositories

mkdir /Sites/kain/mydomain.com/repositories

Assign this directory to the _www user

chown -R _www:_www /Sites/kain/mydomain.com/repositories
cd /Sites/kain/mydomain.com/repositories

Start creating a new directory that will host the git repository:

mkdir myproject.git

After that go in the Web section, and create (or copy) a new website.

Relevant General setup:
Host name: repositories.yourdomain.com
Web folder: /Sites/kain/mydomain.com/repositories

Relevant Options setup:
Folder listing: enabled (not really necessary but useful for testing what you see in index later)
WebDAV: enabled
Rest disabled

Relevant Realm setup:
create a new realm by clicking the plus sign
Realm name: myproject
Authentication: Basic (will try Digest/Kerberos in future)
Select Location for the select widget instead of Folder and write

/myproject.git

In the right section now add your users that will have access, after that change their permissions.
Since my repos are private I only keep the users that have write access there by setting their permissions to Browse and Read/Write WebDAV, None for Everyone.

Relevant Logging setup:
enable logging as you please

Relevant Security setup:
I choose to use SSL for this tutorial, so you are going to use a certificate and port 443.

Save and close Server Admin.

Let's go back to our SSH server session.
Enter your project directory and exec those commands:

cd /Sites/kain/mydomain.com/repositories/myproject.git
sudo git init --bare
sudo chown -R _www:_www .
sudo mv hooks/post-update.sample hooks/post-update
sudo git update-server-info

Now switch to your client.

Edit your ~/.netrc file with those informations:

machine repositories.mydomain.com
login myusername
password mypassword

Enter your project dir and exec those commands:

cd myproject
git config --global http.sslVerify false # If you don't own a valid SSL certificate on your server, skip --global to enable this option only for this project
git init # gitify the directory
git add . # add files to git repo
git commit -m "hi" # first commit
git remote add origin https://repositories.mydomain.com/myproject.git/ # end slash is important
git push origin master --force -v

Now edit this file, always in your project root dir:

mate .git/config

And add

[branch "master"]
	 remote = origin
	 merge = refs/heads/master

This way you can do git push and git pull normally.

Done.

22Nov/090

AudioBox.fm

Hi readers,
I'm putting together a new project, dubbed audiobox.fm.

It's in active development; if you feel you can leave your email or follow the project on Twitter to be notified when doors open for beta stage.

I'm going to limit the initial access to the app, so hurry :)

Cheers.

Tagged as: No Comments
16Nov/090

My ToolBox

Being a computer junkie I usually use a certain number of tools, ranging from productivity to my amusement. Here's a generic list.

Hardware
MacBookPro5,1 Intel Core 2 Duo 2,53 GHz 4 GB RAM
Time Capsule 1TB
Random Logitech Mouse

Development
TextMate
Safari
Firefox
Xcode and Interface Builder
Terminal.app
Navicat Premium
phpMyAdmin
phpPgAdmin
Sequel Pro
Parallels 5
GitX

System Utilities
Server Admin and Workgroup Manager
Angry IP Scanner
SuperDuper!
DiskWarrior
WhatSize

Imaging
Adobe Photoshop CS3
Skitch

Video
VLC
ReduxEncoder
IShowU HD
HandBrake

Networking
ForkLift
Tweetie
Skype
Transmission
X-Chat Aqua
Adium

Music
iTunes
CoverSutra
XLD
Mp3/Tag Studio (on Parallels) plus various home-made tools

Everyday Apps
Mail.app
Disco
AppCleaner
RAR Expander

Still something is missing, but you get the picture.

Filed under: Tips No Comments
5Nov/09Off

About ripping articles

My apologies, the Cappuccino series articles has been suspended indefinitely. I won't go into details, so don't ask.

Also sorry if it sounds cryptic to you.

1Nov/091

Install id3lib-ruby on Snow Leopard

We are going to use MacPorts for this.

Install the prerequisite:

sudo port install id3lib

And install the 64-bit gem:

env ARCHFLAGS="-arch x86_64" gem install id3lib-ruby

Test:

16:53 ~ $ irb
>> require 'id3lib'
=> true
24Oct/090

Cappuccino: the jake branch


Those steps assumes you have already setup your Cappuccino source similarly to my environment.

If not, please read previous articles.

Keep in mind that the "jake" branch will be merged into master at some point, and the switch should happen sometime in the upcoming weeks; at least I've been told this way so take this information with a grain of salt.
Therefore: those instructions can turn obsolete at any time and probably are.

If you read my previous article you already know that I'm running Snow Leopard.
At any point do not close the Terminal window until you did every step.

Download Narwhal from GitHub.

cd ~/src/
git clone git://github.com/tlrobinson/narwhal.git

Activate paths and use tusk to install required packages.

source ~/src/narwhal/bin/activate
tusk install jake browserjs narwhal-jsc jack

Build Narwhal JSC:

cd ~/src/narwhal/packages/narwhal-jsc
make webkit
export NARWHAL_ENGINE=jsc

Build Cappuccino:

# if you don't already have a cappuccino source remember to have a CAPP_BUILD defined and
# git clone git://github.com/280north/cappuccino.git ~/src/cappuccino_source
cd ~/src/cappuccino_source
git checkout -b jake origin/jake
rm -rf $CAPP_BUILD
jake
jake debug # if you want also the Debug build
jake docs # if you want up to date local xml/html documentation, requires doxygen

20091024-d7sqyb241537a5fcbjd5i5sui9

Building Cappuccino with jake instead of rake will take some minutes to compile.

I read about some people reporting that the process gets stuck, this didn't happened to me (well it did, but because I was compiling against master by mistake); the jake building process took about 4 minutes to complete on a 2,53 GHz Unibody Intel Core 2 Duo Macbook Pro.

We are going to tweak our $PATH in order to find the new Cappuccino Tools and Narwhal binaries like jake.
Keep in mind that you probably already have them in /usr/local/bin so we must adjust our PATH with some precedence.
Old utilities in /usr/local/bin aren't modified by this procedure, since the jake task that should install them is currently commented.

Here is my .bash_profile related code:

# CAPP_BUILD, directory for built Cappuccino framework.
export CAPP_BUILD="$HOME/src/cappuccino_framework"
 
# Cappuccino jake branch.
PATH="$CAPP_BUILD/Release/CommonJS/objective-j/bin:$PATH"
PATH="$HOME/src/narwhal/bin:$PATH"
 
# Finalize PATH
export PATH=$PATH

This will make capp, nib2cib and press available on command line.

Reopen Terminal and test:

03:52 ~ $ which capp
/Users/kain/src/cappuccino_framework/Release/CommonJS/objective-j/bin/capp
22:42 ~ $ cd Sites/
22:42 ~/Sites $ capp gen MySecondApp -t NibApplication
/Users/kain/src/cappuccino_framework/Release/CommonJS/objective-j/lib/capp/Resources/Templates/NibApplication -> MySecondApp
# ...

Have fun!

Tagged as: No Comments
23Oct/090

On Cappuccino spriting and why it matters, part 1

sheets4qx
The first working builds of sprited Cappuccino were released around 14th October.

If you're familiar with Cappuccino this means frameworks that include images/themes and also entire applications can be sprited.
This will be the default behaviour for Cappuccino applications, we'll talk about advantages in a bit, first, a word about spriting if you're wondering what is it.

Developed around mid-1970, spriting is a technique still used today; if you're into CSS you might have used it.
If you're old enough and curious you may have also dug into a videogame's resource files and you may have seen images composed of multiple little images in a grid.

In fact spriting mostly consists in placing images into a larger image grid, and display the image we are requesting by supplying its correct position in the grid.

What that means in Cappuccino?

Simply put, by using spriting, Cappuccino can squeeze all your project's images in a single file, thus reducing the number of requested/served files.

This is a big gainer, since Cappuccino also makes intensive use of themes; Aristo, the now default theme, counts around 111 files.
They will be only 2 after spriting kicks in for all users in master branch.

The wonderful thing is that it's built-in, you are not required to do this kind of stuff. Couple that with the fact that will be using Narwhal-JSC soon, Cappuccino will be *really* fast.

I will discuss with Francisco Tolmasky (tolmasky) of 280 North about how spriting works in Cappuccino and as usual will report my findings here, in a second part of the article.

Tagged as: No Comments
22Oct/093

Foundation.j and AppKit.j 404

During some Rails integration with Cappuccino I noticed that my Rails' logs were filling fast with 404s.

At first I thought I did something wrong in including my Frameworks, however it turns out that's a normal behaviour by design.

I won't go into the details here on why it's happening, but here's a solution:

Ruby on Rails

To avoid the stack trace generation overhead in Rails simply put down those routes:

  map.connect '/Frameworks/Foundation/Foundation.j',
    :controller => 'application',
    :action     => 'handle_cappuccino_404',
    :conditions => { :method => :get }
  map.connect '/Frameworks/AppKit/AppKit.j',
    :controller => 'application',
    :action     => 'handle_cappuccino_404',
    :conditions => { :method => :get }

Then in your handle_cappuccino_404 method you can simply render :nothing => true:

class ApplicationController < ActionController::Base
  def handle_cappuccino_404
    render :nothing => true
  end
end

I think we can't rescue_from ActionController::RoutingError at this point.
Please also read another solution involving Rails Metal below.

Rails Metal

I was thinking about using Rails Metal to avoid even the extra bits:

script/generate metal cappuccino

Then in app/metal/cappuccino.rb:

require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)
 
class Cappuccino
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/Frameworks\/Foundation|AppKit\/Foundation|AppKit.j/
      [200, {"Content-Type" => "text/html"}, []]
    else
      [404, {"Content-Type" => "text/html"}, ["Not Found"]]
    end
  end
end

No need for the method described above: you can safely remove routes and the 404 handling method from your controller.

Sinatra

To avoid it in Sinatra:

# from http://github.com/mchung/cappuccino_sandbox
get "/Frameworks/AppKit/AppKit.j" do
  [404, {}, "Look elsewhere"]
end
get "/Frameworks/AppKit/Foundation.j" do
  [404, {}, "Look elsewhere"]
end
Tagged as: 3 Comments