YQL and Open Data tables

YQL (Yahoo! Query Language) is a SQL-like language that lets you query, filter, and join data across the web. Open Data tables are YQL plugins (XML) that can mapped onto any web service or source on the internet. Together, they provide developers a simple way to access data via a RESTful web api in XML or JSON format. For our example we use jQuery to retrieve our information as a JSON object.

HTML

jQueryUI’s datepicker is a life saver. HTML5′s input placeholder is a nice little enhancement as well. I commented out the resources the you will need to include in your tag.

<!-- Header References
http://code.jquery.com/jquery-1.5.js
http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.8/themes/base/jquery-ui.css
http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.7/jquery-ui.js -->

<div id="inputSymbol"> 
        <p>Enter Stock</p> 
            <input id="txtSymbol" class="required" Placeholder="Symbol" />   
            <input id="startDate" class="datePick required" type="text"  Placeholder="From" />   
            <input id="endDate" class="datePick" type="text" Placeholder="To"  /> 
        <button ID="submit">Submit</button> 
    </div>     
<div class="realtime"> 
    <div><p>Name</p><span id="symbol"></span></div> 
    <div><p>RealtimeBid</p><span id="bidRealtime"></span></div> 
</div>   
<div class="historical"> 
    <div><p>Date</p><span id="date"></span></div>
    <div><p>Price</p><span id="closeValue"></span></div> 
</div> 

CSS

Used some CSS3 on the textbox for the required field indicator

*{margin:0; padding:0}  body{padding:1em; color:#555; font-family:verdana; text-align:center}

p{padding:0.5em 0; font-weight:bold} input:focus { outline:none; }

input, button{padding:0.4em 0.3em;  margin:0.5em 0em}
input{border:1px solid #999; border-left:1.05em solid #aaa;-moz-border-radius: 15px; border-radius: 15px;}

.required{ border-left:1.05em solid #E8725C;} 

#inputSymbol, .realtime, .historical{ 
    padding:0.5em 0.5em; margin:0% 20%;  
    text-align:left; 
    border-bottom:1px solid #aaa
}
.realtime div, .historical div, .realtime div span, .historical div span{ display:inline-block }
.realtime div, .historical div{width:45%}

#date span, #closeValue span { display:block; color:#666; font-size:90%}
.ui-datepicker { font-size:11px !important} /* skrink datepicker */

jQuery

After we pull our JSON object, we output our columns of interest to their appropriate HTML elements.

var yqlURL="http://query.yahooapis.com/v1/public/yql?q=";
var dataFormat="&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";

$(function() { //Load jQueryUI DatePicker by class name
    $( ".datePick" ).datepicker({dateFormat: 'yy-mm-dd'} );
});
 
$("#submit").click(function() {
    var symbol = $("#txtSymbol").val();
    var startDate=$("#startDate").val();
    var endDate=$("#endDate").val();

    var realtimeQ = yqlURL+"select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20(%22" + symbol + "%22)%0A%09%09&"+ dataFormat;
    var historicalQ = yqlURL+"select%20*%20from%20yahoo.finance.historicaldata%20where%20symbol%20%3D%20%22"+ symbol +"%22%20and%20startDate%20%3D%20%22"+ startDate +"%22%20and%20endDate%20%3D%20%22"+ endDate +"%22"+ dataFormat;

    $(function() {
        $.getJSON(realtimeQ, function(json) {//YQL Request
            $('#symbol').text(json.query.results.quote.Name);//Assign quote.Param to span tag
            $('#bidRealtime').text(json.query.results.quote.BidRealtime);
        });
    });  
    $(function() {
        $.getJSON(historicalQ, function(json) {            
            $.each(json.query.results.quote, function(i, quote) {//loop results.quote object 
                $("#date").append('<span>' + quote.Date + '</span');//create span for each record
            });            
            $.each(json.query.results.quote, function(i, quote) { //new each statement is needed
                $("#closeValue").append('<span>' + quote.Close + '</span');
            });
        });
    });
});

Editable Demo

If anyone is familiar with any free stock market APIs out there, I would be interested in checking it out and taking it out for a test drive.

Good Luck,

Max

First, find what processor you are running on, ie 32 or 64 bit. If you don’t know use this guide http://support.apple.com/kb/ht3696

Second, download and install the appropriate MySQL version  from http://dev.mysql.com/downloads/mysql. In my case it was 64-bit MySQL 5.1.37.

After that you’ll need to install ruby gems from scratch. You can find more details in my other post How to setup RoR+Passenger+MySql+Ubuntu Server on EC2,but here is an excerpt bash script with all you need:

echo Installing ruby gems...
cd /usr/local/src
sudo wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
sudo tar -zvxf rubygems-1.3.5.tgz
cd rubygems-1.3.5/
sudo ruby setup.rb

echo Symlinking...
sudo ln -s /usr/bin/gem1.8 /usr/local/bin/gem
sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

Note: If you find that the gem command still won’t run try this

sudo ln -s /usr/bin/gem /usr/local/bin/gem

Finally, execute the following:

sudo gem update --system
sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config

Note: make sure you set  ARCHFLAGS flag to the appropriate architecture.

Good luck and enjoy your new OS.

I’ve been using swftools, mainly pdf2swf functionality, for a while now and installation always sucked. Google usually provides good results for common problems, but lack of a step-by-step installation guide is lacking.

Main site, http://swftools.org, has pretty good wiki for documentation and a tidy installation walkthrough. Unfortunately for me I had a ton of problems getting pdf2swf to work properly on Amazon’s AWS Ubuntu server (ami-5059be39 – ubuntu-intrepid), but fortunately for you I finally got it working and also put together this guide, which will hopefully save you some time.

Note: swftools version that comes default with aptitude would not work for me hence the need to compile from source.
Note2: this guide assumes the distro is up to date and all system libraries have been installed from Installing RoR on Ubuntu post

Get necessary libraries and extract

wget http://swftools.org/swftools-0.9.0.tar.gz
tar -zvxf swftools-0.9.0.tar.gz
wget http://www.ijg.org/files/jpegsrc.v7.tar.gz
tar -zvxf jpegsrc.v7.tar.gz
wget http://download.savannah.gnu.org/releases-noredirect/freetype/freetype-2.3.12.tar.gz
tar -zvxf freetype-2.3.12.tar.gz

Note: v8 produced errors in swftools install so I grabbed v7, which worked wonderfully.

Now order of install. First we install jpeg:

cd jpegsrc.v7
sudo ./configure
sudo make
sudo make install

Freetype is a little trickier. For install you have to reset cache, set flags, run ranlib (not sure why but it wouldn’t work without for me, if you know I’d love an explanation)

cd freetype-2.3.12
rm -f config.cache
sudo ranlib /usr/local/lib/libjpeg.a
sudo ldconfig /usr/local/lib
sudo LDFLAGS="-L/usr/local/lib" CPPFLAGS="-I/usr/local/include" ./configure
sudo make
sudo make install

If you want to be able to convert pdf files into text you have to install xpdf. Fortunately aptitude does provide us with the right libraries this time.

sudo apt-get install xpdf-reader

And now for the final step:

cd swftools-0.9.0
sudo LDFLAGS="-L/usr/local/lib" CPPFLAGS="-I/usr/local/include" ./configure
sudo make
sudo make install

Big thanks to swftools for making this possible.

Best,
Mikhail

Edit (3/3/2011): Updated code formatting.

Triple checked to make sure IMAP support was enabled, still nothing.

Then I stumbled along a post suggesting to try unlocking captcha on my account (https://www.google.com/accounts/UnlockCaptcha). I went ahead and everything worked fine afterward. Awesome!

Compared to my previous hosting company – hostmonster.com – Amazon is way more expensive. However, I now find that taking care of my server administration needs is much easier and (gasp!) fun.

sudo apt-get update

sudo apt-get dist-upgrade

sudo apt-get install git-core
sudo apt-get install apache2 php5-mysql libapache2-mod-php5 mysql server
sudo apt-get install phpmyadmin

Download WordPress and extract to a directory. I used /home/ubuntu/projectname.

Then sym link this directory to /var/www/projectname. When you are done, edit apache conf file, which is located at /etc/apache2/sites-enabled/projectname, to point to the deploy directory:

DocumentRoot /var/www/<strong>projectname

You can use the 000-default file if you want, but I usually rename it to the project name I’m working on to avoid confusion (or if you have more than one subdomain or website running on the same machine).

Last step, restart apache:

sudo /etc/init.d/apache 2 restart

Good luck.

Edit (3/3/2011): updated code formatting.

1. Prep
I’m assuming you got your Amazon EC2 account setup and lunched the official Ubuntu server AMI (ami-5059be39 – ubuntu-intrepid is the latest one at the time of this writing).

If you haven’t done that check out this great post Starting Amazon EC2 with Mac OS X by Robert Sosinksi. I also highly recommend the Elasticfox Firefox plug-in to manage your EC2 instances.

2. Install

SSH into your server. Go to /home/ubuntu and create setup.sh file. Copy and paste the following:

# Setup Ruby on Rails, MySql, Apache + Passenger
# on Amazon Ubuntu instance (ami-5059be39 – ubuntu-intrepid)
# 10/09/09

echo Updating system...
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt-get install build-essential
sudo apt-get install curl libcurl4-openssl-dev
sudo apt-get install zlib1g-dev libssl-dev libexpat1-dev

echo Installing tools...
sudo apt-get install apache2 apache2-threaded-dev
sudo apt-get install ruby ri rdoc ruby1.8-dev irb libreadline-ruby1.8
sudo apt-get install libruby1.8 libopenssl-ruby libopenssl-ruby1.8
sudo apt-get install mysql-server libmysqlclient15off mysql-client-5.0
sudo apt-get install mysql-common mysql-server-5.0 libmysqlclient-dev
sudo apt-get install libmysql-ruby libmysql-ruby1.8 psmisc

sudo apt-get install subversion
sudo apt-get install git-core gitweb

echo Installing ruby gems...
cd /usr/local/src
sudo wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
sudo tar -zvxf rubygems-1.3.5.tgz
cd rubygems-1.3.5/
sudo ruby setup.rb

echo Symlinking...
sudo ln -s /usr/bin/gem1.8 /usr/local/bin/gem
sudo ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby
sudo ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc
sudo ln -s /usr/bin/ri1.8 /usr/local/bin/ri
sudo ln -s /usr/bin/irb1.8 /usr/local/bin/irb

echo Installing Gems...
sudo gem install sys-proctable --no-rdoc --no-ri
sudo gem install rails --no-rdoc --no-ri
sudo gem install mysql --no-rdoc --no-ri
sudo gem install capistrano rspec rdoc --no-rdoc --no-ri
sudo gem install passenger --no-rdoc --no-ri
sudo gem install haml --no-rdoc --no-ri
sudo gem install hpricot --no-rdoc --no-ri
sudo gem install json --no-rdoc --no-ri
sudo gem install newrelic_rpm --no-rdoc --no-ri
sudo gem install sqlite3-ruby --no-rdoc --no-ri
sudo gem install will_paginate --no-rdoc --no-ri
sudo gem install authlogic --no-rdoc --no-ri
sudo gem install cached_model --no-rdoc --no-ri

echo Installing Phusion Passenger
sudo passenger-install-apache2-module

Save the file and type sudo bash ./setup.sh to start.

Note: –no-rdoc –no-ri makes it so that no gem documentation is installed. Simply remove that line if you want to download documentation files.

Make sure to press Y when prompted. You should have plenty of space to get everything.

Let’s see what this script does now. The first part updates the system, then it installs apache, ruby, mysql server, and necessary libraries. Then we install subversion and git (if you only need one just remove the other. You are using a version control system…right?)

The next part installs ruby gems. I’ve had some trouble with aptitude so I built from source. The script just grabs rubygems source from rubyforge, extracts, and installs it. If you want to do this manually don’t forget to symlink.

Next step is to install your gems. Those listed in the script are the ones that I commonly use when starting out a rails app, but feel free to add/remove which ever you need.

Last part installs Passenger Phusion. The folks at modrails.com made it extremely easy to setup so just follow their instructions. I usually put

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.4
PassengerRuby /usr/bin/ruby1.8

at the end of apache config file (/etc/apache2/apache2.conf). Then create your app-specific config file at /etc/apache2/sites-enabled/client-app
and paste

<VirtualHost *:80>
ServerName 123.123.123.123 #Server ip or www.yourhostname.com
DocumentRoot /home/ubuntu/yourapp/public  # Rails public directory!
</VirtualHost>

Restart apache

sudo /etc/init.d/apache2 restart

And voila!

Edit (3/3/2011): updated code formatting