Thursday, April 29, 2010

Getting Data: Real-Time Quotes

As promised I'll discuss the options for real-time data... it seems that getting real-time data has become increasingly easier and some brokerages actually have APIs that help you get real-time data or they allow you to download small interval historical prices (minutes, seconds or even ticks).

In this post I'll discuss how to get free "real-time" data and how you can use that data to build your own historical prices database. Yahoo offers delayed quotes (up to 15 minutes), but it also has "real-time" data and we can take advantage of that with a little programming.

First I'd like to introduce you to the Yahoo "API", which is nothing more than a collection of tags. You can see all the available tags here: http://www.gummy-stuff.org/Yahoo-data.htm

Getting real-time quotes is much like getting historical data from Yahoo: you will build a URL with the tags you're interested in and you'll use that URL to get a CSV file containing a quote. Here is how to construct a URL:

http://finance.yahoo.com/d/quotes.csv?s= + STOCK_SYMBOL(S) + &f= + TAG(S)


Let's try it with one symbol and one tag: the symbol will be "GOOG" (Google) and the tag wil be "g" (day's low):

http://finance.yahoo.com/d/quotes.csv?s=GOOG&f=g


If we want to get multiple tags we can just string them together (will return a CSV file with the high and the low of the day):

http://finance.yahoo.com/d/quotes.csv?s=GOOG&f=hg


Let's fetch a CSV file containing Google's high and low values for the day on the first line and Microsoft's high and low values for the day on the next line:

http://finance.yahoo.com/d/quotes.csv?s=GOOG+MSFT&f=hg


Here is an example that downloads all the data for the specified tags and prints them to the screen alongside their description:

// A dictionary with tags where the key is the tag
// and the value is the description of the tag.
private Dictionary _tags;

private void DownloadData(String symbol)
{
string url = String.Format(
"http://finance.yahoo.com/d/quotes.csv?s={0}&f=", symbol);

//Get page showing the table with the chosen indices
HttpWebRequest request = null;
DFDataSet ds = new DFDataSet();
Random rand = new Random(DateTime.Now.Millisecond);
try
{
while (_running)
{
foreach (String key in _tags.Keys)
{
lock (_sync)
{
request = (HttpWebRequest)WebRequest.CreateDefault(
new Uri(url + key));
request.Timeout = 30000;

using (var response = (HttpWebResponse)request.GetResponse())
using (StreamReader input = new StreamReader(
response.GetResponseStream()))
{
Console.WriteLine(String.Format("{0} {1} = {2}",
symbol, _tags[key], input.ReadLine());
}
}
}
Console.WriteLine(Thread.CurrentThread.Name + " running.");
Thread.Sleep(60*1000); // 60 seconds
}
}
catch (Exception exc)
{
Console.WriteLine(exc.Message);
}
}


At this point you should have everything you need to build your own historical database:
1. Download the CSV file with all the tags you're interested in.
2. Open a connection to your database.
3. Bulk insert the CSV file into your database.

9 comments:

  1. hey Kiril - I was just looking for this, this is great. I got the data sorted now, I'm going to write a very simple app to do some automated learning and predicting.

    I'm a coder with a phd in a machine learning applications - i'll post anything i find in my investigations here if you want to exchange some ideas.

    ReplyDelete
  2. @Anthemius, I'm glad this is helpful!

    ReplyDelete
  3. привет Кирилл, у тебя есть какой либо майл чтобы послать сообщение?

    ReplyDelete
  4. Hi Kiril, what programming language do you use ? Is looks like some JScript?

    ReplyDelete
  5. Kiril, may you publish this code to Git , I want to play with it.

    ReplyDelete
  6. Kiril, may you publish this code to Git , I want to play with it.

    ReplyDelete
  7. Hi Kiril, what programming language do you use ? Is looks like some JScript?

    ReplyDelete
  8. I'm suddenly seeing Yahoo providing only something like 90 days back. I need more like 2 years which I was getting OK until I think yesterday. Suggestions for an alternative?

    ReplyDelete