Genetic Program Generated Strategy

I've been experimenting with Genetic Programming for some time now and we're starting to see some interesting results. I recently tried some USD/EUR currency data with 350 days training and 150 days validation, one of the resulting strategies was:

COS(ADD(Low, Close))

In simple words, add the low and the close then take the cosine of the result. A value greater than 0 is a Buy signal, a value less than 0 is a Sell signal, and a value equal to 0 is a Hold signal.

Over the 150 validation days this strategy made 13.7% and projected over a year it yielded around 25% profit. In the last year the USD/EUR has gone up about 9.2% and using that simple strategy is projected to produce 2.5 times more gains than the buy and hold.

Algorithmic Trading and Legos

I am a member of a High Frequency Trading group on linkedin and I found a post that was really interesting:

Statistical Modeling is like building with LEGOS

LEGOS:
1. Square red pieces can be stacked to build towers.
2. Long skinny pieces can be used as roads.
3. Long skinny pieces on top of square red towers can build bridges.
4. Curved skinny pieces can make turning roads or bridges.
5. Tubes, motors, wheels, etc. can make pretty much anything you can imagine.

STATISTICS:
1. Univariate statistics can be used to clean errors out of your data.
2. Correlations of past data with current data can predict future events.
3. Linear Regression can combine many correlations to predict even better.
4. Logistic Regression can predict curved relationships.
5. Simultaneous equations, Non-Linear Regression, etc. can be applied to improve pretty much any business process you can imagine.

THOUGHT PROVOKING QUESTION
Which of the following best describes how is your company is organized?

A. We hire really creative builders who have access to all the pieces?
B. We hire creative people who have only ever seen the red square pieces to tell the builders what to build.
C. We hire builders who have a big sack of Legos with lots of different pieces, but they’re not too interested in building anything they haven’t seen before.
D. We use robots to automatically build “stuff” that we use as bridges. The bridges usually work, but sometimes we fall through.
E. We don’t have any Legos.

Here is the original post: http://www.linkedin.com/groupAnswers?viewQuestionAndAnswers=&gid=86999&discussionID=16118824&sik=1272838667655&trk=ug_qa_q&goback=.ana_86999_1272838667655_3_1

I'm working on filling my bag with Legos!

Getting Data: Real-Time Quotes

As promised I'll discuss the options for real-time data... it seems that getting real-time data has become increasingly easier and some brokerages actually have APIs that help you get real-time data or they allow you to download small interval historical prices (minutes, seconds or even ticks).

In this post I'll discuss how to get free "real-time" data and how you can use that data to build your own historical prices database. Yahoo offers delayed quotes (up to 15 minutes), but it also has "real-time" data and we can take advantage of that with a little programming.

First I'd like to introduce you to the Yahoo "API", which is nothing more than a collection of tags. You can see all the available tags here: http://www.gummy-stuff.org/Yahoo-data.htm

Getting real-time quotes is much like getting historical data from Yahoo: you will build a URL with the tags you're interested in and you'll use that URL to get a CSV file containing a quote. Here is how to construct a URL:

http://finance.yahoo.com/d/quotes.csv?s= + STOCK_SYMBOL(S) + &f= + TAG(S)

Let's try it with one symbol and one tag: the symbol will be "GOOG" (Google) and the tag wil be "g" (day's low):

http://finance.yahoo.com/d/quotes.csv?s=GOOG&f=g

If we want to get multiple tags we can just string them together (will return a CSV file with the high and the low of the day):

http://finance.yahoo.com/d/quotes.csv?s=GOOG&f=hg

Let's fetch a CSV file containing Google's high and low values for the day on the first line and Microsoft's high and low values for the day on the next line:

http://finance.yahoo.com/d/quotes.csv?s=GOOG+MSFT&f=hg

Here is an example that downloads all the data for the specified tags and prints them to the screen alongside their description:

// A dictionary with tags where the key is the tag 
// and the value is the description of the tag.
private Dictionary _tags;

private void DownloadData(String symbol)
{
    string url = String.Format(
        "http://finance.yahoo.com/d/quotes.csv?s={0}&f=", symbol);

    //Get page showing the table with the chosen indices
    HttpWebRequest request = null;
    DFDataSet ds = new DFDataSet();
    Random rand = new Random(DateTime.Now.Millisecond);
    try
    {
        while (_running)
        {
            foreach (String key in _tags.Keys)
            {
                lock (_sync)
                {
                    request = (HttpWebRequest)WebRequest.CreateDefault(
                        new Uri(url + key));
                    request.Timeout = 30000;
                    
                    using (var response = (HttpWebResponse)request.GetResponse())
                    using (StreamReader input = new StreamReader(
                        response.GetResponseStream()))
                    {
                        Console.WriteLine(String.Format("{0} {1} = {2}", 
                            symbol, _tags[key], input.ReadLine());
                    }
                }
            }
            Console.WriteLine(Thread.CurrentThread.Name + " running.");
            Thread.Sleep(60*1000); // 60 seconds
        }
    }
    catch (Exception exc)
    {
        Console.WriteLine(exc.Message);
    }
}

At this point you should have everything you need to build your own historical database:
1. Download the CSV file with all the tags you're interested in.
2. Open a connection to your database.
3. Bulk insert the CSV file into your database.

Hello World!

Since the topic is Machine Learning and Artificial Intelligence applied to trading, why don't we start by looking at how nature deals with trading: http://www.rattraders.com/

I thought it was an interesting concept... they're essentially letting nature evolve trading strategies. While rat's tend to reproduce quite fast it's still quite slow compared to what your average computer can do.

In the spirit of evolution I made my own application that uses Genetic Programming algorithms to create trading strategies and I use them to trade on the stock market. I post my results on twitter in order to get a time-stamp: www.twitter.com/darwins_finches

So far the results have been quite promising: from 09-09-09 to 4-13-2010 we have accumulated about 25% profit when the stocks we trade have only gone up 18.77%. The S&P 500 has gone up 16.76% in the same period, so we're outperforming both the Buy & Hold strategy for our stocks and the S&P 500.