Genetic Programming: C# Reflection Performance

With Genetic Programming it's always good to have a flexible set of functions which one can plug in and out of the framework in order to test out different solution hypothesis.  After asking around what might be a good way to tackle this problem I heard about Dynamic Method Invocation via Reflection. With reflection it's extremely easy to find all the methods of a given class, so if we want to add a new method then all we have to do is write it and let the GP will automatically pick it up. I wrote a little program to test this out...

First I wrote a class containing some functions:

class Functions
{
    private readonly String _symbol;
    public Functions()
    {
        _symbol = "INVALID";
    }

    public Functions(String symbol)
    {
        _symbol = symbol;
    }
    
    #region ConstNumbers
    public double GPConstNumZero()
    {
        LongOperation();
        return 0.0;
    }
    
    public double GPConstNumPI()
    {
        LongOperation();
        return Math.PI;
    }

    #endregion

    #region OneParamFunctions
    public double GPSin(double val)
    {
        LongOperation();
        return Math.Sin(val);
    }
    
    public double GPMACD(double period)
    {
        LongOperation();
        // Do the MACD calculations here
        return Double.MaxValue;
    }
    #endregion

    #region TwoParameterFunctions
    
    // N/A
    #endregion

    private void LongOperation()
    {
        // I'll explain what goes here a little later
    }
}

Then I wrote my test program:

class Program
{
    // A dictionary that will map functions to the number of parameters
    // where the key is the number of parameters and the functions
    // that have the same number of parameters are added in the list
    // corresponding to the key.
    static Dictionary> _functions = 
        new Dictionary>();

    // A stopwatch for performance analysis
    static Stopwatch _stopwatch = new Stopwatch();

    static void Main(string[] args)
    {
        // Start the Dynamic Method Invocation Test
        DyanmicInvocationTest();
        Console.WriteLine("Press any key to exit!");
        Console.ReadKey();
    }

    static void DyanmicInvocationTest()
    {
        // Get all the method information
        Functions fn = new Functions();
        Type typeFn = fn.GetType();
        MethodInfo[] methods = typeFn.GetMethods();
        foreach (MethodInfo method in methods)
        {
            if (method.Name.StartsWith("GP"))
            {
                ParameterInfo[] pi = method.GetParameters();
                if (!_functions.ContainsKey(pi.Length))
                {
                    _functions.Add(pi.Length, new List());
                }

                _functions[pi.Length].Add(method.Name);
            }
        }
        
        // Setup the performance analysis
        int numRuns = 100 * 1000;
        long time = 0;

        // Run the dynamic invocation performance test
        Console.WriteLine("Running dynamic invocation performance test");
        for (int i = 0; i < numRuns; ++i)
        {
            time += DynamicInvoke(fn);
            _stopwatch.Reset();
        }

        Console.WriteLine("Average time dynamic = " + (double)time/(double)numRuns);
        time = 0;

        // Run the normal invocation performance test
        Console.WriteLine("Running normal invocation performance test");
        for (int i = 0; i < numRuns; ++i)
        {
            time += NormalInvoke(fn);
            _stopwatch.Reset();
        }

        Console.WriteLine("Average time normal = " + (double)time / (double)numRuns);
    }

    static Int64 DynamicInvoke(Functions fn)
    {
        Type typeFn = fn.GetType();
        foreach (int key in _functions.Keys)
        {
            //Console.WriteLine(
            //    String.Format("num param {0}, num fn {1}", 
            //    key, _functions[key].Count));
            foreach (String function in _functions[key])
            {
                _stopwatch.Start();
                switch (key)
                {
                    case 0:
                        typeFn.InvokeMember(function,
                            BindingFlags.Default | BindingFlags.InvokeMethod,
                            null,
                            fn,
                            new object[0],
                            null);
                        break;
                    case 1:
                        typeFn.InvokeMember(function,
                            BindingFlags.Default | BindingFlags.InvokeMethod,
                            null,
                            fn,
                            new object[1] { 1.0 },
                            null);
                        break;
                    default:
                        break;
                }
                _stopwatch.Stop();
            }
        }
        return _stopwatch.ElapsedTicks;
    }

    static long NormalInvoke(Functions fn)
    {
        _stopwatch.Start();
        fn.GPConstNumPI();
        fn.GPConstNumZero();
        fn.GPMACD(1.0);
        fn.GPSin(1.0);
        _stopwatch.Stop();
        return _stopwatch.ElapsedTicks;
    }
}

Earlier I didn't show you the LongOperation method of the Functions class because I wanted to explain what I was testing. Since remote method invocation can be expensive, I wanted to see when I was going to be hit the hardest if I'm going to do dynamic method invocation and I contrived two test cases:

1. Invoke all the functions, but only return the constants inside them (e.g. when a lot of the data is pre-calculated or it already exists as constants).
1. Invoke the methods dynamically.
2. Invoke the methods directly.
2. Invoke all the functions, do some calculations and return the constants inside them (e.g. when a lot of the data is being computed at run-time).
1. Invoke the methods dynamically.
2. Invoke the methods directly.

So for the 1st run with no calculations I just ran the  code as you see it, but for the second test I made a small change and let the LongOperation do some mathematical calculations (take the square root of a number):

private void LongOperation()
{
    for (double i = 0.0; i < 1000.0; ++i)
    {
        Math.Sqrt(i);
    }
}

When running the functions with the calculations I found that the performance hit was about 50% (i.e. methods invoked dynamically through reflection were about 50% slower than the methods invoked directly).

Table 1.0 Avg Ticks: dynamic (315.11694) normal (211.98494)

The truly horrifying part was when I ran the functions without any calculations and the performance hit was 3,967%... yes, you read it right! Invoking methods through reflection is nearly a 4,000% slower if you're not doing any calculations, but simply returning constants. See the graph below:

Table 2.0 Avg Ticks: dynamic (102.74611) normal (2.52593)  

So naturally those numbers are mortifying, but there is some hope on the horizon: I found a blog by Gerhard Stephan (no clue who he is) where he discusses how to get a 2000% performance increase over the methods invoked with reflection. I'm still trying to understand what he's doing, but for now it's not looking very promising.

Conclusion:
I try to have as much of the data pre-calculated as possible which reduces my calculation time during the evolution of the strategies and maximizes the evolution iterations, so reflection is not looking very promising for me. If I had a lot more calculations in my functions, then I might consider reflection as a decent alternative. For now I'll keep looking for any ways to get that performance increase and if I can't figure out something within the next couple of days I'll have to drop this idea.