Release Management Pattern for Expert Advisors

The Challenge

As an FX algotrader who is implementing his own Expert Advisers, I make code changes from time to time which must be reflected in the production environment as soon as possible. But running Experts on different hosts with different brokers means that I need to login to each box, update the SVN repo, compile and install the Expert again. What if this task could be automated? So every time I check in a new version, the Expert auto update and restarts itself.

That´s exactly the setup I am going to explain in this post. Once understood, you will be able to implement the same method in your code and have your Experts always running the last version as soon as you check in, sort of release management setup for Metatrader.

Before divining into the actual setup, make sure you familiarize yourself with MQL5Storage and how to register to get it by visiting this link.

Metatrader Setup

Before looking at the code changes, we need to make sure Metatrader is set up properly to update the local repository. We do that by navigating to the MQL4 folder where the SVN root is located and checking out the files again (It might sound weird to check out via the terminal even we check out inside the platform, but it is necessary as the platform does not create the .svn folder).

We do this by checking out the repository by typing:

svn co https://storage.mql5.com/svn/personal/REPO_NAME.

Once the MQL4 folder is checked out, you may test is by typing svn info and get an informative response such as:

Working Copy Root Path: C:\Users\...\MQL4
URL: https://storage.mql5.com/svn/personal/.../MQL4
Relative URL: ^/..../MQL4
Repository Root: https://storage.mql5.com/svn/personal
Repository UUID: ...
Revision: ...
Node Kind: directory
Schedule: normal
Last Changed Author: ...
Last Changed Rev: ...
Last Changed Date: 2018-01-18 17:36:26 +0100 (Thu, 18 Jan 2018)

Next, in the /files folder create a file called SVNUP.bat with the following contents:

svn cleanup ../
svn up ../

This file will be used later when updating the repository from within the Expert Advisor.

Also, create a file called revision.txt with a numerical value representing the current SVN version you work on, in my case I marked the starting value as 1.

If you follow the above steps, the MQL4 folder should have the .svn folder containing the repository information: 

and the Files folder should have two new files, SVNUP.bat and revision.txt:



Let´s move on to the next step...

Shell Execute Function

For the release management tool to work, you must be familiar with ShellExecuteW function available by importing shell32.dll in Windows. As noted in MSDN, this function call allows you to perform an action on a file. We will use this function in our calls for the following actions:

1. Update the local SVN repository using the SVNUP.bat we created in the previous step.


2. Compile the relevant files using a call to Metaeditor /compile

Starting with the first step, in my code I created an hourly timer event to check whether something changed in the repository, sort of pulling behavior. In the Timer I execute the following code snippet:

int HINSTANCE=ShellExecuteW(NULL,NULL,"SVNUP.bat","",StringConcatenate(TerminalInfoString(TERMINAL_DATA_PATH),"\\MQL4\\Files"),0);
if(HINSTANCE<=32)
{
SendNotification(StringConcatenate(IntegerToString(AccountNumber()),": ShellExecuteW SVNUP.bat failed - ",ShellExecuteErrorDescription(HINSTANCE)));
return false;
}

This basically calls the SVNUP.bat file I created in the previous stage which check and updates the local SVN repo. If the return code is smaller than 32, it means some error occurred while calling ShellExecuteW function. If all went well, the local repo has the latest files locally, so we can go ahead and check whether something actually changed.

As mentioned above, I created a text file called Revision.txt where I save a numerical value which tracks the current version. Each time I make a code change worth updating, I change this value, i.e 19 -> 20...

Next, following a SVN update call, I have a function checking whether this version changed by reading the file and comparing the value to the current one in the memory:

int GetRevision()
{
string RevisionFileName="Revision.txt";
if(FileIsExist(RevisionFileName)==false)
    {
    SendNotification(StringConcatenate(IntegerToString(AccountNumber()),": Revision file doesn't exist"));
    return -1;
    }
else
    {
    int RevisionFileHandle=FileOpen(RevisionFileName,FILE_TXT|FILE_READ);
    if(RevisionFileHandle==INVALID_HANDLE)
        {
        SendNotification(StringConcatenate(IntegerToString(AccountNumber()),": Operation FileOpen failed, ",ErrorDescription(GetLastError())));
        return -1;
        }
    else
        {
        int Output=(int)StringToInteger(FileReadString(RevisionFileHandle));
        FileClose(RevisionFileHandle);
        return Output;
        }
    }
}

Now that I have the revision value, I call the following check (worth mentioning that I let the adviser sleep for a minute to make sure the updated files are available already and got synched locally after the svn up call):

Sleep(60000);
int Output=GetRevision();
if(Output==-1)
    return false;
else if(Revision!=Output)
{
    Revision=Output;
    SendNotification(StringConcatenate(IntegerToString(AccountNumber()),": Revision updated to ",IntegerToString(Revision),", initializing experts"));
    if(CompileFile(FilePath)==false)
        return false;

I am checking whether the revision value changed, and if it did, I call CompileFile function which compile the path to the file I work on:

bool CompileFile(string Filename)
{
int HINSTANCE=ShellExecuteW(NULL,NULL,"metaeditor.exe",StringConcatenate("/compile:\"",Filename,"\""),TerminalPath(),4);
if(HINSTANCE<=32)
    {
    SendNotification(StringConcatenate(IntegerToString(AccountNumber()),": ShellExecuteW metaeditor.exe ",Filename," failed - ",ShellExecuteErrorDescription(HINSTANCE)));
    return false;
    }
return true;
}

This is achieved again by using ShellExecuteW and calling Metaeditor, which with the /compile parameter is compiling the input file. After compilation, I let the adviser sleep again a minute before restarting the it:

Sleep(60000);
ChartApplyTemplate(ChartID(),"Template.tpl");

In this case, the template Controller.tpl is applying the same adviser on the current chart, thus triggering a restart with the new code I just created.

For completeness, here is the function I am using to describe the error in case ShellExecuteW fails:

string ShellExecuteErrorDescription(int error_code) export
 {
 string error_string;

 switch(error_code)
 {
 //--- codes returned from shell execute
 case 0: error_string="The operating system is out of memory or resources."; break;
 case 2: error_string="The specified file was not found"; break;
 case 3: error_string="The specified path was not found."; break;
 case 5: error_string="Windows 95 only: The operating system denied access to the specified file"; break;
 case 8: error_string="Windows 95 only: There was not enough memory to complete the operation."; break;
 case 10: error_string="Wrong Windows version"; break;
 case 11: error_string="The .EXE file is invalid (non-Win32 .EXE or error in .EXE image)."; break;
 case 12: error_string="Application was designed for a different operating system"; break;
 case 13: error_string="Application was designed for MS-DOS 4.0"; break;
 case 15: error_string="Attempt to load a real-mode program"; break;
 case 16: error_string="Attempt to load a second instance of an application with non-readonly data segments"; break;
 case 19: error_string="Attempt to load a compressed application file"; break;
 case 20: error_string="Dynamic-link library (DLL) file failure"; break;
 case 26: error_string="A sharing violation occurred."; break;
 case 27: error_string="The filename association is incomplete or invalid."; break;
 case 28: error_string="The DDE transaction could not be completed because the request timed out."; break;
 case 29: error_string="The DDE transaction failed."; break;
 case 30: error_string="The DDE transaction could not be completed because other DDE transactions were being processed."; break;
 case 31: error_string="There is no application associated with the given filename extension."; break;
 case 32: error_string="Windows 95 only: The specified dynamic-link library was not found."; break;
 default: error_string=StringConcatenate("unknown error - ",IntegerToString(
 return(error_string);
 }

Summary

In this post, I introduced the method I am using to monitor and update my Expert advisers remotely across the different brokers I work with. Using this method, you will be able to develop, compile and check in your code locally followed by an auto updated of the remote Experts.

I used an hourly check where I am calling a function to confirm whether the code actually changed using the revision file, but you may want to use shorter timer if you want more frequent checks.

This method saved me time login into different hosts and made it quite simple and convenient for me to check in once and have all of the different experts updated at the same time.