My first MT5 optimization on fxDreema...and the fisrt issue!

[Ipod]

@l-andorrà how many blocks does your ea have?

[l'andorrà]

@Ipod It's small. No more than 62 blocks.

[Lacone]

Hello everyone.
I would also like to contribute something to analyzing the error.
Because I have had this problem for a while.
I only use Metatrader5.
My EA consists of 162 blocks and includes 3 timeframes and account queries.
1.
When optimizing with real ticks, the optimization stops and the following error is displayed. Optimization from 01/01/2020
“tested with error "no memory for ticks generating (test events error)"
During live trading, with fast ticks and only one EA, the CPU rose to 100%
The problem was the long code and nesting when updating.
With 1min OHLC, the optimization works.

I revised the EA and use the “once per minutes” block before each tree with multiple blocks.
Now the optimization worked for real tick and live trading.
In the backtest there was only a small deviation between the EA with on ticks and once per minutes.

3. After a small change to the EA, the optimization did not work again.
   Now I have created the EA once with fxDreema. And once via the Metatrader Editor (compilation).

I found that the file of the EA in fxdreema is larger than the EA in the Metatrader Editor. The results in the backtest are identical.

The backtest in the Metatrader Editor ran about 50% faster.
When optimizing only 16 tasks, the EA generated by Metatrader worked. The EA generated by fxdreema did not work.
With more than 16 tasks during optimization, the EA generated by Metatrader no longer worked either.

Now I have carried out a test on a different computer.
Previous PC:
16 GB RAM and 4 threads
Second PC:
32 GB and 12 threads

Now the optimization worked with real ticks and 640 tasks during optimization, without any problems.

CONCLUSION
Currently, the PC is the weakest link when it comes to optimizing MT5 with real ticks.
I think fxdreema is a great structured program for logical programming.
I am aware that fxdreema has to generate a much longer code than a hand-programmed EA. This is also the case with competing programs.
But it still has to be applicable.
Perhaps fxdreema can say something about it itself.
Otherwise, I would send the text to support again.
Perhaps it would be possible to provide a list of which blocks and their links should be avoided.
And which ones use the most resources, such as loops and account queries over a longer period of time.

Best regards
Lacone

[l'andorrà]

@Lacone THANK YOU very much for this detailed analysis. Simply priceless!!

[TipsyWisdom]

@l-andorrà said in My first MT5 optimization on fxDreema...and the fisrt issue!:

@Ipod It's small. No more than 62 blocks.

the project may be small, but you are trying to optimize more than 1000 scenarios.

You shouldnt ever be optimizing that many paramaters. In my experience, projects that have that many variables are made to try to find a strategy utilizing data, and it just doesnt work. EAs that work are simple trading strategies that are proven to have worked.

[TipsyWisdom]

As with what has been said though, I have done many things to optimize and speed up my historical data searching;

1. terminate failing EA scenarios...i.e. if my profit drops 10% below the starting balance, it terminates the rest of the scenario from completing.
2. terminate EAs that dont create enough profit over time
3. terminate EAs that fail X # of trades in a row
4. never operate on a tick by tick basis. Always operate on candle closing scenarios, even for check profit settings...i.e. ALL ACTIONS MUST BE CHECKED ONCE PER MINUTE AT THE FASTEST.

[TipsyWisdom]

in order to troubleshoot, I would also recommend not optimizing and just running 1 scenario on live backtest, so you can read the journal.

[Lacone]

Hello.
Unfortunately, the error also occurred on PC2 with 32GB and 12 threads. I didn't change the EA.
The RAM per thread is probably valid.
You can see it in the Windows Task Manager.

For testing, I deactivated 4 threads so that more RAM is available per thread.
The optimization then worked and the error didn't occur.

My next steps.

• Revise the fxDreema program
• When optimizing with long histories, no real ticks, but every tick or 1min OHLC.
• Increase the RAM when you get the chance.

@ TipsyWisdom
Thank you for your tips and information.
I don't use the optimization to find the best parameter, but to test the robustness of the trading system.
But that's more to do with the topic of "How do I create a trading system?"
Like your tips.
One more note. It's not about the trading system. The EA works in live trading and in backtesting and visual backtesting.
It's just about the error "not enough memory for tick generation", which only occurs when optimizing with real ticks and long histories.

[Ipod]

@Lacone how much ram usage is happening when testing in the task manager?

CPU wise there should always be at least one or two cores not being used.

Also ram should be max 98% during the tests, if the ram runs out or all cores are used it can cause problems.

[Lacone]

@iPod
Thanks for the tip.
I noticed that too.
After I deactivated 4 threads, it was about 3.2 GB per thread.