Scan Historical Data for Conditions/Values

[MrDaisyBates]

I'll try to describe this the best I can, but I haven't gotten my exact idea fully developed yet, so it's likely a little vague, but:

I know how to make an EA that runs in real time and checks for conditions as it passes events, but what if I want the EA to scan the last, say 1000 bars, find situations that meet specific criteria, then extract something from it (like the highest/lowest price or something).

How do I specify the candles that are looked over and, since I don't know if the event will have happened one time, five times, zero times, or a hundred times, how does one handle variables in situations like this? Is it even possible? Do I need to make like 100 variables or something, then some how create some loop that puts data into the next one whenever it fills the previous one?

Maybe a more clear example.

Every time 4 bull/bear candles form back to back, store the lowest/highest price into a variable.

My thinking is that, if I wanted a EA, that I loaded up an hour ago, to make a decision today on a market condition from five days ago, how would I specify the candles for it to search through for specific conditions and how do I best handle the variables in this situations?

Thanks, peeps!

[l'andorrà]

@cdwilder1 You definitely need a loop structure looking at the past candles. I strongly recommend you to ask for mr. roar's help. He the best user creating loops.

[MrDaisyBates]

@l-andorrà said in Scan Historical Data for Conditions/Values:

d you to ask for mr. roar's help. He the best user creating lo

Thank you for your reply, bud!

@roar , is there any way I could possibly convince you to entertain the idea of tackling this concept? I'm really interested in loop type stuff because it really seems like a lot of the magic/power in EA building is going to rely on this idea.

Thanks in advance!

[roar]

@cdwilder1 hi!

Here's your example https://fxdreema.com/shared/ov3z3743b
Its... a bit tricky for a new user, I think

[l'andorrà]

@roar Impressive, as always.

[MrDaisyBates]

@roar Thank you so much for your help, bud!

You're absolutely right, though. I've never seen this P1 P2 thing or this box out to the side.

Would it be possible to explain the general logic behind this in words so that I can grasp it. I'm new, exactly as you suggested, but I'm a fairly logical thinker, so I'm not intimidated by it...I just don't understand it yet. Hahahaha.

[MrDaisyBates]

Looking over it again, I don't understand the green V1-V6. Are these just normal variable labels in this context or does the green color suggest something?

I don't know what the white, orange, and yellow areas are, either.

Best I can figure is, you've created a variable called LoopID to identify what pass of the loop you're on? And the V1+1 means that whatever loopid you're on (like your 5th one), you add 1 onto it so that it's now loopid 6, so then it'll run through as that. Hahahaha.

But, yeah. I've got no idea what the P1 P2 thing are, where you got those little boxes from with all the little colored circles, what each colored column means, etc. I'm really intrigued and I feel like I'm pretty much going to be using this in almost every EA I build for the rest of my life, hahaha, so I'm really curious to learn.

And, by no means do you have to hold my hand or whatever. If you have a link to a really good tutorial that explains all this stuff in great detail, that's totally fine with me, too! I pick stuff up well in that context, also. I'm just so new to this, that I don't even know what to search for or how to phrase this stuff. I know a lot of you guys have been doing this for years and probably have some great advice on what's the most effective way to learn this stuff correctly. It's not the first time I've jumped into the deepend prematurely.

Thanks again in advance, guys!

[roar]

@cdwilder1 its a pleasure to help such a fast learner!

The box to the side is just a quick method to modify your variables, without the official "modify variables" block. Open the box by clicking the "Variables" tag in the corner

The green V's are shortcuts for corresponding variables, just to save your fingers from writing the longer variable name. The 3 colored columns are as follows:

• White column modifies the variable before the main block does its comparison
• Orange column modifies the variable only if the result from the main comparison is TRUE
• Yellow column modifies the variable only if the result from the main comparison is FALSE

Blue P's are shortcuts for the comparison values. Just for example, this would record the gap between the MA's into variable "streak", only when MA5 is above MA20:

[roar]

@cdwilder1 lets go through each block from top to bottom, just to clear any misunderstandings (using the same example link https://fxdreema.com/shared/ov3z3743b

Block #2 (once per bar)

• Firstly, its often good to have this block on top of your project, as it saves your CPU when backtesting. You rarely need to run your code on every new tick, especially when youre only interested in the past candles.
• I also reset all the variables here, so the following loop structure resets for every new candle.
• Take exit from the orange output, that activates on the first price updates on each candle. The yellow output would activate for all the other price updates on each candle.

Block #1 (condition) (sorry for the silly numbering lol)

• The condition restricts the loop to max 500 iterations. Otherwise your tester might run into endless loop and crash
• This block also increases the ID number for each cycle. At the beginning, loopid is 0, and it grows by 1 on each cycle (until we break the loop or reach 500)
  "V1+1" is shortcut for "loopid+1", either way works.
• We have 2 variables for the bull/bear status of successive candles, current_is_bull and last_is_bull. Here I save the "current" variable value into the "last" variable, before the "current" value gets updated again in the next blocks. This part is probably a bit difficult to grasp, but hopefully it becomes clear later.
• Take exit from the orange output (TRUE)

Block #3 (condition)

• Here we check the bull/bear status of some history candle, identified by the loopid variable.
• If candle is a bull, modify the variable to 1. Otherwise (so its bear), modify variable to 0.
• Take exit from either of the outputs, orange or yellow

Block #4 (condition)

• This block is a text-based condition, but it behaves just like the "normal" condition.
• Here we check if there are successive bull-bull or bear-bear pairs (so the two variables are either both 1 or both 0).
• If the condition is true, increase streak by 1 and continue to next block with orange exit.
• If false, reset the streak and go back to the beginning of the loop through yellow exit (block #6 is just some "cable management", would make more sense in a bigger project)

Block #5 (condition)

• Here we check if the loop has found 4 successive bulls or bears
• If true, proceed to orange exit
• If false, go to next loop iteration through yellow exit

Block #7 (modify variables)

• This block just finds the highest and lowest prices
• As you can see, we can use variable calculations in any edit field, for example "loopid - 4" for the starting candle ID

Blocks #8 and #9 (draw line)

• Draw lines to the highest and lowest point of each bull/bear streak, just for visualization purposes

[MrDaisyBates]

Thank you so much for the excellent replies, @roar !

It was more than I expected, but exactly what I needed, so I genuinely appreciate it and learned a bunnccccch from it. It made perfect sense. I had to take my time through it, but it was all really clear, honestly. Hahaha. You're a good teacher.

So, if I wanted it to continue this loop (even after it found the most recent bear/bull streak, I could simply loop the output of the draw command in block 9 back to the Pass or condition 1 block and that would continue the loop until it hit the 500 loopid, I assume.

If I wanted it to just pick the highest and lowest points from a streak (not just four blocks of it, since not all streaks will end at exactly 4), then I could add another condition in (after the streak >= 4 condition) that looks for a contrary candle (a bear for a bull streak and a bull for a bear streak) and, if true, then it would draw at the high and low of the (loopid - streak) instead of loopid - 4. Hahaha. I'm sure I'm missing something there, but that's the basic logic at least.

Thanks again for the reply. I really learned a lot! Especially how the little variables tab off the side of everything works. That's going to be a complete game changer. You have NO idea how many formula and modify variable blocks I've been using in past builds. Hahahahaha. Soooooo many.

It does beg another question, though, if you don't mind.

In your example, you have it drawing objects. What if I wanted to store these values? There's no logic in place in MT4/FxDreema that would allow something to the effect of "store output into streak + 1" on the first pass, but if it runs into the condition again, "store output into streak + 2", etc. I would need to have a "streak1", "streak2", etc series of variables and program in some logic in that stores the first encounter into Streak1, then...maybe it changes a bool value to true, then stores the next in Streak2, then a new bool value to true, and it does that up to a set amount of predefined variables? I'm sure that idea would work, but I guess what I'm asking is, is there some functionality like that already built in, or is it something that I'll have to address myself by predefining all these variables in advance?

[MrDaisyBates]

It also seems I'm going to have a lot to learn as far as the text based condition blocks work and stuff. I've seen them used in a few of the example projects floating around and most of them go completely over my head. Yours made sense only because it was that one line and I've been messing with a lot of bool stuff so far, so this idea of "variable == True" is pretty fresh in my mind, so setting two variables equal to one another as a condition makes sense. I guess the rest of it I'll probably be picking up on a need be basis since it's probably not possible to just learn all the possibilities by reading some list or something.

[MrDaisyBates]

Also,

Where you typed "loopid - 4", would "V1 - 4" have also worked?

What if there was a constant 3 that I used as an input in my settings.

Would something like "loopid - constant_3" work?

How about "V1 - C3"? (I'm not sure if constants also get that handy little shortcut like the variables do)

[roar]

@cdwilder1 said in Scan Historical Data for Conditions/Values:

Also,

Where you typed "loopid - 4", would "V1 - 4" have also worked?

What if there was a constant 3 that I used as an input in my settings.

Would something like "loopid - constant_3" work?

How about "V1 - C3"? (I'm not sure if constants also get that handy little shortcut like the variables do)

The C's and V's only work in the side box, I think (not sure, though).
"loopid - constant_3" would work anywhere, that's certain.

About storing the values, I'm not sure what youre after here. Store all the highest and lowest points and their corresponding candle ID ranges? The most efficient way is using an array, I think. Search the forum for array projects, thats a whole new framework to wrap your head around, lol MQL4/5 documentation is great help as well https://docs.mql4.com/array

I modified my example. https://fxdreema.com/shared/J1g1qH0dd

• Now it has constants for the desired minimum streak, and also the loop length.
• The streak can also be longer than minimum
• Minimum/maximum price now accounts also for the candle wicks immediately outside the bull/bear streak. This may or may not be your intention..
• I connected every block back to the loop, so it will now always run through the whole loop until max iterations is reached (while the first version stops at the first match).
• I made this version work by adjusting the streak by +1 in block #5, and +2 when doing the final calculations. To be honest, I'm not sure why this configuration works like this, but it seems to work lol - this is how I build these projects, trying and failing until something works

I probably missed some of your questions here, just remind me in that case

[MrDaisyBates]

Thanks again, man. You're an absolute legend!

It's all made very good sense and everything and I'm still getting a lot from this (especially seeing how you've modified it). I feel confident that I can handle the task myself now. It's been extremely helpful getting a glimpse into the thought process of someone with much more experience in this area and it's helped frame things out for me in a way that I can better use them.

Arrays...I haven't seen/heard of them in context of MT4, but from just now glancing at it, it seems like you might be right. And I know my question is kind of vague. My apologies. Not trying to be cryptic or anything like that. I have some ideas of how I want to use it, but I don't have just one specific way in mind, though I was mainly thinking of using an EA on range bars, isolating streaks could point towards a supply/demand zone, so being able to extract a price from that could be useful. (Especially with the idea of extracting the low or high from the candle BEFORE the streak started, if it's lower/higher, like you intuitively did yourself). I was also interested in storing multiple outputs because of the idea that price could bull streak, for example, multiple times in a row and each one could provide a potentially useful supply level if they've not been broken yet. This means it could/would be useful to store the outputs of, say, the last 10, for example. (or until they're broken by some other criteria in the EA).

I'm sure the loops will be a really helpful part of this, but it's sounding like arrays are also a new framework, as you described it, that I'm going to need to tackle now. Hahahahaha. The joys of learning!

[MrDaisyBates]

@cdwilder1

Oops. Forgot to tag you. Hahaha. @roar

[MrDaisyBates]

I'm also starting to look more into the array stuff, as you'd suggested.

I'm assuming, to work with arrays, it's going to all be custom code stuff? I've not seen any blocks titled with the word "array" in it.

It's not a problem or whatever. Hahaha. Was planning on learning the basics of coding EAs by hand at some point anyways, but like many of us, I kinda got FxDreema to NOT have to do a lot of coding.

[MrDaisyBates]

I'm looking through the shared project again, Roar, and I am curious:

Could you possibly explain the logic behind the streak being set to -1, why if condition five is false streak gets adjusted in block 6 by +1, and the V6+2 in the same block? I'm trying to understand the thought process behind this. I'm sure you have very good reasons for doing it this way, but I don't immediately understand why. Hahah.

Thanks again, bud. Sorry to load you down.

[roar]

@cdwilder1 yeah, working with arrays is usually custom code.
But in essence, an array is just a storage space (a table) of multiple variables. And the number of these part-variables is dynamic, you can "declare" 10000 variables inside the array just as easily as 2 variables - thats the useful thing in arrays.

Could you possibly explain the logic behind the streak being set to -1, why if condition five is false streak gets adjusted in block 6 by +1, and the V6+2 in the same block? I'm trying to understand the thought process behind this. I'm sure you have very good reasons for doing it this way, but I don't immediately understand why. Hahah.

• I dont see where I set streak to -1. I set current_is_bull to -1 in the first block, just because the value has to be something else than 0 or 1. If it was 0 and the loop detects a bear candle in the first iteration, that one bear candle would be counted twice, in a sense.
• I adjusted streak by 1 after my visual testing suggested trying it
  I think the reason is that streak actually counts the successive bars after the first bull/bear candle has been detected -> when there is actually 2 successive bull candles, the streak variable value is only 1 at that point, before making the adjustment...
• Reason behind the +2 adjustment: 1 for the reason I explain above, +1 to extend the id range to the candle before the streak began (reading from right to left), in case there's a candle wick making the extreme point

I find this a bit difficult to explain, I'll make a picture as well Hope you get at least something out of it

[MrDaisyBates]

Gotcha! Yes! The Current is Bull was set to -1 and for some reason, while trying to take it all in, I thought it was the streak one in hindsight. I can completely see why it needs to be something other than 1 or 0 now that you've clarified it.

And that makes sense on the streak adjustment thing, too, now that you've pointed it out. The first time it detects the pattern (two simultaneous candles the same direction), the streak isn't 1, it's 2. Hahaha. So, that makes sense now that you've phrased it that way.

Ah! You make a good point that I didn't even think about--that it's counting from right to left, because as loopid changes, the candle number is increasing (and they increase from right to left, as you said) hahahaha. Now I've got to reorient my mind about how all this stuff works again, because I was instinctively working from left to right whenever I was trying to make sense of this stuff originally. Oof.

Thanks again for all the clarification! It's been really helpful! I don't think I have any other pressing questions at the moment. That might change as I dig further in.

There isn't any critical differences in how this works for MT5 coding or anything, is there?

I've not messed with MT5 in FxDreema at all, but I've noticed in some of your tutorials and stuff that you've used MT5 coding and I feel like I've seen Miro and some others using MT5 also. It's leading me to the assumption that you guys are comfortable in MT4 and helping others in MT4 since it's the most commonly used of the two (I think), but it seems like you guys probably prefer MT5 (otherwise you would present project ideas and stuff in MT4 only since it's the most common).

The reason I ask is because, from starting to look into MT5, it seems much more robust for EA building/testing, which is super exciting to me. I'm still early enough in this EA building journey to transition over this way, I guess I'm just curious if any major part of this doesn't apply to MT5 FxDreema building? Surely it's all pretty much the same on the FxDreema end of things (and different on the custom scripts and stuff).

Thanks for the additional info on arrays, too, btw. That's very beneficial. I've spent a lot of time in Excel previously, so I kind of see arrays as a table now, in essence.

[MrDaisyBates]

@roar

Sorry. Forgot to tag you again. Hahaha.