Pattern Select

Image Processing

Allow multiple matches of a found pattern If this option is selected, the pattern may appear more than once in the Search Area. When this option is not selected, the Search Pattern Action Step returns an error if the pattern is found but occurs more than once.

If the coordinates of the pattern found are to be reused subsequently in a different Action Step (e.g. for mouse clicks relative to pattern), the coordinates used in the case of multiple identical patterns found will be the ones for the pattern furthest to the left within the Search Area. If multiple identical search images are located on top of each other in the same position on the left, the topmost image will be selected.

Hide mouse moves the cursor during the search to the upper left corner of the screen to the position with the coordinates x=0 and y=0. By this way unwanted mouse hover effects can be avoided. Independently of this setting the cursor is not included in the search. So also search patterns covered by the cursor symbol are found. After the search the cursor is set back to its original position.

Margin for automatic search area [pixel] Number of pixels around a found pattern for defining the search area automatically. Valid values lie between 0 and 999; the default-value is 100.

Neighborhood Pixel indicates the number of pixels by which the position of a pixel found may differ from the specified position. The default value for this setting is 1.

Please note that it is easier to search for similar images if the value is greater than 1, i.e. the images are not 100% identical based on pixels. The percentage score may vary by some percentage values depending on position though.

Original Desktop Size Screen resolution at the time the image search was last tested. This size is required in order to scale the Search Area (but not the Search Pattern) if the Workflow is used on a device with a different screen resolution.

Pattern Steady check enables the option 'Wait For Pattern Steady Timeout' and checks whether the coordinates of the pattern found change again within this time period. If not, the search is successful and ends; otherwise the search is repeated.

Reduce Score By [%] When testing the Search Pattern settings, matches by this percentage below the Score are also shown. These are marked yellow in the test result window.

In the wizard, this setting is shown under Search Settings and appears as a yellow zone on the Required matching precision (%) slider.

Repeat search until nothing found If the option to Repeat search until nothing found is enabled, the image search will be carried out within the Timeout interval until none of the images being searched for are found anymore.

Score [%] indicates the accuracy in percent by which the pattern found must match the specified pattern in order to be considered found. In the wizard, this setting is labeled Required matching precision (%) under Search Settings and appears as a green zone on a slider.

Search Mode There are three search modes:

Wait for Pattern Steady Timeout [ms] If the option Pattern steady check is selected, a check for whether the pattern found is still in the same position, or whether it still does not appear in the Search Area, is performed according to the interval value entered here (or in the wizard under Search Settings > Check whether found match stays in same location every [time in ms] milliseconds). If this is the case, the search is successful and ends. Otherwise, the search is repeated either until it is successful or the Action Step times out.

A value of 0 means that the position of the pattern found is not rechecked after having been found. The Action Step does not then wait beyond its runtime.

Find Coordinate

If there is at least one other Action Step that uses patterns before the Action Step Search Pattern, the outbound variable Find coordinate becomes the inbound variable for the subsequent Action Steps.

Pattern Case Index returns the case index, which is executed based on a pattern found.

Find Coordinate Top left corner of the pattern that has been found successfully. If a search is made for multiple patterns and they are found, the coordinates returned are of the image on the far left of the series of patterns being searched for as specified in the wizard. If multiple identical patterns are found in the same position on the far left of the Search Area, the topmost one is used.

Found Pattern Name The name of the matched pattern as entered in the Pattern Gallery.

Found Matched Pattern Count Number of found search patterns from the Regular category. The number of matches can consist of matches for different patterns. Matches of patterns of the category Ignore and Exclude are not counted.

Found Matched Pattern Count is 0, if Search Pattern returns an error. Even, if matches for patterns of the Regular category are found.

The Pattern Select wizard is for selecting Search Patterns and the key search settings.

The basic functionality of the Pattern Strip shown here, including all pattern operations, is described in The Pattern Strip Module.

Click OK to confirm the configuration settings or Cancel as applicable. The Test button evaluates the search and displays the results in the Search Results window.

Entering text into the Filter… field hides in stages all patterns in the lower sections of the wizard with names not containing that text. The filter text may appear in any position within the pattern name, but must be continuous.

Example: If the Search Pattern Action Step contains patterns with the names House, House1, Mouse, and Mouse1, the search filter will find all images when use is entered, only House1 and Mouse1 when 1 is entered, and House and House1 in the case of H.

This section lists patterns that are to be found. The number of patterns that can be found is limited to 50.

There are three ways of searching for patterns.

Match only one pattern is successful if exactly one of the selected patterns is found in the Search Area. An error is returned if none or a number of the selected patterns are found.

Match one or more patterns is successful if at least one of the selected patterns is found in the Search Area. An error is returned if none of the selected patterns are found.

Match all patterns is successful if all of the selected patterns are found in the Search Area. An error is returned if none or not all of the selected patterns are found.

If the option Match all patterns is enabled, only the Case branch for the image with top priority is carried out. In the graphical information that is the first search pattern and/or the top Case in the list.

If a pattern appears multiple times, a successful search will occur as above if Allow multiple matches of a found pattern is enabled in the Search Settings. The Action Step will otherwise return an error if the same pattern appears multiple times.

The order of the Cases in the Action Step corresponds to the order of the search patterns in the wizard. The Workflow continues with the steps for the first Case that arises.

Be sure to fill out the DoElse block too. An empty DoElse block causes an error when executed, resulting in the entire Workflow being terminated.

Patterns may be chosen from the Pattern Gallery or created by means of a screenshot (Capture).

Click to select one or multiple patterns and delete or move them as a group. When a pattern is selected, a white tick will appear on a green background in the top right corner. To undo the selection, click the Clear Selection button. A vertical black line indicates where the pattern can be moved to using drag and drop. If Match one or more patterns is selected the first pattern in the list will have a blue border. This image can be used as a reference point for other Action Steps as the Workflow progresses (see Other Uses of Matched Patterns).

One way of defining a pattern is to take a screenshot that shows it.

The function of the Capture Pattern button is described in Adding a Pattern from Screen Capture

All patterns captured using the Pattern Select Action Step are transferred to the Workflow's Pattern Gallery.

If the Pattern Gallery for the Workflow already contains patterns, they can be added to the list of patterns for the Action Step.

The Select Patterns to add window shows all the images that the Workflow's Pattern Gallery contains. A white tick on a green background in the top right corner indicates patterns which are already included in the Search Pattern Action Step. Patterns already included in another category of the same Action Step are not shown.

To add patterns to the search, simply use the mouse to click and select. Use the same method to deselect patterns which should no longer be included in the search.

Click OK to confirm the selection and add it to the pattern list for the Action Step.

If checking for the single occurrence of a pattern that may be part of other patterns, you must exclude patterns containing the searched-for pattern within them from the search. The number of patterns that can be excluded is limited to 50.

Example: Search for the word staff as a noun but only if it doesn’t form part of another term, e.g. 'staffing'. This pattern must be included in the green field among the patterns being searched for. Other patterns containing this term, e.g. staffing or staffed must be added here.

Areas in a single color are not taken into account within the pattern. It is therefore not sufficient in the above example to capture staff with free space at the right edge, because this free space will not form part of the pattern being searched for due to a lack of features.

The patterns selected can be moved between the categories Regular and Ignored. Use the Move to… button in the menu bar to do this.

The settings needed most frequently for all types of search can be set and modified using the Search Pattern Action Step wizard under Search Settings.

The Required matching precision in percent is shown as a green zone on a slider. A yellow zone to the left of the green indicates the range that was entered in the Properties under Reduce Score by [%] to monitor criteria during the test in an advanced search.

You can use either the general Timeout or your own.

If the option Check whether found match stays in same location every [time in ms] milliseconds (Wait for Image Steady Timeout [ms] in the property pane) is enabled, it may make sense to choose a different Timeout depending on this value.

Because the repeated image search has to be completed in an interval defined here within the timeout, the recommended maximum interval value is ½*timeout.

Move mouse to 0,0 while pattern search is active moves the cursor during the search to the upper left corner of the screen to the position with the coordinates x=0 and y=0. By this way unwanted mouse hover effects can be avoided. Independently of this setting the cursor is not included in the search. So also search patterns covered by the cursor symbol are found. After the search the cursor is set back to its original position.

The search area is displayed on the right side of the settings section. Click the button Define manually to define the search area by drag&drop. The button Define automatically automatically defines a search area containing all regular matches visible at the time of the definition of the search area. The ResetTo Full Screen button turns the search area back to full screen.

The Search Area can be set to either a fixed or dynamic (relative) position. To define a dynamic position, use the Relative To picklist, which lists all variables of type Coordinates that are available at this point; these include, for example, Outbound Variables from a previous Search Pattern Action Step in the same transaction. When a coordinate is selected in the wizard as “Relative To”, an offset vector is immediately calculated. While the Workflow is being designed, this points from the selected coordinate to the top-left corner of the Search Area. At Workflow runtime, the calculated offset vector is applied to the real-time value of the Search Area that the coordinate relates to. This means that the Search Area at Workflow runtime is positioned at exactly the same distance from the “Relative To” coordinate as at the time of design.

The option to Repeat search until nothing found repeats the image search until no more search patterns can be found or until it times out. One possible use is to close multiple dialog boxes which may be encountered in an unknown order and position.

If the option to Repeat search until nothing is found is enabled, the maximum Timeout is less than double the time entered here. If multiple patterns are included and the first one is found within the timeout, the timeout period will be extended by the specified Timeout once only. All other patterns must then be found within this remaining Timeout period. Otherwise the program enters an error state and executes the OnError Case. The Timeout is therefore adjusted to the duration of the steps that need to be completed within the Cases.

If the Option to Repeat search until nothing is found is enabled, the following is to be noted with regard to the various search options:

For all modes, the following is applicable:

Pressing the Test button will launch the search independently of Workflow execution and open the Search Results window where the search is evaluated. This can provide hints about possible improvements that could be made to the Search Settings.

The screen content at the time the test was performed takes up the main part of the window Search Result.

A blue border surrounds the Search Area. The area outside the Search Area is shown grayed.

Matches are highlighted in color. The color legend at the right side of the window works simultaneously as a check box panel for switching the display of the different match types on and off. If there are no matches of a certain type, the according check box field is shown grayed.

Search Patterns of the list Match all / only one / one or more pattern(s) found within the defined precision are highlighted green. Matches from the list Ignore all matches within these patterns are highlighted light blue and matches from the list Make sure none of these patterns appear are highlighted red.

A yellow highlighting shows that the search pattern could have been found if the precision had been set lower. This reduced precision is calculated by deducting the Reduced Score from the Score. In each case, the precision of the match is stated.

Matches outside of the Search Area are marked in gray color irrespective of their type. . Patterns with a Color Spot that are found but do not match the Color Spot color at a specific position are marked in blue. You can see the type of the match in the details pane on the right side of the window.

On opening the window the size of the screenshot is adjusted so, that the whole screenshot is visible. You can zoom in to or out of the image by dragging the slider at the right side of the screenshot. It is also possible to do this by mouse wheel. Double click on the slider shows the image in its original size. If the image is enlarged, scroll bars for adjusting the position of the view appear on the right side and on the lower border of the image.

Below the slider the current mouse position in relation to the screen shown in the screenshot is displayed dynamically.

Mouse hover over a match shows the name, a thumbnail and the matching precision as a tooltip.

Clicking a Match on the screenshot shows details to this Pattern on the right side of the window. Clicking a pattern in the tab Pattern at the lower border of the window shows details to all matches to this pattern. Clicking a match in the screenshot only shows details for this single match. Details contain the name, type, size in pixels and a thumbnail of the pattern along with a color-marked listing of the found matches. For all matches, the coordinates of the upper left corner and the matching precision are displayed. The coloring is the same as described above.

Clicking a match in the details pane toggles the highlighting of the according matches in the screenshot. Highlighting of a Match is indicated in the details pane by a colored border and the intensifying of its background color. Clicking a Match in the details pane displays an animated arrow pointing to this match in the screenshot. In the process, the screenshot is resized so it can be seen in its entirety within the window.

The tab Search Results Summary shows a textual overview over the search settings and the search result.

If the search failed because of one single pattern, this pattern is shown in tab Failure Reason.

If mouse clicks are to be performed relative to a matched image and, at the same time, the search is for more than one image, the image that the mouse clicks relate to must be at the top of the list.

If the first image on the list is not found, the mouse click will be inserted relative to the first image from the list that is found.

If the first image on the list that is found occurs multiple times, the image located on the far left of the screen will be used. If there are multiple images in the identical lateral position on the left, the topmost one will be used.

Directly before editing the Mouse click Action Step the reference pattern must be found by performing the Test function for the Pattern Select Action Step. Otherwise the previously found search pattern will always be used as the