Troubleshooting Advanced Web UI Automation Issues

One of the more common issues you might encounter is a website with many different elements on the page, but no unique identifiers, and elements are all of the same type or class. This happens often when trying to identify menus or menu items, or tabular data, among others. Here is an example page where there is a table of data, but every element in the table (columns, rows, data) all have the same tag, with no other identifying information. The data in each of the table cells is simply created with a <td> tag and has nothing else to it that can be used to identify a specific item in the table.

Image

This is an issue if you only need to grab a certain piece of information from this table, or if it’s a menu item you are trying to click on that you can’t accurately identify to take you to the next page. 

Solutions

Let's imagine that we only need to grab the data from the fourth row of the fourth column.  That value would be Definiebas3 in the above table. How can we grab that data when we can't accurately identify it?

1. Use the Restrict to Specific Match Property of the Web Browser - Get Value Action

Many of the web browser actions have a property to try to locate a specific match number of the web element we specified by either tag or attribute. Many times, you may see multiple items that have the same HTML information, and we can simply say to choose the second match from the list of matches found. In this instance, it located the data in column 4, row 4 by match 32. 

Image

This works great if the matching elements always load in the same order, which unfortunately isn't always the case. The order of the page elements can change based off things like screen size, position, items moving on the page, or extra items being loaded into the page. When that happens, it may not always be the same match number, and the automation will end up capturing the wrong information since it is no longer match 32, in this instance. 

2. Capture Entire Table

This obviously is specific to the generic elements being that of a table, as above, but can be a great way to get the information we need in situations where we can simply extract everything, and then use other methods to parse and find the information needed. The easiest way to extract the entire table is to either use the Automate Recorder, or the Web Browser - Extract Table action.

The recorder is easiest but can sometimes have issues getting precise enough to locate the table boundary for capture due to the cursor shape. If that is the case, you can try the manual select icon within the Extract Table action for more precise location. The table boundary can be hard to find as there is no content in that HTML element, so it is typically a few pixels around the edge of the visible table. 

Image

This will attempt to save all the elements of the table into an Automate Dataset.

Image

Image

We can then parse this dataset using our standard set of text and looping actions within Automate to pull out whatever data is needed.

Another option that can work well if you are not able to extract the data as a table, is to extract the entire HTML source of the page itself. You can do this either via the Web Browser - Extract Source action, or you can simply make a regular REST API Get request to the URL of the page, which should return the source in JSON format. 

Image

You can then use the text actions to search for your information by keywords or using a Regex pattern to extract the needed data. 

The next issue, dealing with dynamic elements, has become more common as web developers have begun using newer coding methods such as AJAX requests, dynamic triggers, and JavaScript to load items on a webpage dynamically. This can make identifying the elements on the page challenging because they may have not actually loaded into the HTML for the Automate Recorder to read and locate them. This could be something like a login button not becoming clickable until the username and password fields are filled in, or menu items only loading after the top-level menu has been selected.

Image

As you can see, each time a menu item is selected, it dynamically loads the content for the next level of the menu, making it impossible to simply select the item you want if it is buried multiple levels deep, as that HTML element hasn't been loaded for the recorder to capture. 

Solutions

Working with dynamic elements can be challenging, and often there can be many ways of dealing with them, depending on the type of element you’re trying to interact with. In this example, we have a menu that loads dynamically as each level is accessed, and we want to select the "CSV" option at the end to download a CSV file. To interact with selections in the bottom most level, we have a couple of ways we can find success.

1. Multiple Recorder Steps

Rather than try to interact directly with the lower most selection (CSV) in the menu tree, we can instead select each item in the tree until we get to that level to finally select the item we want. Depending on how the menu loads, this method may not work in every case, but does work for our example here. 

Image

Because the menu levels stay open even when not in focus, we are able to select each level individually and select it with the recorder. It will then click on each of the items in the menu tree until it gets to the CSV option at the end, like how a user would interact with the menu directly. 

2. Keyboard Shortcuts

When we cannot identify the exact element we need to click on or manipulate, one of the easiest solutions is to see if we can utilize keyboard shortcuts to navigate to the area we need. Many times, simply utilizing the tab key and/or arrow keys to move between items, along with the enter key, can get us where we need to go. Or you may be able to use a combination of keyboard shortcuts to open the secondary menu, followed by a recorder step to interact with the lower most menu item directly once it has been opened.

Here, we can simply use a combination of tabs to get into the menu, followed by arrow keys to navigate through the sub-menus, and finally use the enter key to select the item we want ("CSV") from the list. 

Image

Another common use case for web automation is filling in data into various form fields. This could be for shopping or e-commerce sites, portals for vendors or partners where you need to input purchase or contract data, or perhaps for your company CRM or ERP systems where you need to manually input data. Most of the time, Automate is able to correctly identify and fill in these fields with no issue.

However, there are cases where the inputting of data into a field will trigger something else on the page to update. Having Automate manually set that information puts the data into the field on the page, but it does not trigger the page to update or refresh. Most commonly this is caused by JavaScript that is set to only trigger those changes on a click, hover, or enter, which Automate isn't doing when it enters data to those fields. You may encounter this in elements like drop down menu selections or date pickers.

Here is an example where we must select the country of origin for shipping a package via a popular global shipping company. Selecting the country of origin from the drop-down menu will update the fields to capture the required information for that country, like city, postal code, and check boxes for residential or commercial, as well as the various shipment types that can be sent from those countries. Here we can see the form when comparing the country of Argentina against Australia. The fields update to not only capture the required information, but also conform to the naming standards of that information in that country.

Image

Image

The Automate Recorder can correctly identify this drop-down menu. It is identified as a menu item, and we can select to enter data into this field. If I select a different country from the list, such as France, the field will update with that value, but the fields below will not get refreshed. If I then manually select France from the list, it will refresh with the correctly updated fields. 

Image

This is caused by JavaScript that is set to execute with an onChange event. We can verify this in the HTML by using the developer tools and inspecting that element. 

Image

For this JavaScript to get executed, we will need to trigger that onChange event, similar to how a user would manually select something by clicking into that drop-down field and selecting a new item from the list. 

Solutions

Depending on what type of JavaScript event we need to trigger, we may have different options as far as how we can deal with it in our automated task. For example, if the trigger is simply a focusOut, where we simply need the element to lose focus, or a mouseLeave, where we need the mouse to leave that element, we can try using a keyboard shortcut to move to the next field or click into the white space of the page with our mouse. Other times, as we see here, we need other methods to trigger the JavaScript. Just note that not every page will allow you to inject JavaScript into it, so this solution may not always work.

1. Inject JavaScript

Automate has a native action that will allow you to inject some JavaScript code into the browser session. This can be useful for triggering JavaScript events like this to update pages or refresh screen data. Here we will use the Web Browser - Inject JavaScript action to inject our JavaScript code into our browser session to trigger the onChange even after we have filled in the field with our new country.

The first thing we need to do to write our JavaScript is to identify the HTML element we need to trigger the event from. You can typically grab this information by using the browser's developer tools. There are different ways you can reference the element, but usually the easiest is by ID, which we already have from capturing that field in the recorder previously to fill in the new country. The ID in this instance is "orig_Country". 

Image

Now that we have this information, we can write our JavaScript to trigger the onChange event and insert that step after our recorder step to update the menu. Coding JavaScript is outside the scope of this article, but there are numerous free resources on the web that can help you get started. We are going to use the getElementbyId method to get the information about this element by its HTML ID (orig_country). We will then use the dispatchEvent method to tell the browser that the field has changed and trigger the refresh. 

Image

So now we have our three steps to navigate to the page, set the value of the Ship From country to our country of choice (France) and then run our JavaScript to force the page to refresh with the new value. 

Image

The final common issue that we see when working with web automation is that of sites utilizing iframes to embed content. This is most typically seen when dealing with applications that display as a website where the application itself it simply embedded in an iframe of the site. The site itself is usable to the end user, the same as any other webpage, but the Automate Recorder is unable to identify anything within the iframe itself because it does not have access to the DOM.  Here is an example page where it is taking a common search engine and embedding it into an iframe.

As you can see, the recorder is able to identify the items on the main page to select the search engine, input the search text and click the search button. 

Image

If I perform a search on this page, it will display the results in the lower iframe. I can manually click on anything on that page, the same as if I had performed the search myself using that search engine, however, if I use the recorder, the only thing identified is the entire frame itself and none of the content contained within it. It simply highlights the entire frame and labels it as an iframe called "theiframe". 

Image

This can make working within apps or pages that use iframes challenging within our automations as we cannot accurately identify the HTML elements on the pages within the frame so we can interact with them. 

Solutions

Working with iframes in cases where we cannot identify any HTML elements contained within can be one of the most challenging web UI problems to deal with. However, we do have a couple of tools in our Automate toolkit that can assist with this problem.

1. Keyboard Shortcuts

Like our dynamic elements example where we attempted to navigate a dynamic menu via keyboard shortcuts, another great time to use keyboard shortcuts is when dealing with situations where you cannot interact directly with the elements of the page, like if they are in an iframe.  Going back to our example embedded search, I am able to use the tab key (11 times) to get into the search box, then use keyboard inputs to put in my new search term (Fortra), followed by the enter key to get my new results. 

Image

2. X,Y Coordinates

If keyboard shortcuts do not work, we can always fall back on the traditional method of navigation, which is to use the X,Y coordinates of the item we want to interact with on the screen. For this solution, we need to make sure we follow our best practices and always have the same screen resolution set and maximize our window. We can then make use of Automate's input actions to move our mouse and click on a static set of X,Y coordinates on our screen to move about the page.

Here we have an Input - Move Mouse action. With the action properties window of that action open, we can either manually enter the X,Y coordinates of the item on the screen, or we can simply move our mouse to the item and have the coordinates identified automatically for us, which we can then enter into the corresponding boxes within the action. We then follow that by an Input - Click Mouse action to click into the search box. Next, use the Input - Send Keystrokes action to type in the new search term, followed by the Enter key. 

Image

As you can see, working with websites in automated tasks can sometimes be more challenging than initially expected. As different sites were built with different coding methods and technologies, we need to have multiple tools in our automation toolkit to be able to handle these various scenarios.

Automate provides these tools to help users easily develop UI automations with websites. By utilizing the Automate Recorder, along with some of the additional tips and tricks we discussed previously, we can successfully navigate many of the challenges that these troubling websites can bring.