How to Automate Login to a Website and Download a File

Python, JavaScript, C#, PowerShell, or Script-Free?

Computers are great at repeating mundane tasks so that users don't have to. Traditionally, though, you needed a computer science degree to make them do it. However, robotic process automation (RPA) makes it far easier to record and repeat those tasks, creating new opportunities for non-techies across the organization and taking the heat off the IT department.

Let's look at how to automate a login to a website and download a file. It's a task that users do manually every day to retrieve everything from procedure manuals to bug reports.

Writing a Script to Log In to a Website and Download a File

Traditionally, you'd automate this two-part process by writing a script using one of the popular languages available to developers, such as Python, Java, C#, or PowerShell.

There's a little work involved in using Python to log in to a website and download a file. The most obvious, but least successful way to do it is to use the Python requests library to access the URL in the login form's action attribute. That means manually reading the page source to find that URL and then passing it along with the values for the username and password through a requests.post instruction.

In Java, you can do something similar using the HttpURLConnection class, while C# coders might use the HttpWebRequest and HttpWebResponse classes. Those using PowerShell to log in and download a file might start by opening their browser as an object, using the Navigate action to reach the login URL, passing in the username and password, and then using click() to hit the login button.

These are basic login approaches, though, and prone to failure. Depending on the login form's structure, you might not be able to use them at all. For example, some sites will insist that your request includes url-encoded session tokens, leaving you with a 400 Bad Request response when you fail to pass them.

This is why many scripted attempts to automate website logins fail at the first hurdle.

Challenges With Log In to a Website and Downloading a File

Password Storage and Retrieval

What do you type into the web form at login? Your username and password, naturally. That might seem simple at first glance, but where do you store those credentials? The most obvious place is in the script itself, but that's also the most insecure, because you'll be storing them in plain text. An alternative is to encode them in a separate encrypted file on the system, but then you'll need to decrypt that automatically. That means forcing the user to enter a decryption password.

All these passwords will be locally stored with no central management. What if you or the user want to update your password? What if the user leaves? With this scripting approach, you're on your own.

Scripts also get more complex if the site you're interacting with supports multi-factor authentication (MFA). While MFA is a great way to make a site more secure, it also means getting to grips with OAUTH, the token-based system typically used with MFA, introducing another layer of complexity into your script.

Code Complexity

Then there's the problem of handling multiple outcomes. Scripts are great for this because you can use if-then or case statements to deal with multiple eventualities, but accommodating every single outcome increases the complexity of your code. A robust script must handle every kind of exception, from different response codes in HTTP headers through to login error messages.

All these considerations increase the complexity of your code, which grows as you address more web pages. Each page will have properties that require different code. This will force you to try and build some kind of master script that reads from a list of different parameters for each page, or more realistically to create an entirely new script for each new page that you visit. Either way, each time you add a new page to your list, you will have to tweak and debug the script.

Maintainability

The work doesn't end there. Website developers often change their code. At some point, the class name they were using for an element in their login form will change, or they'll relocate the naming convention for the file that you want to download. This makes your portfolio of scripts brittle and prone to break, which will increase your development overhead.

These challenges remain when you try to download a file once you've logged in. It might be a static file, such as a user manual that rarely changes, but it's more likely to be a regularly updated file. You might have to download the file based on the position of its link in the HTML code, or perhaps by parsing the file's name, which might require some work with regular expressions.

Each language has its own approach here, which generally involves navigating the page's document object model—the hierarchy of different elements making up the page. In Python, you'd use a library like BeautifulSoup, which has its own syntax for parsing HTML elements.

One of the biggest problems you'll face here is timing. In the early days of the web, login pages were simple affairs with static HTML. Today, they often use JavaScript that loads some content after the main body of the page. That can throw off simple scripts that try to read the page as soon as it loads. BeautifulSoup won't see those elements.

All this makes scripting using native classes an intimidating undertaking for all, but the most hardened developers. There are third-party frameworks designed specifically for browsing that will ease the process. Python has twill and webbot. Python, Java, C#, and PowerShell developers can also use Selenium, a library that automates browser actions, making it good for testing and browser automation. These will make things simpler, but they're still scripting solutions. They mitigate some challenges, but not all of them.

Go Code-Free to Log In to a Website and Download a File

The alternative to scripting is to use an RPA tool that captures all these steps graphically. Fortra's Automate provides a step recorder that allows for do-it-yourself UI-driven automation by capturing a user's interactions on browser-based or desktop applications. Each new action is a step that the user can edit at any time using simple dialog boxes without writing any code.

The Automate Recorder enables users to select from over 700 different actions. Users can begin by selecting an 'open web browser' action from the GUI Recorder menu and entering the URL that they want to surf to. The tool automatically waits for the entire page to load, and allows users to enter an optional wait time, just to be sure that everything has loaded before proceeding.

The tool follows the user’s interactions, recording clicks on individual fields in a form. The user can then enter his/her email address, select the password, and click the page's 'log in' or 'submit' button. The tool records these steps separately and does not use static screen coordinates in the process.

Unlike roll-your-own scripting solutions, Automate stores passwords in a secure central repository. It protects these passwords, so that not even administrators can view them, and only authorized users can access them.

• Once logged into the site, the user can simply enter the URL for the file to download or mimic user navigation if necessary. When the browser reaches the file location, they can enter specific mouse clicks and keystrokes to simulate themselves issuing a 'save' command and can even open the file afterwards. The tool also offers scraper functionality that can parse tables, filenames, and link positions on a page.

Users can enter exception handling steps to deal with different header responses and take proactive action such as sending an email if any step of the automation fails, automatically incorporating informative exception information into the email.

• After recording an automation task, users can schedule it to run via a simple workflow, which can contain other automation tasks or stand alone. It can run on any system in the network that the user has access to, and execution can be dynamically load balanced among various systems as well.

One advantage of Automate is that while it's a no-code solution, each step of an action is in human-readable form and can be tested for immediate feedback. This allows users to review and edit the steps where necessary, move steps around, and/or copy them. They can reuse these steps for different sites, altering the parameters and editing steps to quickly customize their automation. Automate's enterprise solution also allows users to share scripts with others via a central database.

If you're a developer, scripting automations like webpage logins and file downloads can be fun. It takes a lot of problem solving and can often feel like you're trying to outwit the website's original developer. But most business users don't have the time or skill for such an undertaking. Graphically-driven RPA helps them to automate things quickly and easily, leaving them to get on with their real work, while the Automate software takes care of the rest.