Archive for January, 2013
I had a discussion about different test automation design considerations with another tester the other day and thought I might share some of what we talked about. I’d love feedback on what you think are good/bad ideas and also what you think is important that I didn’t cover. Also, stay turned as I’m breaking this up into multiple articles. As an aside, i’m intentionally writing this in psuedo code so it is mostly language agnostic and mostly based on the idea/concepts. However, at some point I may amend this with some example code or post it to github when I have some more time.
For some background, the tools and framework we are using:
Selenium Webdriver, Java, Eclipse
One of the first things I do before I start automating is think about the context, if you are starting from scratch (no existing test automation for the app) and the site functionality is already built, I start with automating the most used/critical functionality, for example, you might need to automate the Registration, Login, My Account, Forgot Password, etc…
If you already have test automation for the site, then you will hopefully have created some test automation patterns that make it easy to extend your test automation scripts. Some of the design considerations I am talking about here will make it easier for this to be done
When creating the tests I tend to write my tests from the ‘Scenario’ perspective, however i’m not currently using any BDD framework to create these, but I find that writing tests from a User Scenario perspective is very helpful because you are modeling tests after User Behaviors and thinking of real people who are using your site.
What that means for this test site is that I can have New Users with Promo Codes (i.e they get discounted), New Users without Promo Codes (no discount), I have New Users that don’t like to provide ALL of their information during signup, and others who put everything in, I have Administrators and Operations personnel that need access, and it is from this vantage point that I start to create my automation tests.
I almost always start off with creating a new user first, for the simple reason that the Registration test code is usually a dependency and can be used to chain together other test Scenarios which will be demonstrated below.
I’ll create a test class and call it RegisterNewUserTest()
This class will sort of act like a driver class (think back to CS 101), which is really just a series of mini-workflows and the Registration process abstracted is really 3 different pages/screens (i.e enter demographic info, enter security and billing information, verification & display dashboard)
Since a Product Owner could easily change this structure around, putting some of the information from the first page onto the second, and vice versa, we want to have a design that makes it so that you make as little changes as possible when this occurs, also, if they add a fourth page then we can simply just make another page and put all of our logic in there. So the basic structure starts to come into view
demographicPageTest() ; //all the changes that deal with the demographic information goes here
//fourthPageTest(); this could be added very easily into the workflow if needed
The other common concern is about test data. When you are testing a web application you don’t typically want the test data being created inside the actual test code, that’s because it will make it more difficult to edit or makes it such that if you wanted to run lots of different data through in a loop fashion to test the same test with different data you can not accomplish this. Here’s a use case I tend to think about, if I’m registering a new user, I might have a user from California, a user from Oklahoma, male and female users, users with different billing information. From my experience you can create separate tests for these scenarios, but it’s been more helpful to create different test data that can get fed into the tests.
What i’ve usually done is to create some associated test data in XML, but i’d imagine you can use JSON, Excel, or any other format you prefer.
I might call the test data something similar to the test class I already created, that way it’s understood that this data is relevant to this test class, let’s call it RegisterNewUserTest.xml. Inside this xml would be a series of different types of users. Alternatively, one of the other methods I like is just describe the data that is in the file which is to say something like MilitaryStudentsWithPromo.xml. This file would only contain data that matches the description. Either of these methods is okay, the last one is more descriptive such that if the test fails when it is processing the MilitaryStudentsWithPromo.xml data it is better context on why the test might have failed, i.e Maybe you have a problem with only the Military workflows in the system?
So here is how you might populate the test data in the xml, this should be done with some structure that way you can call up specific pieces of data in the code, OR support the ability to loop through the test data if you want to run different data combinations into the test
Notice that we have two users, with similar structure, but different data. You can use this structure to change the outcome of the test and check for negative scenarios like what if I put a username with invalid characters, or a City that doesn’t exist in the drop down. If this data had lived inline with the code it starts to limit your options on dynamically changing the test all without recompiling the code
This test data can now be passed as reference data to the class / method signatures
In Part II i will go over the following
Refactoring (Dependencies inside of the workflows, think a testers version of dependency injection)
Should you delete the test data or not? If your not deleting the data, then you need unique log in creation information
Should you be using assertions or logging?
One of the patterns that I think deserves more exploration is the PageObject pattern, after doing some research on this, I think this is a complimentary technique that would fit into the design I have presented pretty well, but since I have not tried it yet I’ll have to get back with you when I do.
I’ve actually written a test automation library very close to what I’ve described here and it was a great success for the following reasons:
It was easy to read and follow
It was easy to add new changes (if adding a new test you can reuse alot of the tests that were created by calling them and chaining together another test)
It reduced how much new code needed to be added because it was based on modeling the web app and putting that into reusable classes