One of the things that I have always grappled with is the lack of emphasis placed on test environments. Although several books pay lip service to the topic, there has been nothing concrete, nothing tangible that provides good patterns to use when creating and utilizing test environments ( i would love to be proved incorrect however) Mostly, the test environment verbiage usually speaks to the test environment(s) as some monolithic entity without really speaking to the nuances and the thought process behind creation, build out and usage. You see, there are actually different stages in the software delivery cycle devoted to testing and the ‘QA’ or testing group typically owns one or more of the stages in that process.
There is movement towards a more team focus on quality, which i am definitely an advocate of, but typically it is still incumbent upon QA to provide independent test verification services and that usually means loosely ‘owning’ a particular testing stage(s). As the industry moves more and more towards Agile methodologies and Service Oriented Architectures there is an increasing need to have multiple software projects concurrently queued up in the pipeline with the possibility of service integration occurring once or more in the cycle before deployment. When rapid delivery is the objective, this can create a burden on your test environment, processes, cause scheduling nightmares and expose other gaps or inefficiencies earlier in the life-cycle.
Some of the questions that typically come to my mind when addressing testing environments are:
- What types of testing needs to be conducted? i.e. Performance, Replication, Load Balancing, Redundancy, Functional, Integration, etc…
- How many test stages are there or should there be and what are the triggers for build promotion? In other words, how does software move through the Delivery process?
- How many sub-environments / ecosystems are needed within this particular environment(s) or test region?
- What are the bottlenecks in the current environment or process and how can it be made more efficient?
- Are there opportunities for sharing test environment resources?
In addition to the questions above, I’ve seen multiple variables come into play when discussing test environments, for example
- UAT, Staging, Readiness, QA, Systems Integration, Alpha/Beta, Performance
- Average number of active projects in progress
- Regression testing of existing products or services
I’m currently in the process of expanding on our test infrastructure and process and may be able to speak how we have addressed some of the questions/challenges in the near future. In the meantime, what’s worked for you?