Testing. It’s quite a commonly used word, isn’t it? Especially in technical fields such as IT. Testing any system is always a unique invaluable experience that allows you to bring almost any system or algorithm to perfection (within the limits of available resources and specified criteria). There’s plenty of information about performance testing out there, but it’s not that easy for the uninitiated to understand it at all. Today I want to talk about a complex but very interesting topic of Software Performance Testing in the most simple and understandable words. I will answer the questions, tell you about my real experience, give examples and recommendations. I hope it will be interesting for both techies and not so much :) Let’s go!
1. What is Software Performance Testing?
Let’s start with the theory. Software Performance Testing — is a set of certain actions aimed at obtaining system testing results using various approaches and tools. While conducting such testing, we can check reliability, critical points, scalability, system quality, etc. Simply put, you need to find out how the system works under a certain load, under certain conditions and so on. This is where Software Performance Testing comes in handy.
Before designing any system, it is always necessary to determine in advance what functional and non-functional requirements it must meet (amount of server response time, max degree of parallelism, amount of RAM, CPU capacity, etc.). There are a lot of such requirements. What is more, defining them is not enough, it is important to fully comply with them.
Here is an example. Supposing the system must withstand 1000 connections and 1000 users at the same time, but our network (provider) does not allow this for some reason. Let’s try to understand why this is happening. Looking at all the data of this task, we understand that the system can hold 1000 users at the level of code and architecture, but the network allows only 800 users. The answer is obvious. The system cannot handle 1000 users because the network has an actual bandwidth of only 800 users.
Conclusion: the system does not pass this testing, does not meet the requirements that we set (1000 connections and 1000 users in parallel).
2. What types of Software Performance Testing are there?
There are 4 major types of performance testing:
• load testing;
• stress testing;
• endurance testing;
• configuration testing.
For example, we have created an application and it is necessary to check how it will work if there is a certain number of users in the system or a certain number of transactions in a given period of time. We made 100,000 transactions within 1 day, the load and loading were carried out, the system withstood — this is called load testing.
Stress testing is slightly different. It helps to find out how reliable and stable the system will be at the moment of extreme, critical loads. In other words, how efficient the system is during peak hours.
Example. There is some kind of application that people used before quarantine in the usual way. Let it be a video-conferencing application. But suddenly, many companies started to work remotely and people had to communicate online more. The system is designed for a certain number of users, but now their number has dramatically increased. This is a stressful situation for the application. The same thing happens during stress testing. Suppose the system has to withstand a load of 1 million records, and we are trying to load 2 million there and see how the system behaves under such conditions. In this way, we determine the reliability of the system at an abnormal peak load.
At first glance, load testing and stress testing are very similar. But in fact, stress testing is a given load at a certain time interval, and stress testing is a given load that exceeds the expected maximum, when we go beyond the limits, trying to pass critical points.
Endurance testing is an analysis of how a system can withstand an expected load over time.
Let’s say we know that an app can handle 100 million users per day. But what if you check its stability with the same maximum load not only within one day, but within one week? How will the system behave in this case? As a result of such an experiment there could be found some defects in memory (potential memory leaks), in processes, etc.
Here is a simple example. A laptop works for a long time without rebooting. This continues for a week, two, three. After four weeks of such continuous work, everything starts to freeze: the browser, the system, the video. Such testing has shown how stable the system’s behavior is with the expected current load over a long period of time. We do not reach the peak load, but over time we can see how data processing speed decreases, memory leaks occur, response time increases etc.
In my view, these are the 3 most common types of Software Performance Testing. There is the fourth one — configuration testing. This is traditional performance testing, when you combine different types of testing and try to identify not only weaknesses in the system under certain loads, but also analyze its effectiveness. Simply put, it’s finding out how effective the system is in a combination of certain parameters of a given load. This is work during peak hours, long intervals of time, with the expected result in order to obtain an optimal configuration in which the system can withstand stress, heavy loads and operate stably.
3. How often does the Technorely team conduct Software Performance Testing prior to the implementation of a project?
We always test any system by default, but at the basic level (the minimum criteria that any system must satisfy).
We carry out in-depth Software Performance Testing only when it is necessary, it all depends on the customer’s preferences and the goals of the project (requirements that must be met in a particular case).
4. What are the differences in system testing in different industries?
In general, there are no differences, because testing consists of basic approaches. The only difference is that when testing one system, you need to pay more attention to one type of testing, and less to the others. For example, doing in-depth stress testing but superficial endurance testing. It really depends on the situation.
Let me give you an example of a cryptocurrency exchange project. We had initial requirements from the customer that the product had to meet. They had to be tested before development began.
We created a system with the required parameters, and observed it work in such conditions. Ideally, performance testing is done ahead of time (before development), but it can be done later. However, in the latter case, there is one substantial “BUT”. When testing is carried out later, there is a risk that the system will not correspond to the expected results, so the resources, money, time spent on creating the application will simply go down the drain.
I recommend doing Software Performance Testing in advance and using a combination of several types of testing. How to test is another matter. Usually the standard scheme is used, but there could be cases like with Stellar. In the process of working on this project, we had to use all types of Software Performance Testing in order to get the desired result.
5. Who on the team conducts the testing?
All team members working on a project, including tech lead and team lead, participate in brainstorming. DevOps engineers create an environment in which tests run. Testers are directly involved in testing.
6. What project is the most memorable and why?
The most memorable project is Stellar, because testing alone took us about three months. We had project requirements that we had to meet and we needed to get the expected outcome.
At the beginning it was quite an interesting task, but then, it got more monotonous and complex. Yet we have gained tremendous experience of in-depth testing, because not many companies are willing to do this. Our team spent a lot of time on brainstorming, calculations, various approaches.
Take a look at this example. We have a goal to withstand 1 million transactions in a certain period of time. At first, the system cannot handle even 100,000 transactions. Step by step, we make 200,000 transactions possible. Then we run the test and find that with this configuration, the system can withstand 200,000 transactions, but not 250,000. We try to figure out the flaws and what can be improved. Then we test again. Now the system can handle 500,000 transactions. That’s good, but what else can we do? We look at the type of testing, input and output data, CPU, memory, system bandwidth, etc. Finally, we reach 1 million transactions. But this is not the finish line yet.
The system in this test consumes a certain amount of resources to withstand 1 million transactions. We have reached the top, the system is stable, and it collapses not at the application level, but at the network level. So we start to explore the network. The algorithm in the system is excellent — it copes, what you need is to increase the bandwidth, expand the channels.
Then we look at how many resources are required to withstand the peak load at the moment of stability. It requires a certain number of servers, memory, network traffic, and money. Let’s say it’s $10,000. Next, we tackle the optimization of resources, trying to reduce resource consumption and get a higher efficiency ratio. We reach a minimum in terms of budget and configurations, but this is already the next stage.
7. Was the client satisfied with the final result?
Good question. The client was very happy with the test results, but was not quite satisfied with the system itself. Ultimately, the system did not withstand the initially expected load. Blockchain Stellar, despite different configurations, rewriting and improving the core, code, bases in the system and the application, still did not allow us to achieve certain results. So, we came to the conclusion that this kind of blockchain is not suitable for that purpose. Without checking this out for ourselves, we would never know about this.
8. How long on average does it take to test a particular system? What determines the duration of testing?
It all depends on the requirements. We tested Stellar for about 3 months as we had higher expectations. To achieve the result, we did everything step by step, gradually: first 100,000, then 200,000, 300,000, etc.
9. Does it ever happen that during testing you are looking for one problem, but come across another one?
Well in fact it does, but in software testing this does not often happen. Performance testing opens our eyes on how exactly the current system fits a given load, expectations and requirements. 100,000 versions can be created, but in the end only 500 work. They reveal the real result. Theory does not work 100% here. Testing and analysis is practice. And to get a practical result, you need to spend a lot of time and resources.
10. How are testing conditions created?
Testers run the tests, write test cases, devOps engineers set up the test environment. We need to get a certain result so that we can launch, for instance, 1 million users on the system. For this, there are special tools and services that are needed for load testing, for example: Apache JMeter, Visual Studio, LOT test, etc. So as not to waste time creating your own system, there are ready-made solutions with the automation of various actions.
When working on Stellar, we used both standard tools and custom scripts with a custom scenario, because the tools, like Apache JMeter, unfortunately, do not allow checking the system for compliance with the initially specified criteria.
To sum it all up
I can talk about Software Performance Testing and my experience in it for hours on end. This is an insanely interesting topic with a lot to discover yet. You shouldn’t be afraid if you want to achieve a desired result. Try, experiment. As I mentioned above, testing is based more on theory than practice. You won’t find answers to many questions until you put your hypotheses to the test. This is real knowledge, this is experience, this is a thrill. I can say with confidence that our team will cope with any task in quality assurance. If you need help, please contact us. We are always happy to help!