What is Performance Testing ?

What is Performance testing?

Performance testing is a type of software testing and part of Software Engineering practice known as “Software Performance engineering”. Performance testing is performed to validate non-functional requirements such as software’s response time and throughput with particular workloads and also to identify any bottlenecks that would prevent from achieving performance goals of software.  Performance testing helps to find out how responsive and stable the software is when it is subjected to increased usage or at peak workloads.  Performance testing helps to validate software quality attributes like reliability, scalability and efficiency in using available hardware resources.

Why Performance testing is required?

Performance testing is not intended to find functional defects but performance testing is equally important compared to functional testing when it comes to success of software.  Performance testing is key to success of software; would you use email software that is too slow? You will not, even though the mailing software is free from functional defects.  With increasing number of web applications, performance testing is getting inevitable.  Performance of website or software has direct impact on the business. Lets say there are 2 shopping sites, web site A takes 10 to 15 seconds to load and web site B takes 2 to 3 seconds, in this case web site A will lose out on business to web site B.  This point has been proved by a survey that most of the online shoppers abandon websites if it takes more than 4 seconds to load and most of them never visit the web site that took more time to load!!! Read the survey

So now you realize how important quality and performance of an application is for sustaining and growing in this competitive world

Performance issues found during performance testing

Performance testing focuses on only on finding issues related to performance goals (response time, throughput, Scalability).  One or more components that affect performance goals (response time, throughput, Scalability) are known as bottlenecks.

Issue with response time – Response time is the time taken by the software to service requests i.e. load or display requested page or form OR process data submitted through a form and displaying acknowledgement page.  For e.g. time difference between hitting enter button on browser address bar till the time requested page loads completely in browser window, is known as response time.  Response time can increase with increase number of concurrent requests.  Response time is the key for success of software as mentioned in previous section “Why Performance testing is required?” of this article.

Issue with Throughput – Throughput is the data transfer rate that the software can send back to user.  Software throughput can drop drastically as number of request is increased gradually due to bottleneck.  Drop in throughput would also mean delayed response.

Issues with Scalability – Scalability is one of the software quality attribute which checks if software works as expected even when load or number of concurrent requests increases to anticipated peak load.  In most of the cases, due to performance bottlenecks software may not be scalable i.e., software may not function as expected or may even crash.

What are Performance bottlenecks?

Performance bottlenecks are one or more components (code, database query, hardware, network device, bandwidth etc.,) that reduce efficiency of software partly or entirely.  Objective of performance testing is to find Performance bottlenecks and retest once the issues causing Performance bottlenecks are fixed.

Performance bottlenecks can be due to one or more of the below mentioned issues.  Most of the bottlenecks found during performance testing would be usually be result of one or more of the below mentioned issues.
-  Functions or Program APIs used consume more resources or they are not efficient
-  Memory leaks
-  SQL Query performance issues, some of the database queries are taking longer time to execute, need fine tuning
-  Database tables are not indexed
-  Un-used Database or network connections that are not closed in timely manner.
-  Deadlocks due to 2 or more processes competing for same hardware or database resources.
-  Network performance issues, issue with Network device, network latency, network congestion or less Bandwidth provided
-  Hardware resources provided are not enough for running software
-  Disk is fragmented as a result disk reads are taking more time.
-  Very frequent updates to log files or database tables that are causing delay in response.

Types of Performance testing

Load testing is a type of performance testing that is conducted more often than other types of performance testing.  Objective of load testing is to check the behavior of the software at normal and anticipated peak load conditions.  Load testing is performed to check if critical functions work as expected and software’s response time, throughputs are with in the acceptable levels.  Load testing is performed by simulating multiple users using the software concurrently.  Load testing is very hard and expensive if done manually; hence they are mostly done with load testing tools.

Stress testing is a type of performance testing that is conducted to check the behavior of the software over and above the anticipated peak load conditions.  Objective of stress testing is to find upper limit or failure point of software due to excess load and to confirm that the software does not crash and does not lead to data loss or data corruption at peak loads.  Stress testing helps to check robustness, error handling and availability of software when subjected to excess load.

Configuration testing is a type of performance testing that is conducted to increase performance of software by tweaking software or hardware configuration.  Many times, minor modification or tweaking in configuration can boost software performance in a big way.  Configuration testing also looks at increasing performance of the application by providing more resources like load balancers, additional memory, co-processors etc.,

Volume testing is a type of performance testing that is conducted to check stability and response of the software when subjected to higher or excess volume of data, search or output data etc., for e.g. Opening or saving a 10 Mb file in MS word or performing a spell and grammar check with word file containing 100K lines of text, are some of the examples of volume testing.  Unlike load testing, volume testing inputs 1 or very few concurrent users but increases workload by increasing the volume of data to be processed or to be data to be generated by the software.

Soak Testing is also know as Endurance tests or longitivity testing.  Software is subjected to expected load conditions over longer period of time to check if software performance deteriorates due to continuous usage or continuous load conditions.  Soak testing also helps to indentify some of the performance issues that might not be found with load or stress testing.  Soak testing is one of the tests that can help to identify defects related to memory leaks, network or database connection leaks, excessive disk usage etc.,

Spike testing is a type of performance testing conducted to find ability of software to withstand sudden increase or surge or spike in load, hence the same spike testing.  Spike testing is very important especially for popular web based applications like facebook, Gmail, yahoo or hotmail etc., where-in load is generated in spikes during morning and evening when most of the users would be logging in all over the world.  Spikes are generated when most of the users perform an action like login or check emails etc., at same time.

Refer to article on Performance test planning article

Paid Performance testing tools

Below is some of the popular performance testing tools used in IT industry.

LoadRunner is a Performance testing tool by HP that is used to for detecting performance bottlenecks with wide range of monitoring and analysis tools it provides. LoadRunner has different tools, Virtual User Generator (VuGen), Load Generator, Controller, Analysis and the AJAX TruClient. LoadRunner supports wide range of technology and applications like Java, Ajax, .Net, Citrix, Silverlight, SOA, Mobile, ERP, Mainframe etc.,

IBM Rational Performance Tester (RFT) is Performance testing tools by IBM, it is an eclipse based performance testing tool.  Rational performance test scripts are not stored as code instead is represented as test tree; each branch represents a request to server or response from the server.  RFT provides option to insert custom Java code to include custom logic into the script. RFT is used for performance testing of web applications or servers.  Provides additional features that can help in identification of bottleneck developed using J2EE.

Open source load testing tools

Below is some of the popular Open Source load testing tools

OpenSTA
is an open source load testing tool by opensta.org, it is a load generation tool for web servers, it supports both HTTP and HTTPS protocols, test scripts are recorded in proprietary language called as SCL (Script control language), it also supports data parameterization.  OpenSTA can generated 100s to 1000s of virtual users, also provides tools to monitor Windows Performance. This is one Open Source load testing tool worth trying.

LoadUI is a open source load testing tool by loadui.org and Sourceforge Project Site mainly for testing web services.  LoadUI can be used for distributed load testing. LoadUI Pro version from SmartBear Software provides server monitoring tools.

Performance test parameters

Below is a comprehensive list of parameters measured during performance testing or load testing.

Physical Memory Usage – Available physical memory at different points of time. Physical memory usage is important to monitor, rapid or constant increase of used memory with constant load might be an indication of memory leak and a bottleneck.

Virtual Memory Usage – Available virtual memory at different points of time.  Rapid or constant increase of used memory can be an indicator of delayed garbage collection or could be an indication of memory leak.

CPU Utilization – Percentage of CPU utilization to run active processes at different points of time.

CPU Interrupts per second – Number of interrupts received by CPU

Disk Usage – Hard disk usage for read and writes at different points of time.

Database connections – Number of database connections opened at any given point of time.

Database Locks – Number of locks on database objects like database table or row or rows at different points of time. More the number of locks or longer durations locks are placed would hit overall performance of the software.

Network Usage – Network bytes total per second, which is number of bytes sent and received at different points of time.

Network bottleneck – Network output queue length, is the number of data packets waiting in output queue to be dispatched.  Bigger the size of Network output queue length implies network traffic congestion OR bandwidth available is smaller than required OR too much of network traffic than required.

Hits per second – Is the number of requests sent to server or web server e.g. a HTML page can contain 10 to 100s of elements like JavaScript, CSS, images, frames, videos etc., each of them will be sent a request.

Bandwidth Usage – Bandwidth usage refers to transfer of data in bytes per second across different networked servers or web services. E.g.  assuming application server and database servers are installed on different servers, now application servers connects to database and send SQL query for execution and gets back query results, all of these refer to Bandwidth usage.

Page faults per second – is a metric to check how frequently page faults occur. More page faults would be more frequent virtual memory access, hence reduced response time.

No. of active sessions – Number of active session are different from concurrent users.  Number of active sessions means the number application sessions that are still active.

Number of open connections – refer to number of network sockets or database connections opened at a given point of time.  Connection Pooling should be used to effectively utilize network or database connections. Connection pooling reduces time delay and resource overheads required for opening and closing connections.

Performance Testing Metrics

Below is the list of mostly common Performance testing metrics captured during most of the performance testing carried out.

Transactions per second (TPS) – is the measure of number of transactions handled by server per second.  Transactions include one or several requests and responses.

Average Response Times – Response time is the time difference between “request to” and “response from” the software.  Response times are measured as “Time to return first byte” or “Time to return last byte”.  Average response time is an average of all individual response times of similar transactions.

Throughput – is the amount of data expressed in terms of kilobyte per second that is sent as part of request or received as part of response.  Throughput indicates size of the data a software can receive or deliver.

Peak Response Times – This metrics indicates the longest time taken for a response.  However, the average response time may be still within the required levels.  Peak response time indicates a potential problem one of the virtual users faced during a particular transaction.  Peak response times could be result of an expensive SQL query for particular virtual user OR the size of the text content sent for a particular virtual user OR big image or other big sized file that had to be sent as response for a particular virtual user.

Error Rates – This is an important performance metric that indicates the threshold or break point of the software or server.  Higher error rates means software is not able to service the most of the requests, so the other requests are getting timed out or the server is not picking up the requests beyond a particular number of requests.  Lower error rates at peak loads indicate robustness and scalability of the software.

Requests per Second – This performance metric indicates the number of request sent to server or software to process, this metrics is important when it is compared in conjunction with other metric like average response time.  We can plot a graph for requests per second and average response time on two co-ordinates and see how response time of the software or server is affected and the peak load that the software under test can handle i.e. at what requests per second the average response time falls below acceptable limit.

Concurrent Users – This performance metric indicates the number of virtual users that are currently active in a load scenario i.e. the virtual user has not completed its run.  The number of concurrent users and requests per second are not one and the same, as some of the concurrent users may be idle due to “think time” or waiting for response for the request that was sent earlier etc.,

Hearty thanks to our new guest writer Jagadeesh Rao, Jagadeesh has 7+ years experience on Performance Testing, he has worked on popular Performance and load testing tools like LoadRunner, Rational Performance Tester, LoadUI, OpenSTA. We would thanks Jagadeesh again for writing a detailed article on Performance testing and hope you will continue to write more of such articles for our software testing blog in future.

Other Interesting Articles:

• QTP interview questions
• All Types of Software Testing
• Software Testing Tools
• Types of software testing
• Smoke and sanity testing

Share this article with friends and colleague if you like it OR Share it with your boss if you got bored :-)

•