Introduction to performance testing

This post is to provide an overview of what performance testing is, why it is important and a call to action.

The journey ahead

This blog is part one of a three-part introductory series that will focus on web page performance. In this blog you will be introduced to the basics of software performance testing and encourage you to start measuring the performance of your software.

Part 1 – You’re reading it!

Part 2 – Focusing on web page performance. You will be introduced to multi-page web applications, single-page web applications and standard performance metrics provided by the browser. Additionally, you will be introduced to why the measurement of Salesforce Lightning pages takes a little more effort.

Part 3 – Finally, we land on Salesforce Lightning web page performance specifically. How Salesforce provides a custom metric called EPT, how to use EPT and some hidden gems in Salesforce instrumentation data.

Why should I care?

I would not be surprised if almost everyone reading this has been on a call with a customer service agent who says, “It will just be a few seconds while your data loads”, or similar. Slow customer service can lead to customers leaving, giving bad reviews or advising friends not to use the service.

One of the reasons that time is one of the primary performance goals is that time is literally money. There is also plenty of research on B2C (business to consumers) that show every additional second of page load time leads to increased bounce rates.

Time is an obvious cost for everyone. However, the cost/benefit analysis has to apply across all resources. For example, increased memory usage might require a newer expensive computer and increased bandwidth usage may require costly network roll out.

Back to School

If you have had the pleasure of doing a Software Engineering based degree, odds are you had to do a class where you needed to create a functional design document, a non-functional design document and then build and deliver the software. Performance lands in the non-functional realm of software engineering. I like to think of functional design as the “what” document and the non-functional design as the “how” document. Other notable aspects of non-functional design include reliability, scalability, accessibility and security in addition to performance.

What is performance testing?

In this series we are going to primarily focus on the performance of web pages. The question we ask is “how long does it take from when you click a link or press a button for the web page to show (render) the content?”. An example is going to http://www.bing.com, typing in your search term “Lightning Performance” and clicking the search button. Once you have hit the search button how long does it take for the list of search results to show in the browser. Performance testing measures how long this search result page takes to show the results.

Note that time, how long the page takes to show, is just one of many metrics that performance testing can cover. Some examples of these other metrics include, CPU time, network usage, hard drive usage, RAM utilization and cache hit/miss ratios to name a few.

Performance test types

There are several flavors of performance testing. Similar to functional testing, there is always discussion over the definition of each test type and below is how I define them.

Performance unit testing – Other names I have seen used are micro benchmarking or component testing, depending on the domain. Examples of this are:

Running a single method, like adding an item to a list.
On a blank page render a component, like a multi-select picklist.

Performance regression testing – Ensures that the performance does not get worse over time and meets defined goals. Performance defense is another way to think about the goal of these tests.

Performance load/scale/stress testing – I like to bucket these types of tests together. Primarily as the execution is the same and the goals are slightly different. How I think about these:

Load testing – Execute test to hit a resource goal. For example, call an API at 100 requests per second (RPS) and halt the test if CPU usage is greater than 50%.
Scale testing – Execute tests to hit increasing resource goals. For example, call an API at 50 RPS, then 100 RPS and finally 150 RPS and halt the test if the CPU usage is greater than 80%
Stress testing – Execute tests until the application falls over or the machine becomes unresponsive. For example, call an API at 10 RPS and every five minutes increase the RPS by 10 until the service fails.

Performance resource leak testing – Verify that resources are not leaking during execution. Memory (RAM) is the most common test for web applications. Other examples include file handles, ports and hard disk space.

Performance longevity testing – This is really a mix of performance stress testing and performance resource leak testing. In this case the goal is to find out how long it takes for a resource to leak enough until the application crashes. For a web application leaking memory is a common cause of browsers becoming unresponsive.

Performance testing goals

There are two primary ways of defining the goals:

Budget based – For example, ensure that the homepage takes less than one second to load.
No worse – For example, is this version of the home page slower than the last version?

These goals are often used in the same project. For example, the project has a goal to have the home page load in less than one second. Feature A is to replace widget X with widget Y and is not to have a load time worse than the previous version. Feature B is to add a new feature and they are allowed to be 10% slower to load than the previous version, however it still must load within one second.

The goals are often a matrix and include multiple resource types across multiple machines. This complexity will be explored in a later blog post.

Where to start?

Everything discussed above is only possible with the collection of various resource utilization. One, usually, easy way to start is to collect metrics on time.

How long did it take for an API to return?
How long did the web page take to load?

Great, I have started collecting metrics, what’s next?

Start monitoring this information. There are competing quotes on the value of measuring. In this case, I am proposing that you start measuring so you can set goals.

Create non-functional requirements for your features. If you are an agile house, add them to the user story acceptance criteria.
Create a non-functional requirements document that outlines the project goals.
Engage your customers and product representatives. What is their perception of your products’ performance.

Run performance tests. Look at your integration tests and identify a subset (which could be all) and start running those tests multiple times in series in a controlled environment. A later blog post will discuss setting up an environment that will reduce the noise in the system to improve the signal for resource(s) your performance tests are measuring.

Conclusion

Performance testing is a broad and complex discipline. There are many variants of performance tests, and their matrix of goals is also often and complex. Testing functional requirements can be complex, but at the end of the day they are usually pass/fail. Performance testing and goals setting are great conversations to have with your team and customers.

If you aren’t doing any performance testing, start collecting metrics and use these to inform your goals.

I look forward to you joining me in future blogs on the journey to mature your performance testing practices, with a specific focus on Salesforce Lightning Platform.