While digging in the UE4.22 source code I found some strange files with *.spec.cpp extension. You can find them in Plugins/Online/OnlineSubsystem/Source/Test directory. It appears they are files used for Automation Tests, but their API looks odd, complately different than the documented one. I was trying to find any information about it, but without success, so… I researched this new way of writing tests and I must say it is much more convenient and professional than the old one.

In this article I will show some simple tests written in the old Automation Tests and their equivalents written using the new API.

Edit 25.07.2019 – Epic gave us an official documentation for Automation Spec here. But I still recommend this article to compare side by side an old system with a new one.

Project

Przechwytywanie

The project we are testing is a First Person Template with three enemy Characters that are destroyed immediately after being hit by a projectile. Player must be spawned in a way that it aims to the one of the enemies.

Utils

Every time I write Automation Tests I use some of my utility functions.

Automation Test Flags Mask for easier setting up test mask. More about test masks here.

GetWorld – because in many places we need to get an access to the current game world.

Exit – usually to run test correctly it must be performed on an opened game map and this map should be closed after the test is finished.

PressKey – for the best game behaviour simulation we must simulate a key press.

Simple Test

With all of these utils at our disposal we are ready to write a simple test. For example – I want to test if there are three enemies on the map when it starts. I will simply write the code of a test here and I will write every important thing in a comment.

You might be wondering what is this latent automation command under the comment nr 5. The FExitGameCommand is just an another way of closing the game map.

One thing that always bothers me in Automation Tests is that even if any of Tests fails, the final result of the RunTest function is usually true. The RunTest function should return false only when it will not be able to finish the whole tests set.

The other annoying thing is that for every test we must ensure that World, or any other important variable is available and do that extra check if we can continue the test or not.

There must be a better way…

Simple Test – a new way

The issue we had in the old API is solved here. The game World is obtained before each defined test and is checked using TestNotNull function. If the BeforeEach phase fails tests will not be performed. In this example we have only one test, but you can put as many tests as you want by using It functions. Each of these tests will  open and close the map and check if the game World is valid. Also, the whole structure of the test is more readable and better standardized in my opinion.

Complex Test

For a complex test we do the same check of the number of enemies but for multiple maps. Every map is inside /GameFirstPersonCPP/Maps directory.

To run the same test, but with different parameter sets, the GetTests method must be implemented which collects those parameters. There is a proper logic behind this solution but why can’t we just use good old for loop?

Complex Test – a new way

In the new API we can simply use a for loop to define multiple tests for different maps. The Describe function here is important as it creates a scope for BeforeEach, It and AfterEach set of functions. Without it, there would be three BeforeEach definitions and all of them would run before each defined test (test maps would opened 9 times).

Latent Commands

Now it will get more interesting as we will start using Latent Commands in order to check gameplay mechanics. In short – latent commands are functions that runs across multiple frames. One common latent command is FEngineWaitLatentCommand which makes the test to wait the given amount of seconds.

Unfortunately the way latent commands work is not intuitive. When I was learning about them I thought they would make the whole test to wait until it’s finished, but in fact it just enqueues itself and it ticks after the test body is finished!

It means that everything that must wait for a latent command to finish must be a latent command itself.

Consider we want to test if after the shoot the number of enemies will decrease. The test could look like this:

As you can imagine with more complex integration tests it can get pretty messy. You have to divide your test into multiple latent chunks which will wait for each other. You might already guessed that the new API has solved it much better.

Latent Commands – a new way

To write latent tests in new API we use LatentIt instead of It.

As you can see there are no chunks of test logic floating around the whole file, only one, unified test.

Because this test is run on separate thread it won’t block game logic and it can simply checks interesting us values during the gameplay.

The only issue is that if there is a need to run some logic that must be run on a game thread (like actor iterating), we have to ensure that it will be run on a game thread. AsyncTask function is the easiest way to do it.

However, I still think this is much easier and cleaner way of writing latent tests than in old API.

Conclusion

There you have it – a quick introduction to the new Automation Tests API in UE4. I covered only basics, there are more types of files like *.fake.h and *.mock.h hiding in the Engine/Online/BuildPathServices/Private/Tests directory. It looks like a professional unit testing framework, kinda similar to the Google Test.

You can check the project with all of the described tests from GitHub.

I hope Epic will write some documentation about this new Automation Tests API soon, as it looks really powerful and professional.