Testing

Introduction

Why Do We Test?

What are Test Levels?

What are some Approaches to Testing?

Goals

Objectives

Test Constructs

establish a subject under test establish a context in which to test that subject perform actions on the subject evaluate the results

Figure 1. Basic Test Concepts

Automated Test Terminology

Maven Test Types

Test Naming Conventions

Lecture Test Naming Conventions

Spring Boot Starter Test Frameworks

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope> (1)
        </dependency>

Spring Boot Starter Transitive Dependencies

[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.7.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.7.0:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.7.0:test
[INFO] |  +- org.springframework:spring-test:jar:5.3.20:test

[INFO] |  +- org.junit.jupiter:junit-jupiter:jar:5.8.2:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-api:jar:5.8.2:test
[INFO] |  |  +- org.junit.jupiter:junit-jupiter-params:jar:5.8.2:test
[INFO] |  |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.8.2:test

[INFO] |  +- org.assertj:assertj-core:jar:3.22.0:test
[INFO] |  +- org.hamcrest:hamcrest:jar:2.2:test

[INFO] |  +- org.mockito:mockito-core:jar:4.5.1:test
[INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.12.10:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.12.10:test
[INFO] |  |  \- org.objenesis:objenesis:jar:3.2:test
[INFO] |  +- org.mockito:mockito-junit-jupiter:jar:4.5.1:test

[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.7.0:test
[INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.0:test

[INFO] \- org.junit.vintage:junit-vintage-engine:jar:5.8.2:test
[INFO]    +- org.junit.platform:junit-platform-engine:jar:1.8.2:test
[INFO]    +- junit:junit:jar:4.13.2:test

Transitive Dependency Test Tools

JUnit Background

tests that perform actions on the subjects within a given context and assert proper results test cases that group tests and wrap in a set of common setup and teardown steps test suites that provide a way of grouping certain tests Test Suites are not as pervasive as test cases and tests

Figure 2. Basic JUnit Test Framework Constructs

JUnit - API Annotations

public class MathTest extends TestCase {
    protected void setUp() { } (1)
    protected void tearDown() { } (1)
    public void testAdd() { (2)
       assertEquals(4, 2+2);
    }
}

public class MathTest {
    @Before
    public void setup() { } (1)
    @After
    public void teardown() { } (1)
    @Test
    public void add() { (1)
       assertEquals(4, 2+2);
    }
}

JUnit API - Lambdas

assertEquals(5,2+2);//fails here
assertEquals(3,2+2);//not eval
assertEquals(4,2+2);

assertAll(//all get eval and reported
    () -> assertEquals(5,2+2),
    () -> assertEquals(3,2+2,
        ()->String.format("try%d",2)),
    () -> assertEquals(4,2+2)
);

JUnit 5 Evolution

JUnit 5 Areas

JUnit Platform foundation for launching tests on a JVM — from anything defines TestEngine API for JUnit and 3rd party TestEngines and Extensions to use JUnit Jupiter ("new stuff") evolution from legacy provides TestEngine for running Jupiter-based tests JUnit Vintage ("legacy stuff") provides TestEngine for running JUnit 3 and JUnit 4-based tests

Figure 4. JUnit 5 Modularization

JUnit 5 Module JARs

JUnit Vintage Basics

JUnit Vintage Basic Syntax Example Output

16:35:42.293 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - setUpClass (1)
16:35:42.297 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - setUp (2)
16:35:42.297 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - 2+2=4
16:35:42.297 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - tearDown (2)
16:35:42.299 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - setUp (2)
16:35:42.300 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - 1+1=2
16:35:42.300 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - tearDown (2)
16:35:42.300 INFO ...testing.testbasics.vintage.ExampleJUnit4Test - tearDownClass (1)

JUnit Jupiter Basics

JUnit Jupiter Basic Syntax Example Output

16:53:44.852 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - setUpClass (1)
(3)
16:53:44.866 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - setUp (2)
16:53:44.869 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - 2+2=4
16:53:44.874 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - tearDown (2)
(3)

16:53:44.879 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - setUp (2)
16:53:44.880 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - 1+1=2
16:53:44.881 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - tearDown (2)
(3)
16:53:44.883 INFO ...testing.testbasics.jupiter.ExampleJUnit5Test - tearDownClass (1)

JUnit Jupiter Test Case Adjustments

Test Instance

Shared Static State - PER_METHOD (default)

@TestInstance(TestInstance.Lifecycle.PER_METHOD) //the default (1)
class StaticShared {
    private static int staticState; (2)
    @BeforeAll
    static void init() { (3)
        log.info("state={}", staticState++);
    }
    @Test
    void testA() { log.info("state={}", staticState); }  (4)
    @Test
    void testB() { log.info("state={}", staticState); }

Shared Instance State - PER_CLASS

@TestInstance(TestInstance.Lifecycle.PER_CLASS) (1)
class InstanceShared {
    private int instanceState; (2)
    @BeforeAll
    void init() { (3)
        log.info("state={}", instanceState++);
    }
    @Test
    void testA() { log.info("state={}", instanceState); }
    @Test
    void testB() { log.info("state={}", instanceState); }

Test Ordering

...
import org.junit.jupiter.api.*;

@TestMethodOrder(
//        MethodOrderer.OrderAnnotation.class
//        MethodOrderer.MethodName.class
//        MethodOrderer.DisplayName.class
          MethodOrderer.Random.class
)
class ExampleJUnit5Test {
    @Test
    @Order(1)
    void two_plus_two() {
        ...
    @Test
    @Order(2)
    void one_and_one() {

Explicit Method Ordering is the Exception

Assertion Basics

Assertion Libraries

JUnit Assertions

import static org.junit.jupiter.api.Assertions.*;
...
        assertEquals(expected, lhs+rhs); (1)

Hamcrest Assertions

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.*;
...
        assertThat(beaver.getFirstName(), equalTo("Jerry")); (1)

AssertJ Assertions

import static org.assertj.core.api.Assertions.*;
...
        assertThat(beaver.getFirstName()).isEqualTo("Jerry"); (1)

import static info.ejava.examples.app.testing.testbasics.Assertions.*;
...
        assertThat(beaver).hasFirstName("Jerry");

Example Library Assertions

package info.ejava.examples.app.testing.testbasics.jupiter;

import lombok.extern.slf4j.Slf4j;
import org.hamcrest.MatcherAssert;
import org.hamcrest.Matchers;
import org.junit.Assert;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

@Slf4j
class AssertionsTest {
    int lhs=1;
    int rhs=1;
    int expected=2;

    @Test
    void one_and_one() {
        //junit 4/Vintage assertion
        Assert.assertEquals(expected, lhs+rhs); (1)
        //Jupiter assertion
        Assertions.assertEquals(expected, lhs+rhs); (1)
        //hamcrest assertion
        MatcherAssert.assertThat(lhs+rhs, Matchers.is(expected)); (2)
        //AssertJ assertion
        org.assertj.core.api.Assertions.assertThat(lhs+rhs).isEqualTo(expected); (3)
    }
}

Assertion Failures

java.lang.AssertionError: expected:<3> but was:<2> (1)

Assertion Failure Context Output

java.lang.AssertionError: math error expected:<3> but was:<2> (1)

Testing Multiple Assertions

JUnit Jupiter Multiple Assertion Support

@Test
void junit_all() {
    Assertions.assertAll("all assertions",
        () -> Assertions.assertEquals(expected, lhs+rhs, "jupiter assertion"), (1)
        () -> Assertions.assertEquals(expected, lhs+rhs,
            ()->String.format("jupiter format %d+%d!=%d",lhs,rhs,expected))
    );
}

Asserting Exceptions

JUnit Jupiter Exception Handling Support

import org.junit.jupiter.api.Assertions;
...
    @Test
    public void exceptions() {
        RuntimeException ex1 = Assertions.assertThrows(RuntimeException.class, (1)
            () -> {
                throw new IllegalArgumentException("example exception");
            });
    }

AssertJ Exception Handling Support

Throwable ex1 = catchThrowable( (1)
        ()->{ throw new IllegalArgumentException("example exception"); });
assertThat(ex1).hasMessage("example exception"); (2)

RuntimeException ex2 = catchThrowableOfType( (1)
        ()->{ throw new IllegalArgumentException("example exception"); },
        RuntimeException.class);
assertThat(ex1).hasMessage("example exception"); (2)

AssertJ Exception Handling Support (cont.)

assertThatThrownBy( (1)
        () -> {
            throw new IllegalArgumentException("example exception");
        }).hasMessage("example exception");

assertThatExceptionOfType(RuntimeException.class).isThrownBy( (1)
        () -> {
            throw new IllegalArgumentException("example exception");
        }).withMessage("example exception");

Asserting Dates

AssertJ Date Handling Support

import static org.assertj.core.api.Assertions.*;
...
    @Test
    public void dateTypes() {
        assertThat(beaver.getDob()).isAfter(wally.getDob());
        assertThat(beaver.getDob())
             .as("beaver NOT younger than wally")
             .isAfter(wally.getDob()); (1)
    }

Hamcrest Date Handling Support

<!-- for hamcrest date comparisons -->
<dependency>
    <groupId>org.exparity</groupId>
    <artifactId>hamcrest-date</artifactId>
    <version>2.0.7</version>
    <scope>test</scope>
</dependency>

import org.exparity.hamcrest.date.DateMatchers;
import static org.hamcrest.MatcherAssert.assertThat;
...
    @Test
    public void dateTypes() {
        //requires additional org.exparity:hamcrest-date library
        assertThat(beaver.getDob(), DateMatchers.after(wally.getDob()));
        assertThat("beaver NOT younger than wally", beaver.getDob(),
                   DateMatchers.after(wally.getDob())); (1)
    }

Mockito Basics

Test Doubles

Mock Support

Mockito Example Test

@Test
public void listMap() {
    //define behavior of mock during test
    when(mapMock.get(stringArgCaptor.capture()))
            .thenReturn("springboot", "testing"); (1)

    //conduct test
    int size = mapMock.size();
    String secret1 = mapMock.get("happiness");
    String secret2 = mapMock.get("joy");

    //evaluate results
    verify(mapMock).size(); //verify called once  (3)
    verify(mapMock, times(2)).get(anyString()); //verify called twice
    //verify what was given to mock
    assertThat(stringArgCaptor.getAllValues().get(0)).isEqualTo("happiness"); (2)
    assertThat(stringArgCaptor.getAllValues().get(1)).isEqualTo("joy");
    //verify what was returned by mock
    assertThat(size).as("unexpected size").isEqualTo(0);
    assertThat(secret1).as("unexpected first result").isEqualTo("springboot");
    assertThat(secret2).as("unexpected second result").isEqualTo("testing");
}

BDD Acceptance Test Terminology

Alternate BDD Syntax Support

Example BDD Syntax Output

JUnit Options Expressed in Properties

junit.jupiter.displayname.generator.default = \
  org.junit.jupiter.api.DisplayNameGenerator$ReplaceUnderscores

Review: Unit Test Basics

Review: POJO Unit Test Setup

@DisplayNameGeneration(DisplayNameGenerator.ReplaceUnderscores.class) (1)
@DisplayName("Standard Tipping Calculator")
public class StandardTippingCalculatorImplTest {
    //subject under test
    private TipCalculator tipCalculator; (2)

    @BeforeEach (3)
    void setup() { //simulating a complex initialization
        tipCalculator=new StandardTippingImpl();
    }

Review: POJO Unit Test

@Test
public void given_fair_service() { (1)
    //given - a $100 bill with FAIR service (2)
    BigDecimal billTotal = new BigDecimal(100);
    ServiceQuality serviceQuality = ServiceQuality.FAIR;

    //when - calculating tip (2)
    BigDecimal resultTip = tipCalculator.calcTip(billTotal, serviceQuality);

    //then - expect a result that is 15% of the $100 total (2)
    BigDecimal expectedTip = billTotal.multiply(BigDecimal.valueOf(0.15));
    then(resultTip).isEqualTo(expectedTip); (3)
}

Review: Mocked Unit Test Setup

@ExtendWith(MockitoExtension.class) (1)
@DisplayNameGeneration(DisplayNameGenerator.ReplaceUnderscores.class)
@DisplayName("Bill CalculatorImpl Mocked Unit Test")
public class BillCalculatorMockedTest {
    //subject under test
    private BillCalculator billCalculator;

    @Mock (2)
    private TipCalculator tipCalculatorMock;

    @BeforeEach
    void init() { (3)
        billCalculator = new BillCalculatorImpl(tipCalculatorMock);
    }

Alternative Mocked Unit Test

@ExtendWith(MockitoExtension.class)
@DisplayNameGeneration(DisplayNameGenerator.ReplaceUnderscores.class)
@DisplayName("Bill CalculatorImpl")
public class BillCalculatorImplTest {
    @Mock
    TipCalculator tipCalculatorMock;
    /*
    Mockito is instantiating this implementation class for us an injecting Mocks
     */
    @InjectMocks (1)
    BillCalculatorImpl billCalculator;

Spring Boot Unit Integration Test Basics

Adding Spring Boot to Testing

@SpringBootTest

package info.ejava.examples.app.testing.testbasics.tips;
...
import org.springframework.boot.test.context.SpringBootTest;
...
@SpringBootTest (1)
public class BillCalculatorNTest {

Default @SpringBootConfiguration Class

package info.ejava.examples.app.testing.testbasics;
...
@SpringBootApplication
// wraps => @SpringBootConfiguration
public class TestBasicsApp {
    public static void main(String...args) {
        SpringApplication.run(TestBasicsApp.class,args);
    }
}

Conditional Components

@Configuration
@ConditionalOnProperty(prefix="hello", name="enable", matchIfMissing="true")
public Hello quietHello() {
...
@SpringBootTest(properties = { "hello.enable=false" }) (1)

Explicit Reference to @SpringBootConfiguration

import info.ejava.examples.app.testing.testbasics.TestBasicsApp;
...
@SpringBootTest(classes = TestBasicsApp.class)
public class BillCalculatorNTest {

Explicit Reference to Components

@Component
@RequiredArgsConstructor
public class BillCalculatorImpl implements BillCalculator {
    private final TipCalculator tipCalculator;
...

@Component
public class StandardTippingImpl implements TipCalculator {
   ...

@SpringBootTest(classes = {BillCalculatorImpl.class, StandardTippingImpl.class})
public class BillCalculatorNTest {
    @Autowired
    BillCalculator billCalculator;

Active Profiles

package info.ejava.examples.app.testing.testbasics.tips;

import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;

@SpringBootTest
@ActiveProfiles("test") (1)
public class BillCalculatorNTest {

# application-test.properties (1)
logging.level.info.ejava.examples.app.testing.testbasics=DEBUG

Example @SpringBootTest NTest Output

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.7.0)

14:17:15.427 INFO  BillCalculatorNTest#logStarting:55  - Starting BillCalculatorNTest
14:17:15.429 DEBUG BillCalculatorNTest#logStarting:56  - Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
14:17:15.430 INFO  BillCalculatorNTest#logStartupProfileInfo:655 - The following profiles are active: test
14:17:16.135 INFO  BillCalculatorNTest#logStarted:61   - Started BillCalculatorNTest in 6.155 seconds (JVM running for 8.085)
14:17:16.138 DEBUG BillCalculatorImpl#calcShares:24    - tip=$9.00, for $50.00 and GOOD service
14:17:16.142 DEBUG BillCalculatorImpl#calcShares:33    - share=$14.75 for $50.00, 4 people and GOOD service
14:17:16.143 INFO  BillHandler#run:24 - bill total $50.00, share=$14.75 for 4 people, after adding tip for GOOD service

14:17:16.679 DEBUG BillCalculatorImpl#calcShares:24    - tip=$18.00, for $100.00 and GOOD service
14:17:16.679 DEBUG BillCalculatorImpl#calcShares:33    - share=$29.50 for $100.00, 4 people and GOOD service

Alternative Test Slices

Maven Unit Testing Basics

|-- pom.xml
`-- src
    `-- test
        |-- java
        |   `-- info
        |       `-- ejava
        |           `-- examples
        |               `-- app
        |                   `-- testing
        |                       `-- testbasics
        |                           |-- PeopleFactory.java
        |                           |-- jupiter
        |                           |   |-- AspectJAssertionsTest.java
        |                           |   |-- AssertionsTest.java
        |                           |   |-- ExampleJUnit5Test.java
        |                           |   `-- HamcrestAssertionsTest.java
        |                           |-- mockito
        |                           |   `-- ExampleMockitoTest.java
        |                           |-- tips
        |                           |   |-- BillCalculatorContractTest.java
        |                           |   |-- BillCalculatorImplTest.java
        |                           |   |-- BillCalculatorMockedNTest.java
        |                           |   |-- BillCalculatorNTest.java
        |                           |   `-- StandardTippingCalculatorImplTest.java
        |                           `-- vintage
        |                               `-- ExampleJUnit4Test.java
        `-- resources
            |-- application-test.properties

Maven Surefire Plugin

$ mvn clean test
...
[INFO] Results:
[INFO]
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  14.280 s

Filtering Tests

import org.junit.jupiter.api.*;
...
@SpringBootTest(classes = {BillCalculatorImpl.class}) //defining custom Spring context
@Tag("springboot") @Tag("tips") (1)
...
public class BillCalculatorMockedNTest {

Filtering Tests Executed

$ mvn clean test -Dgroups='tips & !springboot' -Pbdd (1) (2)
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running Bill Calculator Contract
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.41 s - in Bill Calculator Contract
[INFO] Running Bill CalculatorImpl
15:43:47.605 [main] DEBUG info.ejava.examples.app.testing.testbasics.tips.BillCalculatorImpl - tip=$50.00, for $100.00 and GOOD service
15:43:47.608 [main] DEBUG info.ejava.examples.app.testing.testbasics.tips.BillCalculatorImpl - share=$37.50 for $100.00, 4 people and GOOD service
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.165 s - in Bill CalculatorImpl
[INFO] Running Standard Tipping Calculator
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 s - in Standard Tipping Calculator
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.537 s

Maven Failsafe Plugin

Failsafe Overhead

@TestConfiguration

Example Spring Context

@Primary (1)
@Component
public class StandardTippingImpl implements TipCalculator {

@Component
@RequiredArgsConstructor
public class BillCalculatorImpl implements BillCalculator {
    private final TipCalculator tipCalculator;

Test TippingCalculator

import org.springframework.boot.test.context.TestConfiguration;
...

@TestConfiguration(proxyBeanMethods = false) //skipped in component scan -- manually included (1)
public class MyTestConfiguration {
    @Bean
    public TipCalculator standardTippingImpl() { (2)
        return new TipCalculator() {
            @Override
            public BigDecimal calcTip(BigDecimal amount, ServiceQuality serviceQuality) {
                return BigDecimal.ZERO; (3)
            }
        };
    }
}

Embedded TestConfiguration

@SpringBootTest(properties={"..."})
public class TestConfigurationNTest {
    @Autowired
    BillCalculator billCalculator; (1)

    @TestConfiguration(proxyBeanMethods = false)
    static class MyEmbeddedTestConfiguration { (2)
        @Bean
        public TipCalculator standardTippingImpl() { ... }
    }

External TestConfiguration

@TestConfiguration(proxyBeanMethods = false)
public class MyTestConfiguration {
    @Bean
    public TipCalculator tipCalculator() {
        return new TipCalculator() {
            @Override
            public BigDecimal calcTip(BigDecimal amount, ServiceQuality serviceQuality) {
                return BigDecimal.ZERO;
            }
        };
    }
}