Hi guys! ๐ Today we will learn how to do integration testing for Spring WebClient that invokes external REST APIs with WireMock and JUnit 5.
As an example, we will implement a simple logic responsible for querying the GitHub API using coroutines. Nevertheless, I am pretty sure you will be able to easily adjust this article to your needs. (of course, if you are here only for the WireMock part, then I recommend skipping to the “WebClient Integration Testing With WireMock” chapter).
Lastly, I just wanted to add that you can use this testing approach regardless of whether you are using WebClient, Retrofit, Ktor Client, or any other HTTP client.
Video Tutorial
If you prefer video content, then check out my video:
If you find this content useful, please leave a subscription ๐
Which WebClient Testing Strategy Is The Best?
But before we head to the practice part, let’s try to answer the above question.
Quick answer- there is no such thing as the best testing strategy.
And for the longer answer, let’s take a look at the visualization of the testing pyramid first:
The tests are ordered based on their scope and execution time. The closer we are to the top, the longer the tests take, but they also give us more confidence.
- unit tests form the base, as they are quick to run and pinpoint specific code issues,
- integration tests come next, validating the interactions between units,
- E2E tests, with a broader scope, are placed at the top.
So long story short, although the e2e tests give us the most confidence, we cannot rely only on them, because the feedback loop would take way too long. And this would affect the development process.
So usually, we make concessions and introduce more integration and unit tests.
But which of these two should we pick for the WebClient?
In my opinion- integration tests.
Whenever we are talking about logic responsible for communicating with external API, we must make sure that:
- we query the appropriate URL,
- we pass the necessary headers with appropriate values,
- request/response bodies are properly serialized/deserialized,
- and many, many more.
With unit testing, the amount of mocking here would be so significant that I would question whether we need such a test at all.
Moreover, in Spring we can write integration tests for WebClient with WireMock at almost no effort.
Set Up GitHub API Key
With that said, let’s set up the API key we will use to query the GitHub API.
Note: technically, the endpoint we will use does not require any token. However, for the learning purposes I want to show you how to deal with passing bearer tokens, too ๐
As the first step, let’s navigate to the tokens tab in GitHub settings and click the “Generate new token (classic)”:
On the next page, let’s specify the name, expiration, and desired scopes:
If you don’t know what scopes you will need, please follow the principle of lowest privilege. At the end of the day, we can generate a new token at some point in the future when the requirements change.
When we are ready, let’s hit the generate button copy the token value, and store it securely. We will not see it again!
Query The Api With WebClient
Before we implement the WireMock tests, let’s prepare a simple project responsible for querying the external API with WebClient. So, even if you are here only for the WireMock part, I encourage you to briefly check this paragraph to be on the same page ๐
Long story short, we will implement WebClient with coroutines that will query the “List repositories for a user” endpoint. For more details about it, check out the GitHub API documentation, please.
Generate New Project
As the first step, let’s navigate to the https://start.spring.io/ page and generate a fresh Spring Boot project:
In general, the most important thing is that we will use Kotlin and Spring Reactive Web this time. Versions and metadata are totally up to you.
Of course, let’s click the “Generate” button, download the ZIP file, and import that to our IDE. Personally, I am using IntelliJ IDEA and if you would like to learn more about its setup, then check out my video on how to install IntelliJ for Kotlin development.
Update Application Properties
As the next step, let’s navigate to the resources
package and update the application.yaml
file:
api: github: key: ${GITHUB_API_TOKEN:123} url: ${GITHUB_API_BASE_URL:http://localhost:8090} version: ${GITHUB_API_VERSION:2022-11-28}
Above we can see our custom properties.
We will source values for our GitHub key, URL, and version from environment variables.
Not only is this approach secure (hardcoding the key with token value would be the worst thing you could do), but also flexible (we can pass different values depending on the environment we are running our app in).
WebClient Configuration
Following, let’s configure a WebClient bean that we will use whenever we want to call the GitHub API.
Firstly, let’s add the config
package and put the GitHubApiProperties
data class in there:
@ConfigurationProperties("api.github") data class GitHubApiProperties( val url: String, val key: String, val version: String, )
In my opinion, the @ConfigurationProperties
annotation is the cleanest way to inject values from properties. As we can see above, the prefix and field names simply reflect the structure from the previous step.
With that done, let’s add the GitHubApiConfig
in the same package:
@Configuration @EnableConfigurationProperties(GitHubApiProperties::class) class GitHubApiConfig( private val gitHubApiProperties: GitHubApiProperties, ) { @Bean fun webClient(builder: WebClient.Builder): WebClient = builder .baseUrl(gitHubApiProperties.url) .defaultHeader("Authorization", "Bearer ${gitHubApiProperties.key}") .defaultHeader("X-GitHub-Api-Version", gitHubApiProperties.version) .defaultHeader("Accept", "application/vnd.github+json") .build() }
Quite a few things are happening here, so let’s dive into each one.
Firstly, we annotate the class with @Configuration
and @EnableConfigurationproperties
. The second annotation is necessary to for the previous steps to work.
Then, we inject the GitHubApiProperties
instance and use its values to configure a new WebClient bean. This configuration is quite straightforward. We set the base URL and a bunch of headers that will be sent with every request: the authorization token, GH API version, and the accept header (those two are specific to the GitHub API, so let’s focus on that too much).
Prepare Model Classes
With that done, let’s prepare a bunch of data classes that we will need to deserialize JSON responses from the API (+ one exception class). If you would like to add other fields, please check out the documentation link I shared with you before.
As the first step, let’s add the model
package and the PageableGitHubResponse
:
data class PageableGitHubResponse<T>( val items: List<T>, val hasMoreItems: Boolean, )
The API endpoints allow us to get only a chunk of data using the page
and per_page
parameters. And this class can be reused whenever we’re using a paginated endpoint.
As the next step, let’s implement the GitHubRepoResponse
along with GitHubOwnerResponse
:
data class GitHubRepoResponse( val fork: Boolean, val name: String, val owner: GitHubOwnerResponse, ) data class GitHubOwnerResponse( val login: String, )
As I mentioned at the beginning of this paragraph, we need those two in order to deserialize the actual JSON response payload.
Lastly, let’s add the UpstreamApiException
:
import org.springframework.http.HttpStatusCode data class UpstreamApiException( val msg: String, val statusCode: HttpStatusCode, ) : RuntimeException(msg)
This way, we will be able to throw custom exceptions. Such an exception could be handled later, for example with @RestControllerAdvice.
Make Use Of WebClient
Finally, let’s combine all of that together.
So, let’s add the api
package and implement the GitHubApi
class:
@Component class GitHubApi( private val webClient: WebClient, ) { suspend fun listRepositoriesByUsername( username: String, page: Int, perPage: Int, ): PageableGitHubResponse<GitHubRepoResponse>? = webClient.get() .uri("/users/$username/repos?page=$page&per_page=$perPage") .awaitExchangeOrNull(::mapToPageableResponse) }
Don’t worry about the mapToPageableResponse
, we will add it in a moment.
But before that, let’s analyze the above code.
Firstly, we annotate the class as the @Component
to make it a Spring bean and inject the WebClient
instance we configured earlier. We don’t have other WebClient instances, so there is no need to use @Qualifier
or anything like that.
Following, we add the listRepositoriesByUsername
function that lets us pass the username
along with page
and perPage
. This way, we make this function reusable and allow the invoker to decide how to tackle the pagination.
Additionally, we mark it as a suspend function because the awaitExchangeOrNull
is a suspend function. However, if you are interested in this topic, then I refer you to my other article about WebClient with coroutines.
Lastly, let’s add the missing code:
private suspend inline fun <reified T> mapToPageableResponse(clientResponse: ClientResponse): PageableGitHubResponse<T>? { val hasNext = checkIfMorePagesToFetch(clientResponse) return when (val statusCode = clientResponse.statusCode()) { HttpStatus.OK -> PageableGitHubResponse( items = clientResponse.awaitBody<List<T>>(), hasMoreItems = hasNext, ) HttpStatus.NOT_FOUND -> null else -> throw UpstreamApiException( msg = "GitHub API request failed.", statusCode = statusCode, ) } } fun checkIfMorePagesToFetch(clientResponse: ClientResponse) = clientResponse.headers() .header("link") .firstOrNull() ?.contains("next") ?: false
Long story short, the above code works as follows:
- Firstly, we take the response and check if there are more pages. How? Well, without going into details, GitHub API returns the
link
header. And if its value contains therel="next"
, it means that there are more pages to fetch. - Finally, we check the status code. If it is
200 OK
, we simply read the JSON value. When we get404 Not Found
, we return null (we can expect this status code when we pass a non-existing username, and in my opinion, this case is not exceptional). And whenever we receive another status code, we simply throw theUpstreamApiException
.
WebClient Integration Testing With WireMock
Excellent, at this point we have (almost) everything we need to test our WebClient implementation with WireMock.
We will see only a few example test cases that you will be able to easily adapt according to your needs:
- 200 OK response with an empty list, 200 OK with items, and more pages, and 200 OK with items and no more pages to fetch
- 404 Not Found,
- 401 Unauthorized
When you will be implementing your own integration tests, this is the moment when you should do the homework and analyze the API you are working with. What are the possible happy paths? How should your code behave in case of any error status code? What if we add some latency? And many, many more ๐
Additional Dependencies
At this point, we defined our test cases, so we are ready to get our hands dirty.
So, let’s start by adding the necessary dependencies:
testImplementation("org.springframework.cloud:spring-cloud-contract-wiremock:4.1.1") testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.8.0")
As we can see, the first one- the Spring Cloud Contract WireMock module- allows us to use WireMock in our app, whereas the kotlinx-coroutines-test
is necessary due to our coroutines WebClient implementation.
Add Expected JSON Files
So, if we know what test cases we would like to test, we should prepare the example JSON API responses.
Personally, whenever working with WireMock I am a big fan of externalizing the expected JSON files. We put them inside the /test/resources
directory and refer to them later in the tests. This way, we can improve the readability of the test classes. Of course, unless we want to prepare JSON responses dynamically.
In our case, we could simply invoke the GitHub API and persist responses for different cases. This way, we prepare the following files and put them inside the /test/resources/responses/external/github
:
- list_github_repositories_200_OK_empty_list.json
- list_github_repositories_200_OK_page_1.json
- list_github_repositories_401_UNAUTHORIZED.json
- list_github_repositories_404_NOT_FOUND.json
You can find all of these files in my GH repo here.
The example for 401 Unauthorized
looks as follows:
{ "message": "Bad credentials", "documentation_url": "https://docs.github.com/rest" }
Prepare Util Function
As the last thing before we implement our test class, let’s add the util function inside the util
package (of course, in tests).
This function will be responsible for reading JSON files inside the test resources as String values:
import org.springframework.core.io.ClassPathResource fun getResponseBodyAsString(path: String): String = ClassPathResource(path).getContentAsString( Charsets.UTF_8, )
Implement GitHubApiTest Class
Finally, let’s start implementing the GitHubApiTest
class:
private const val TEST_KEY = "TEST_KEY" private const val TEST_PORT = 8082 private const val TEST_VERSION = "2022-11-28" @AutoConfigureWireMock(port = TEST_PORT) @TestPropertySource( properties = [ "api.github.url=http://localhost:${TEST_PORT}", "api.github.key=$TEST_KEY", "api.github.version=$TEST_VERSION", ], ) @SpringBootTest( webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, ) class GitHubApiTest { @Autowired private lateinit var wireMockServer: WireMockServer @Autowired private lateinit var gitHubApi: GitHubApi private val page = 1 private val perPage = 2 private val username = UUID.randomUUID().toString() }
Conceptually, everything starts with the @SpringBootTest
and @AutoConfigureWireMock
annotations.
We use the @SpringBootTest
annotation in Spring Boot integration tests. This way a new ApplicationContext will be created. In our case, we set the webEnvironment
with RANDOM_PORT
value, so that a reactive context listening on the random port is created.
Additionally, we combine that together with @AutoConfigureWireMock
and fixed test port. This way a WireMock server will be started as a part of Application Context.
Lastly, we do 3 things:
- We use the
@TestPropertySource
to pass values for properties. - We inject the
WireMockServer
andGitHubApi
instances that we will work with later. Please note that we can do that just like we would do with a standard, non-test, Spring component. - We prepare some dummy data for
page
,perPage
, andusername
values we will need for testing.
As a note of comment from my side- this “skeleton” is useful whenever working with WebClient and WireMock.
Test 404 Not Found API Response Case
Wonderful!
As the first test, let’s start with the 404 Not Found
case. Usually, we will start with happy path cases, but for learning purposes, this one is the shortest case ๐
@Test fun `Given 404 NOT FOUND response When fetching repository by username Then should return null`() = runTest { // Given wireMockServer.stubFor( WireMock.get(WireMock.urlEqualTo("/users/$username/repos?page=$page&per_page=$perPage")) .withHeader("Authorization", WireMock.equalTo("Bearer $TEST_KEY")) .withHeader("X-GitHub-Api-Version", WireMock.equalTo(TEST_VERSION)) .withHeader("Accept", WireMock.equalTo("application/vnd.github+json")) .willReturn( WireMock.aResponse() .withStatus(HttpStatus.NOT_FOUND.value()) .withHeader(HttpHeaders.CONTENT_TYPE, "application/json; charset=utf-8") .withBody( getResponseBodyAsString("/responses/external/github/list_github_repositories_404_NOT_FOUND.json"), ), ), ) // When val result = gitHubApi.listRepositoriesByUsername(username, page, perPage) // Then Assertions.assertNull(result) }
We mark our test with the @Test
(JUnit 5), specify the meaningful name for the test (with awesome Kotlin backticks), and wrap our test with runTest
. The last one is necessary as we are working with coroutines and invoke the suspended function. Long story short, in JVM it will behave just like runBlocking
(but it will skip delays).
Next comes the WireMock part. And to be even more specific- stubbing.
Stubbing is nothing else than “the ability to return canned HTTP responses for requests matching criteria” (from docs). Simply said, whenever an outgoing request that matches our criteria is made, a mocked response configured in willReturn
is returned.
So, as we can see, the above logic will work only when:
- the GET request is made,
- URL matches,
- Headers sent have appropriate values
And this part already tests our logic. Pretty neat, isn’t it? ๐
Lastly, we simply invoke the function and assert that null
is returned. Which confirms that everything is working fine.
Test 401 Unauthorized
As we already started with error cases, let’s verify that whenever the endpoint returns 401 Unauthorized
, the UpstreamApiException
is thrown:
@Test fun `Given 401 UAUTHORIZED response When fetching repository by username Then should throw UpstreamApiException`() = runTest { // Given wireMockServer.stubFor( WireMock.get(WireMock.urlEqualTo("/users/$username/repos?page=$page&per_page=$perPage")) .withHeader("Authorization", WireMock.equalTo("Bearer $TEST_KEY")) .withHeader("X-GitHub-Api-Version", WireMock.equalTo(TEST_VERSION)) .withHeader("Accept", WireMock.equalTo("application/vnd.github+json")) .willReturn( WireMock.aResponse() .withStatus(HttpStatus.UNAUTHORIZED.value()) .withHeader(HttpHeaders.CONTENT_TYPE, "application/json; charset=utf-8") .withBody( getResponseBodyAsString("/responses/external/github/list_github_repositories_401_UNAUTHORIZED.json"), ), ), ) // When val exception = assertThrows<UpstreamApiException> { gitHubApi.listRepositoriesByUsername(username, page, perPage) } // Then assertNotNull(exception) assertEquals("GitHub API request failed.", exception.msg) assertEquals(HttpStatus.UNAUTHORIZED, exception.statusCode) }
As we can see, the logic responsible for stubbing only slightly changed.
The interesting part here is the assertThrows
, which not only makes sure that the exception of an appropriate type is thrown but also returns it. This way we can make additional assertions.
Verify 200 OK With Empty List
Following, let’s cover the empty list case:
@Test fun `Given 200 OK response with empty list When fetching repository by username Then should return repository with correct properties and has next false`() = runTest { // Given val linkHeader = """<https://api.github.com/user/64011387/repos?page=3&per_page=2>; rel="prev",""" wireMockServer.stubFor( WireMock.get(WireMock.urlEqualTo("/users/$username/repos?page=$page&per_page=$perPage")) .withHeader("Authorization", WireMock.equalTo("Bearer $TEST_KEY")) .withHeader("X-GitHub-Api-Version", WireMock.equalTo(TEST_VERSION)) .withHeader("Accept", WireMock.equalTo("application/vnd.github+json")) .willReturn( WireMock.aResponse() .withStatus(HttpStatus.OK.value()) .withHeader(HttpHeaders.CONTENT_TYPE, "application/json; charset=utf-8") .withHeader(HttpHeaders.LINK, linkHeader) .withBody( getResponseBodyAsString("/responses/external/github/list_github_repositories_200_OK_empty_list.json"), ), ), ) // When val result = gitHubApi.listRepositoriesByUsername(username, page, perPage) // Then val expected = PageableGitHubResponse( items = emptyList<GitHubRepoResponse>(), hasMoreItems = false, ) assertEquals(expected, result) }
This time, we assign the returned value to the result
and compare that with the expected
.
Check 200 OK With Has Next True
Last before least, let’s make sure that we got the expected objects in a list and that the logic responsible for the link
header check returns true:
@Test fun `Given 200 OK response with payload When fetching repository by username Then should return repository with correct properties and has next true`() = runTest { // Given val linkHeader = """<https://api.github.com/user/64011387/repos?page=3&per_page=2>; rel="next",""" wireMockServer.stubFor( WireMock.get(WireMock.urlEqualTo("/users/$username/repos?page=$page&per_page=$perPage")) .withHeader("Authorization", WireMock.equalTo("Bearer $TEST_KEY")) .withHeader("X-GitHub-Api-Version", WireMock.equalTo(TEST_VERSION)) .withHeader("Accept", WireMock.equalTo("application/vnd.github+json")) .willReturn( WireMock.aResponse() .withStatus(HttpStatus.OK.value()) .withHeader(HttpHeaders.CONTENT_TYPE, "application/json; charset=utf-8") .withHeader(HttpHeaders.LINK, linkHeader) .withBody( getResponseBodyAsString("/responses/external/github/list_github_repositories_200_OK_page_1.json"), ), ), ) // When val result = gitHubApi.listRepositoriesByUsername(username, page, perPage) // Then val expected = PageableGitHubResponse( items = listOf( GitHubRepoResponse( fork = false, name = "controlleradvice-vs-restcontrolleradvice", owner = GitHubOwnerResponse(login = "codersee-blog"), ), GitHubRepoResponse( fork = false, name = "freecodecamp-spring-boot-kotlin-excel", owner = GitHubOwnerResponse(login = "codersee-blog"), ), ), hasMoreItems = true, ) assertEquals(expected, result) }
Test 200 OK With Has Next False
And finally, let’s double-check the flag logic:
@Test fun `Given 200 OK response with payload When fetching repository by username Then should return repository with correct properties and has next false`() = runTest { // Given val linkHeader = """<https://api.github.com/user/64011387/repos?page=3&per_page=2>; rel="prev",""" wireMockServer.stubFor( WireMock.get(WireMock.urlEqualTo("/users/$username/repos?page=$page&per_page=$perPage")) .withHeader("Authorization", WireMock.equalTo("Bearer $TEST_KEY")) .withHeader("X-GitHub-Api-Version", WireMock.equalTo(TEST_VERSION)) .withHeader("Accept", WireMock.equalTo("application/vnd.github+json")) .willReturn( WireMock.aResponse() .withStatus(HttpStatus.OK.value()) .withHeader(HttpHeaders.CONTENT_TYPE, "application/json; charset=utf-8") .withHeader(HttpHeaders.LINK, linkHeader) .withBody( getResponseBodyAsString("/responses/external/github/list_github_repositories_200_OK_page_1.json"), ), ), ) // When val result = gitHubApi.listRepositoriesByUsername(username, page, perPage) // Then val expected = PageableGitHubResponse( items = listOf( GitHubRepoResponse( fork = false, name = "controlleradvice-vs-restcontrolleradvice", owner = GitHubOwnerResponse(login = "codersee-blog"), ), GitHubRepoResponse( fork = false, name = "freecodecamp-spring-boot-kotlin-excel", owner = GitHubOwnerResponse(login = "codersee-blog"), ), ), hasMoreItems = false, ) assertEquals(expected, result) }
Integration Tests With WireMock And WebClient Summary
And basically, that’s all for this tutorial on how to perform integration testing for Spring WebClient that invokes external REST APIs with WireMock and JUnit 5.
I hope you enjoyed this one and I recommend you check out my other articles related to Spring Framework.
One Response
Hi Piotr,
thanks a lot for this article. As always, it’s a pleasure to read. I gained many insights on how to access the GitHub-API and how to configure and integration-test SpringBoot applications.
The Wiremock-Part, however, didn’t convince me. In my opinion wiremock’s fluent API leads to verbose testcode and a low signal/noise ratio.
Also I’m ambivalent about the usage of json files found in `src/test/resources`. On the one hand, one can easily record actual output from github.com and paste it into a resource file. On the other hand, to validate the test’s behavior, one has to switch editors and must ignore almost all lines in `list_github_repositories_200_OK_page_1.json`.
In the case, you are interested in alternative approaches to testing, feel free to check out [my repository](https://github.com/neblung/webclientinttest)
Best regards
Frank