Introducing Errors Into HTTP Requests For Resilience Testing

Posted on March 24, 2024

dotnet

I’ve recently been playing with Polly, a .NET library for adding resilience to code where transient faults may occur. It allows policies to be defined that enable retires, timeouts, rate liming and a host of other options. As a simple example, you can specify that your HTTP API request to fetch some data retries 3 times in the event of an error before giving up.

I already had a HTTP API and .NET client built and wanted to do testing inside the client but couldn’t think of a simple way to introduce errors into the environment. I could change the API backend to randomly return errors but this seemed like unnecessary work and wouldn’t be appropriate for situations where you didn’t have access to the backend code.

After a bit of digging I found Microsoft had released a tool called Dev Proxy. It works by proxying requests to configured endpoints and returning errors at a defined rate. It also has other features like returning mocked data but I didn’t look into these.

Installation

There are releases on GitHub, but I used winget.

winget install Microsoft.DevProxy

Using Dev Proxy

The installation places files in C:\Users\<username>\AppData\Local\Programs\Dev Proxy which contains a configuration file called devproxyrc.json.

Inside this is an array called urlsToWatch which is where URLs which need to be proxied are defined.

  "urlsToWatch": [
    "https://jsonplaceholder.typicode.com/*",
    "http://localhost:8080/*"
  ],

I needed to run it as admin the first time to get the certificate used for HTTPS interception installed, but after that it ran without special privileges.

devproxy.exe --failure-rate 50

Here is an example hitting the jsonplaceholder testing API with PowerShell:

PS C:\> irm "https://jsonplaceholder.typicode.com/users/1"
irm : {
  "message": "Internal Server Error",
  "details": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."
}


PS C:\> irm "https://jsonplaceholder.typicode.com/users/1"
irm : {
  "message": "Service Unavailable",
  "details": "The server is currently unable to handle the request due to a temporary overload or maintenance. Please try again later."
}


PS C:\> irm "https://jsonplaceholder.typicode.com/users/1"
irm : {
  "message": "Internal Server Error",
  "details": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."
}


PS C:\> irm "https://jsonplaceholder.typicode.com/users/1"

id       : 1
name     : Leanne Graham
username : Bret
email    : Sincere@april.biz
address  : @{street=Kulas Light; suite=Apt. 556; city=Gwenborough; zipcode=92998-3874; geo=}
phone    : 1-770-736-8031 x56442
website  : hildegard.org
company  : @{name=Romaguera-Crona; catchPhrase=Multi-layered client-server neural-net; bs=harness real-time e-markets}

And the corresponding Dev Proxy logs:

PS C:\> devproxy.exe --failure-rate 50
Listening on 127.0.0.1:8000...
Set endpoint at Ip 127.0.0.1 and port: 8000 as System HTTP Proxy
Set endpoint at Ip 127.0.0.1 and port: 8000 as System HTTPS Proxy
Hotkeys: issue (w)eb request, (r)ecord, (s)top recording, (c)lear screen
Press CTRL+C to stop Dev Proxy

 request     GET https://jsonplaceholder.typicode.com/users/1
   chaos   ╭ 500 InternalServerError
           ╰ GET https://jsonplaceholder.typicode.com/users/1
 request     GET https://jsonplaceholder.typicode.com/users/1
   chaos   ╭ 503 ServiceUnavailable
           ╰ GET https://jsonplaceholder.typicode.com/users/1
 request     GET https://jsonplaceholder.typicode.com/users/1
   chaos   ╭ 500 InternalServerError
           ╰ GET https://jsonplaceholder.typicode.com/users/1
 request     GET https://jsonplaceholder.typicode.com/users/1
     api   ╭ Passed through
           ╰ GET https://jsonplaceholder.typicode.com/users/1

That’s all there is to it. Most of it is covered in the docs but it took me a little while to find and seems relatively unknown so I wanted to share.

Microsoft also has a good article on resilience and ‘chaos engineering’ which has some other options.