POC App E2E Testing｜Local Snapshot API Mock Server for Reliable Validation

点击这里查看本文章简体中文版本。

點擊這裡查看本文章正體中文版本。

This post was translated with AI assistance — let me know if anything sounds off!

---

[POC] App End-to-End Testing Local Snapshot API Mock Server

Feasibility Verification of Implementing E2E Testing for Existing Apps and API Architectures

Photo by freestocks

Introduction

As a project that has been running online for many years, continuously improving stability is a highly challenging issue.

Unit Testing

Because the app is developed in Swift/Kotlin with static typing, compilation, and strong typing, or converted from Objective-C to Swift (dynamic to static), if interface dependencies are not properly separated during development, adding Unit Testing later is almost impossible; however, refactoring can introduce instability, creating a chicken-and-egg problem.

UI Testing

For UI interactions and button testing; slight decoupling of data dependencies in new or existing screens can achieve this.

SnapShot Testing

Verify whether the UI display content and style are consistent before and after adjustments; similar to UI Testing, newly developed or existing screens can achieve this by slightly decoupling data dependencies.

Useful for converting Storyboard/XIB layouts or UIViews from Objective-C to Swift; you can directly import pointfreeco / swift-snapshot-testing for quick implementation.

Although we can add UI Testing and Snapshot Testing later, the coverage is very limited; most errors are not related to UI styles but to process or logic issues that cause users to stop their actions. If these occur in the checkout process, affecting revenue, the problem becomes very serious.

End-to-End Testing

As mentioned earlier, it is not feasible to simply add unit tests to the current project or consolidate units for integration testing. For protecting logic and processes, the remaining method is to perform End-to-End black-box testing externally, directly from the user’s perspective, to verify whether key workflows (registration/checkout…) function correctly.

For major feature refactoring, first create tests for the workflow before refactoring, then re-validate after refactoring to ensure the feature works as expected.

During refactoring, add Unit Testing and Integration Testing to improve stability and resolve the chicken-and-egg problem.

QA Team

The most straightforward and brute-force method for End-to-End Testing is to have a QA team manually test according to the Test Plan, then continuously optimize or introduce automation; calculating the cost, it requires at least 2 engineers plus 1 leader spending at least six months to a year to see results.

Evaluate the time and cost. Is there anything we can do now or prepare for the future QA Team so that when the QA Team is established, they can directly move to optimization, automation, or even AI implementation(?).

Automation

At this stage, the goal is to implement automated End-to-End Testing within the CI/CD pipeline. The test coverage doesn’t need to be complete; as long as it can prevent major process issues, it is already valuable. The Test Plan can be gradually iterated and expanded later to cover more areas.

End-to-End Testing — Technical Challenges

UI Operation Issues

The app works more like using another test app to operate the app under test, then locating target objects through the View Hierarchy. During testing, it cannot access the logs or output of the app under test because they are essentially two different apps.

iOS needs to improve View Accessibility Identifiers to enhance efficiency and accuracy, and also handle Alerts (e.g., push notification requests).

Android previously had issues finding target objects when mixing Compose with Fragments, but according to a teammate, the latest Compose version has resolved this.

In addition to the common traditional issues, a bigger problem is the difficulty in integrating dual platforms (writing one test to run on both). Currently, we are trying new testing tools mobile-dev-inc / maestro:

You can write a Test Plan using YAML and run tests on both platforms. Details on usage and trial experience will be shared by another teammate soon, cc’ed Alejandra Ts. 😝.

API Data Issues

The biggest variable in App E2E Testing is the API data. Without guaranteed consistent data, test instability increases, causing false positives and ultimately leading to a loss of confidence in the Test Plan.

For example, when testing the checkout process, if a product might be removed or disappear, and these status changes are not controlled by the app, the above issues are likely to occur.

There are many ways to solve data issues, such as creating clean Staging or Testing environments, or using Open API-based Auto-Gen Mock API Servers. However, these methods rely on the backend and external API factors. Since the backend API, like the app, has been running online for many years and some specifications are still being restructured and migrated, a Mock Server is temporarily not available.

Based on the above factors, if you get stuck here, the problem will remain unchanged, and the chicken-and-egg dilemma cannot be resolved. The only option is to “take the risk” by making changes first and dealing with issues as they arise.

Snapshot API Local Mock Server

“As long as the mindset doesn’t decline, there are always more methods than difficulties.”

We can consider a different approach. If the UI can be captured as snapshots for replay and validation testing, can the API do the same? Can we save API requests and responses to replay them later for validation testing?

Introducing the main focus of this article: Building a “Snapshot API Local Mock Server” to Record API Requests & Replay Responses, separating dependency on API data.

This article only presents a POC concept verification and has not fully implemented high-coverage End To End Testing. Therefore, the approach is for reference only, and we hope it inspires new ideas within your current environment.

Snapshot API Local Mock Server

Core Concept — Record & Replay API Data

[Record] — After completing the development of the End-to-End Testing Test Case, open the recording parameters and run a test once. All API Requests & Responses during the process will be saved in each Test Case directory.

[Replay] — During the Test Case run, find the corresponding recorded Response Data from the Test Case directory based on the request to complete the test process.

Illustration

Assuming we want to test the purchase process: after the user opens the app, they click the product card on the home page to enter the product detail page, press the purchase button at the bottom, a login box pops up to complete login, complete the purchase, and a purchase success message appears:

UI Testing on controlling button clicks, input fields, etc., is not the main focus of this article; you can refer to existing testing frameworks for direct use.

Regular Proxy or Reverse Proxy

To achieve Record & Replay API, a proxy must be placed between the app and the API to perform a man-in-the-middle attack. You can refer to my earlier article “APP uses HTTPS transmission, but data was still stolen.”

Simply put, it is an intermediary that passes messages between the App and the API. Like passing notes, all requests and responses between both sides go through it. It can read the contents of the notes or even forge them without either side noticing the interference.

Regular Proxy:

A forward proxy is when the client sends a request to the proxy server, which then forwards the request to the target server and returns the target server’s response to the client. In forward proxy mode, the proxy server initiates the request on behalf of the client. The client must explicitly specify the proxy server’s address and port number and send the request to the proxy server.

Reverse Proxy:

A reverse proxy is the opposite of a forward proxy; it sits between the target server and the client. The client sends requests to the reverse proxy server, which forwards the requests to the backend target server based on certain rules and returns the target server’s response to the client. To the client, the target server appears as the reverse proxy server, and the client does not need to know the real address of the target server.

For our needs, both forward and reverse can achieve the goal. The only thing to consider is the method of proxy setup:

A forward proxy requires setting up a proxy on the computer, phone, or emulator network settings:

• Android can individually set Proxy in the emulator directly

• The iOS Simulator shares the computer’s network environment and cannot have a separate Proxy set up. This means you must change the computer’s settings to enable the Proxy. All traffic from the computer will go through this Proxy, and if you run other network tools like Proxyman or Charles simultaneously, they may forcibly change the Proxy settings to their own, causing the Proxy to fail.

Reverse proxy requires changing the API Host in the Codebase and declaring all API Domains to be proxied:

• The API Host in the codebase should be replaced with the Proxy Server IP during testing.

• Which domains should be declared to use the proxy when enabling Reverse Proxy?

• Only declared domains will go through the proxy; undeclared ones will connect directly.

In conjunction with the iOS App, the following is a POC example using iOS & Reverse Proxy. The same applies to Android.

Let the iOS App Know It Is Running End-to-End Testing Now

We need to let the app know that it is currently running End-to-End Testing so that we can add API Host replacement logic in the app code:

// UI Testing Target:
let app = XCUIApplication()
app.launchArguments = ["duringE2ETesting"]
app.launch()

We perform judgment and replacement at the Network layer.

This adjustment is unavoidable; try not to change the app’s code just for testing purposes.

Using MITMProxy to Implement a Reverse Proxy Server

You can also develop a Swift Server using Swift yourself. This article is just a POC, so it directly uses the MITMProxy tool.

[2023–09–04 Update] Mitmproxy-rodo is now open source

The following implementation has been open-sourced in the mitmproxy-rodo project. Feel free to check it out for reference.

Some structure and content of this article have been adjusted; further adjustments were made after open-sourcing:

• Save the directory structure as host / requestPath / method / hash

• Fix Header information storage: it should be Bytes Data instead of plain JSON String

• Correct some errors

• Add automatic extension for Set-Cookie expiration time function

⚠️ The following script is for demo reference only. Future script adjustments will be maintained in the open-source project.

⚠️ The following script is for demo reference only. Future script adjustments will be maintained in the open-source project.

⚠️ The following script is for demo reference only. Future script adjustments will be maintained in an open-source project.

⚠️ The following script is for demo reference only. Future script updates will be maintained in an open-source project.

⚠️ The following script is for demo reference only. Future script adjustments will be maintained in an open-source project.

MITMProxy

Follow the MITMProxy official website to complete the installation:

brew install mitmproxy

For detailed usage of MITMProxy, please refer to my earlier article “APP uses HTTPS for transmission, but data was still stolen.”

• mitmproxy provides an interactive command-line interface.

• mitmweb provides a browser-based graphical user interface.

• mitmdump provides non-interactive terminal output.

Implement Record & Replay

Since MITMProxy Reverse Proxy does not natively support recording (or dumping) requests and mapping request replay, we need to write scripts to implement this functionality ourselves.

mock.py :

"""
Example:
    Record: mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=loginFlow --set config_file=config.json
    Replay: mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=loginFlow --set config_file=config.json
"""

import re
import logging
import mimetypes
import os
import json
import hashlib

from pathlib import Path
from mitmproxy import ctx
from mitmproxy import http

class MockServerHandler:

    def load(self, loader):
        self.readHistory = {}
        self.configuration = {}

        loader.add_option(
            name="dumper_folder",
            typespec=str,
            default="dump",
            help="Response Dump directory, can be created by Test Case Name",
        )

        loader.add_option(
            name="network_restricted",
            typespec=bool,
            default=True,
            help="If no local mapping data... set true to return 404, false to make real requests to get data.",
        )

        loader.add_option(
            name="record",
            typespec=bool,
            default=False,
            help="Set true to record Request's Response",
        )

        loader.add_option(
            name="config_file",
            typespec=str,
            default="",
            help="Set file path, example file below",
        )
    
    def configure(self, updated):
        self.loadConfig()

    def loadConfig(self):
        configFile = Path(ctx.options.config_file)
        if ctx.options.config_file == "" or not configFile.exists():
            return

        self.configuration = json.loads(open(configFile, "r").read())

    def hash(self, request):
        query = request.query
        requestPath = "-".join(request.path_components)

        ignoredQueryParameterByPaths = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("queryParamters", [])
        ignoredQueryParameterGlobal = self.configuration.get("ignored", {}).get("global", {}).get("queryParamters", [])

        filteredQuery = []
        if query:
            filteredQuery = [(key, value) for key, value in query.items() if key not in ignoredQueryParameterByPaths + ignoredQueryParameterGlobal]
        
        formData = []
        if request.get_content() != None and request.get_content() != b'':
            formData = json.loads(request.get_content())
        
        # or just formData = request.urlencoded_form
        # or just formData = request.multipart_form
        # depends on your api design

        ignoredFormDataParametersByPaths = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("formDataParameters", [])
        ignoredFormDataParametersGlobal = self.configuration.get("ignored", {}).get("global", {}).get("formDataParameters", [])

        filteredFormData = []
        if formData:
            filteredFormData = [(key, value) for key, value in formData.items() if key not in ignoredFormDataParametersByPaths + ignoredFormDataParametersGlobal]
        
        # Serialize the dictionary to a JSON string
        hashData = {"query":sorted(filteredQuery), "form": sorted(filteredFormData)}
        json_str = json.dumps(hashData, sort_keys=True)

        # Apply SHA-256 hash function
        hash_object = hashlib.sha256(json_str.encode())
        hash_string = hash_object.hexdigest()
        
        return hash_string

    def readFromFile(self, request):
        host = request.host
        method = request.method
        hash = self.hash(request)
        requestPath = "-".join(request.path_components)

        folder = Path(ctx.options.dumper_folder) / host / method / requestPath / hash

        if not folder.exists():
            return None

        content_type = request.headers.get("content-type", "").split(";")[0]
        ext = mimetypes.guess_extension(content_type) or ".json"


        count = self.readHistory.get(host, {}).get(method, {}).get(requestPath, {}) or 0

        filepath = folder / f"Content-{str(count)}{ext}"

        while not filepath.exists() and count > 0:
            count = count - 1
            filepath = folder / f"Content-{str(count)}{ext}"

        if self.readHistory.get(host) is None:
            self.readHistory[host] = {}
        if self.readHistory.get(host).get(method) is None:
            self.readHistory[host][method] = {}
        if self.readHistory.get(host).get(method).get(requestPath) is None:
            self.readHistory[host][method][requestPath] = {}

        if filepath.exists():
            headerFilePath = folder / f"Header-{str(count)}.json"
            if not headerFilePath.exists():
                headerFilePath = None
            
            count += 1
            self.readHistory[host][method][requestPath] = count

            return {"content": filepath, "header": headerFilePath}
        else:
            return None


    def saveToFile(self, request, response):
        host = request.host
        method = request.method
        hash = self.hash(request)
        requestPath = "-".join(request.path_components)

        iterable = self.configuration.get("ignored", {}).get("paths", {}).get(request.host, {}).get(requestPath, {}).get(request.method, {}).get("iterable", False)
        
        folder = Path(ctx.options.dumper_folder) / host / method / requestPath / hash

        # create dir if not exists
        if not folder.exists():
            os.makedirs(folder)

        content_type = response.headers.get("content-type", "").split(";")[0]
        ext = mimetypes.guess_extension(content_type) or ".json"

        repeatNumber = 0
        filepath = folder / f"Content-{str(repeatNumber)}{ext}"
        while filepath.exists() and iterable == False:
            repeatNumber += 1
            filepath = folder / f"Content-{str(repeatNumber)}{ext}"
        
        # dump to file
        with open(filepath, "wb") as f:
            f.write(response.content or b'')
            
        
        headerFilepath = folder / f"Header-{str(repeatNumber)}.json"
        with open(headerFilepath, "wb") as f:
            responseDict = dict(response.headers.items())
            responseDict['_status_code'] = response.status_code
            f.write(json.dumps(responseDict).encode('utf-8'))

        return {"content": filepath, "header": headerFilepath}

    def request(self, flow):
        if ctx.options.record != True:
            host = flow.request.host
            path = flow.request.path

            result = self.readFromFile(flow.request)
            if result is not None:
                content = b''
                headers = {}
                statusCode = 200

                if result.get('content') is not None:
                    content = open(result['content'], "r").read()

                if result.get('header') is not None:
                    headers = json.loads(open(result['header'], "r").read())
                    statusCode = headers['_status_code']
                    del headers['_status_code']

                
                headers['_responseFromMitmproxy'] = '1'
                flow.response = http.Response.make(statusCode, content, headers)
                logging.info("Fulfill response from local with "+str(result['content']))
                return

            if ctx.options.network_restricted == True:
                flow.response = http.Response.make(404, b'', {'_responseFromMitmproxy': '1'})
        
    def response(self, flow):
        if ctx.options.record == True and flow.response.headers.get('_responseFromMitmproxy') != '1':
            result = self.saveToFile(flow.request, flow.response)
            logging.info("Save response to local with "+str(result['content']))

addons = [MockServerHandler()]

You can refer to the official documentation and adjust the script content according to your needs.

The script design logic is as follows:

• File path logic: dumper_folder(a.k.a Test Case Name) / Reverse's api host / HTTP Method / Path joined with - (e.g. app/launch -> app-launch) / Hash(Get Query & Post Content) /

• File logic: Response content: Content-0.xxx, Content-1.xxx (second request for the same one)… and so on; Response header info: Header-0.json (same logic as Content-x)

• When saving, files are stored sequentially according to the path and file logic; during Replay, they are retrieved in the same order.

• If the number of times does not match, for example, during Replay the same path is called 3 times, but the recorded data only stores up to the 2nd time; it will continue to respond with the result of the 2nd time, which is the last recorded result.

• When record is True, it sends a request to the target server, gets the response, and saves it according to the above logic; when False, it only reads data locally (equivalent to Replay Mode).

• When network_restricted is False, if there is no local Mapping data, it will directly respond with 404; when it is True, it will fetch data from the target Server.

• _responseFromMitmproxy is used to inform the Response Method that the current response comes from Local and can be ignored. _status_code uses the Header.json field to store the HTTP Response status code.

config_file.json configuration file logic design is as follows:

{
  "ignored": {
    "paths": {
      "yourapihost.com": {
        "add-to-cart": {
          "POST": {
            "queryParamters": [
              "created_timestamp"
            ],
            "formDataParameters": []
          }
        },
        "api-status-checker": {
          "GET": {
            "iterable": true
          }
        }
      }
    },
    "global": {
      "queryParamters": [
        "timestamp"
      ],
      "formDataParameters": []
    }
  }
}

queryParamters & formDataParameters :

Some API parameters may change with each call, such as time parameters in certain endpoints. According to the server design, the value of Hash(Query Parameter & Body Content) will differ during Replay Request, causing failure to map to the Local Response. To handle this, an additional config.json was created. It allows excluding specific parameters from the hash calculation by Endpoint Path or globally, ensuring consistent mapping results.

iterable :

Since some polling check APIs may repeatedly call at regular intervals, the server design generates many Content-x.xxx & Header-x.json files; however, if we don’t mind, we can set it to True, and the response will continuously save and overwrite the first files Content-0.xxx & Header-0.json.

Enable Reverse Proxy Record Mode:

mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=loginFlow --set config_file=config.json

Enable Reverse Proxy Replay Mode:

mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=loginFlow --set config_file=config.json

Assembly & Proof Of Concept

0. Replace Hosts in the Codebase

Make sure that during testing, the API has been changed to http://127.0.0.1:8080

1. Start Snapshot API Local Mock Server (a.k.a Reverse Proxy Server) Record Mode

mitmdump -m reverse:https://yourapihost.com -s mock.py --set record=true --set dumper_folder=addCart --set config_file=config.json

2. Perform E2E Testing UI Operations

Using Pinkoi iOS App as an example, test the following process:

Launch App -> Home -> Scroll Down -> Similar to Wish List Items Section -> First Product -> Click First Product -> Enter Product Page -> Click Add to Cart -> UI Response Added to Cart -> Test Successful ✅

The UI automation method was mentioned earlier. Here, we first manually test the same process to verify the results.

3. Get Record Results

After completing the operation, press ^ + C to stop the Snapshot API Mock Server, then check the recording results in the file directory:

4. Replay Verification of the Same Process, Start Server & Using Replay Mode

mitmdump -m reverse:https://yourapihost.com -s mock.py --set dumper_folder=addCart --set config_file=config.json

5. Execute the previous UI operation again to verify the results

• Left: Test Successful ✅

• Right: Test clicking on products other than the recorded ones. An error will appear (because there is no local data + network_restricted is set to False by default, no local data will directly return 404 and will not fetch data from the network).

6. Proof Of Concept ✅

Proof of concept succeeded. We can indeed implement a Reverse Proxy Server to store API requests and responses ourselves, and use it as a Mock API Server to respond to the app during testing 🎉🎉🎉.

[2023–09–04] mitmproxy-rodo Open Sourced

Follow-up and Miscellaneous Notes

This article only discusses the proof of concept; there are many areas to improve and more features to implement in the future.

1. Integration with maestro UI Testing tool

2. CI/CD Pipeline Integration Design (How to Automatically Start Reverse Proxy? Where to Start It?)

3. How to Embed MITMProxy into a Development Tool?

4. Verify More Complex Test Scenarios

5. For validating the sent Tracking Request, you need to implement storing the Request Body and then extract which Tracking Event Data were sent and whether the events conform to the required process

Cookie Issues

#...
    def response(self, flow):
        setCookies = flow.response.headers.get_all("set-cookie")
        # setCookies = ['ad=0; Domain=.xxx.com; expires=Wed, 23 Aug 2023 04:59:07 GMT; Max-Age=1800; Path=/', 'sessionid=xxxx; Secure; HttpOnly; Domain=.xxx.com; expires=Wed, 23 Aug 2023 04:59:07 GMT; Max-Age=1800; Path=/']
        
        # OR Replace Cookie Domain From .xxx.com To 127.0.0.1
        setCookies = [re.sub(r"\s*\.xxx\.com\s*", "127.0.0.1", s) for s in setCookies]

        # AND Remove security-related restrictions
        setCookies = [re.sub(r";\s*Secure\s*", "", s) for s in setCookies]
        setCookies = [re.sub(r";\s*HttpOnly;\s*", "", s) for s in setCookies]

        flow.response.headers.set_all("Set-Cookie", setCookies)

        #...

If you encounter cookie-related issues, such as the API returning cookies but the app not receiving them, refer to the adjustments mentioned above.

The Last Article on Pinkoi

In over 900 days at Pinkoi, I realized many of my career and iOS/App development and process ideas. Thanks to all my teammates for going through the pandemic and challenges together; the courage to say goodbye is like the courage I had when I first pursued my dream job.

Setting sail to find new life challenges (including but not limited to engineering). If you have suitable opportunities (iOS or engineering management or startup products), please feel free to contact me. 🙏🙏🙏

If you have any questions or feedback, feel free to contact me.

---

This post was originally published on Medium (View original post), and automatically converted and synced by ZMediumToMarkdown.

Improve this page on Github.

Follow Me on Medium 1K+ Followers