The APP Uses HTTPS for Transmission, But Data Still Gets Stolen.
Using mitmproxy for man-in-the-middle attacks to sniff API data transmission on iOS and macOS.
ℹ️ℹ️ℹ️ The following content is translated by OpenAI.
Click here to view the original Chinese version. | 點此查看本文中文版
The APP Uses HTTPS for Transmission, But Data Still Gets Stolen.
This article provides a tutorial on using mitmproxy for man-in-the-middle attacks to sniff API data transmission on iOS and macOS, as well as how to prevent such attacks.
Introduction
Recently, we held an internal CTF competition at my company. While brainstorming topics, I recalled a project I worked on during college when I was involved in backend development (PHP). It was a point-collection APP that had a task list, and upon completing certain conditions, it would call an API to earn points. My boss believed that calling the API over HTTPS encrypted the data, making it secure—until I demonstrated a man-in-the-middle attack, directly sniffing the transmitted data and spoofing API calls to earn points…
Moreover, in recent years, the rise of big data has led to a proliferation of web crawlers; the battle between crawlers and defenses has become increasingly intense, with numerous tricks emerging in the battle between crawling and anti-crawling. It can only be said that as the techniques improve, so do the countermeasures!
Crawlers also target APP APIs, and without any defenses, it’s almost like leaving the door wide open; they are not only easy to operate but also have clean formats that are harder to identify and block. Therefore, if the web version has already exhausted all efforts to block access, yet data is still being crawled, it might be worth checking if there are any vulnerabilities in the APP’s API.
Since I wasn’t sure how to present this topic in the CTF competition, I decided to write an article to document it. This piece is just a basic overview—HTTPS can decrypt transmitted content through certificate replacement and how to enhance security to prevent such attacks. The actual network theory isn’t my strong suit, and I’ve already returned that knowledge to my teachers. If you already have a grasp of this topic, you can skip this article or scroll down to see how to protect your APP!
Practical Operation
Environment: MacOS + iOS
Android users can directly download Packet Capture (free), while iOS users can use Surge 4 (paid) to unlock man-in-the-middle attack features. MacOS users can also use another paid software, Charles.
This article mainly explains how to operate using the free mitmproxy on iOS. If you have the above environment, you can simply open the APP on your phone and mount a VPN to replace the certificate for the man-in-the-middle attack! (Again, please scroll down to see how to protect it!)
[2021/02/25 Update]: A new free graphical interface program for Mac, ( Proxyman ), is available, which can be used in conjunction with the first part of this article.
Install mitmproxy
Install directly using brew:
1
brew install mitmproxy
Installation complete!
p.s. If you encounter “brew: command not found,” please first install the brew package management tool:
1
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Using mitmproxy
After installation, enter the following command in Terminal to activate:
1
mitmproxy
Startup successful.
Ensure the Phone and Mac Are on the Same Local Network & Obtain Mac’s IP Address
Method (1): Connect Mac to WiFi, and have the phone use the same WiFi. Mac’s IP Address = “System Preferences” -> “Network” -> “Wi-Fi” -> “IP Address”
Method (2): Connect Mac via wired network, enable internet sharing; connect the phone to that hotspot:
Mac’s IP Address = 192.168.2.1 (⚠️ Note: This is not the Ethernet network IP; it’s the IP used by Mac as the internet sharing base station.)
Phone Network Settings WiFi — Proxy Server Information
Settings -> WiFi -> HTTP Proxy -> Manual -> Enter Mac’s IP Address -> Enter 8080 -> Save
At this point, if the webpage doesn’t open and a certificate error appears, that’s normal; let’s continue…
Install mitmproxy Custom HTTPS Certificate
As mentioned above, the implementation of a man-in-the-middle attack involves using your own certificate to replace and decrypt data during communication; therefore, we also need to install this custom certificate on the phone.
1. Open Safari on your phone and go to http://mitm.it
Left side shows Proxy settings ✅ / Right side indicates Proxy settings are incorrect 🚫
Apple -> Install Profile -> Install
⚠️ We’re not done yet; we still need to enable the profile in the settings.
General -> About -> Certificate Trust Settings -> Enable mitmproxy
Done! Now we can return to the browser and browse the web normally.
Return to Mac to Operate mitmproxy
You can see the data transmission records from the phone on the mitmproxy Terminal.
Find the desired sniffing record to view Request (parameters sent) / Response (content returned).
Common Operation Key Set:
1
2
3
4
5
6
7
8
9
10
11
12
13
「 ? 」= View key operation documentation
「 k 」/「⬆」= Up
「 j 」/「⬇」= Down
「 h 」/「⬅」= Left
「 l 」/「➡️」️= Right
「 space 」= Next page
「 enter 」= Enter to view details
「 q 」= Return to the previous page/exit
「 b 」= Export response body to specified path text file
「 f 」= Filter record conditions
「 z 」= Clear all records
「 e 」= Edit Request (cookie, headers, params...)
「 r 」= Resend Request
Not Used to CLI? No Problem, You Can Switch to Web GUI!
In addition to the mitmproxy activation method, we can switch to:
1
mitmweb
This allows you to use the new Web GUI for observation and operation.
mitmweb.
The Main Event: Sniffing APP Data
Once the above environment is set up and familiar, we can enter the main event; sniffing the data transmission content of the APP API!
Here, we will use a housing APP as an example, with no malicious intent, purely for academic exchange!
We want to know how the object list API requests and what content it returns!
First, press “z” to clear all records (to avoid confusion).
Open the target APP.
Open the target APP and try to “pull down to refresh” or trigger the action to “load the next page.”
🛑 If your target APP doesn’t open or connect; then sorry, it means the APP has implemented defenses that prevent sniffing. Please scroll directly to the section on how to protect it. 🛑
mitmproxy records.
Return to mitmproxy to check the records, channel your inner detective to guess which API request record is what we want and enter to view the details!
Request.
In the Request section, you can see which parameters were passed in the request.
By using “e” to edit and “r” to resend, and observing the Response, you can guess the purpose of each parameter!
Response.
You can also directly obtain the original returned content from the Response.
🛑 If the Response content is a bunch of encoded data; then sorry, it means the APP may have its own encryption and decryption process, making it impossible to sniff using this method. Please scroll directly to the section on how to protect it. 🛑
Is it hard to read? Is there garbled Chinese? No worries, you can use “b” to export it as a text file to your desktop, then copy the content to Json Editor Online for parsing!
* Alternatively, you can directly use mitmweb to browse and operate through the web GUI.
mitmweb.
After sniffing, observing, filtering, and testing, you will understand how the APP API operates, allowing you to use it to scrape data with a crawler.
* After collecting the necessary information, remember to close mitmproxy and change the phone’s network Proxy settings back to automatic to use the internet normally.
How Should the APP Protect Itself?
If you find that the APP cannot be used after mounting mitmproxy, or the returned content is encoded, it means the APP has implemented protections.
Method (1):
Generally, this involves placing a copy of the certificate information within the APP. If the current HTTPS certificate does not match the information in the APP, access is denied. For more details, see this or look for resources on SSL Pinning. The downside is that you need to pay attention to the certificate’s validity period!
{:target="_blank"}](/assets/46410aaada00/1*31rODDIlYPidTP3L8W_C7A.jpeg)
https://medium.com/@dzungnguyen.hcm/ios-ssl-pinning-bffd2ee9efc
Method (2):
Before transmitting data, the APP encrypts the data, and the API backend decrypts it to obtain the original request content. Similarly, the API encrypts the returned content before sending it back, and the APP decrypts it upon receipt. This process is cumbersome and resource-intensive, but it is indeed a method; as far as I know, some digital banks use this method for protection!
However…
Method 1 still has ways to be bypassed: How to Bypass SSL Pinning on iOS 12.
Method 2 can also be compromised through reverse engineering to obtain the keys used for encoding and encryption.
⚠️ There is no 100% security ⚠️
Or simply leave a hole for it to crawl, collect various evidence, and then resolve it legally (？)
Still, the saying goes:
“NEVER TRUST THE CLIENT.”
More Ways to Use mitmproxy:
1. Use mitmdump
In addition to mitmproxy and mitmweb, mitmdump can directly export all records to a text file:
1
mitmdump -w /log.txt
You can also use Method (2) with a Python script to set parameters and filter traffic:
1
mitmdump -ns examples/filter.py -r /log.txt -w /result.txt
2. Use a Python script for request parameter settings, access control, and redirection:
1
2
3
4
5
6
7
8
9
10
11
12
13
from mitmproxy import http
def request(flow: http.HTTPFlow) -> None:
# pretty_host takes the "Host" header of the request into account,
# which is useful in transparent mode where we usually only have the IP
# otherwise.
# Request parameter settings Example:
flow.request.headers['User-Agent'] = 'MitmProxy'
if flow.request.pretty_host == "123.com.tw":
flow.request.host = "456.com.tw"
# Redirect all access from 123.com.tw to 456.com.tw
To enable mitmproxy with the script, add the parameter:
1
2
3
4
5
mitmproxy -s /redirect.py
or
mitmweb -s /redirect.py
or
mitmdump -s /redirect.py
A Pitfall
When using mitmproxy to observe HTTP 1.1 requests with Accept-Ranges: bytes and Content-Range for long connections that continuously fetch resources, it will wait until the entire response is returned before displaying it, rather than showing it in segments while using persistent connections to continue downloading!
Further Reading
- Using a Check-in Reward APP as an Example to Create a Daily Automatic Check-in Script
- How to Create an Interesting Engineering CTF Competition
- Revealing a Clever Website Vulnerability Discovered a Few Years Ago
- iOS 15 / MacOS Monterey Safari Will Be Able to Hide Real IP
Postscript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
---
title: "iPlayground 2019 Experience"
date: 2019-09-20
---
Since I don't have domain permissions to obtain SSL certificate information, I can't implement it; looking at the code, it doesn't seem too difficult. Although there's no 100% secure method, having an additional layer of protection can at least make it a bit safer. Continuing to attack would require a lot of time for research, but it should deter about 90% of crawlers!
This article might be a bit lacking in substance, as I've neglected Medium for a while (went off to play with my DSLR). This is mainly to warm up my writing skills ahead of this weekend's [iPlayground 2019](https://iplayground.io/2019/){:target="_blank"}; I'm looking forward to this year's agenda 🤩 and hope to come back with more quality articles!
> **_\[2019/02/22 Updated Article\] [What is the Experience of iPlayground 2019?](../4079036c85c2/)_**
If you have any questions or suggestions, feel free to [contact me](https://www.zhgchg.li/contact){:target="_blank"}.
This article was first published on Medium ➡️ Click Here
Automatically converted and synchronized using ZMediumToMarkdown and Medium-to-jekyll-starter.
