Automate Routine Tasks with Python, Google Cloud Platform & Line Bot｜Daily Check-In Script Example

点击这里查看本文章简体中文版本。

點擊這裡查看本文章正體中文版本。

This post was translated with AI assistance — let me know if anything sounds off!

---

Automate Routine Tasks Using Python + Google Cloud Platform + Line Bot

Using a Check-in Reward App as an Example to Create a Daily Auto Check-in Script

Photo by Paweł Czerwiński

Origin

I have always used Python for small tools; some are serious, like automating data scraping and report generation at work, while others are casual, such as scheduling automatic information retrieval or letting scripts perform tasks that were originally done manually.

I have always handled “automation” in a straightforward way by running a Python script on a dedicated computer. The advantage is simplicity and convenience, but the downside is needing a device connected to the internet and power. Even a Raspberry Pi consumes a small amount of electricity and internet data. Also, remote control to start or stop the script is difficult (actually possible, but complicated). This time, during a work break, I explored free and cloud-based methods.

Goal

Move Python scripts to the cloud for execution, schedule automatic runs, and enable remote on/off control via the internet.

This article uses a clever trick I devised to create an automated check-in script for reward-based check-in apps, enabling daily automatic check-ins without manually opening the app; it also sends me a notification upon completion.

Notification complete!

Chapter Order of This Article

1. Using Proxyman for Man-in-the-Middle Attack API Sniffing

2. Write a Python script to spoof APP API requests (simulate check-in actions)

3. Moving a Python Script to Google Cloud

4. Setting Up Automatic Scheduling in Google Cloud

• Due to the sensitive nature, this article will not reveal which check-in reward app is involved. Readers can explore and use similar apps on their own.

• If you only want to learn how to automate with Python, you can skip the first half about Man in the Middle attack API sniffing and start from Chapter 3.

Tools Used

• Proxyman: Man-in-the-middle attack API sniffing

• Python: Writing scripts

• Linebot: Send script execution result notifications to yourself

• Google Cloud Function: Python script hosting service

• Google Cloud Scheduler: Automated scheduling service

1. Using Proxyman for Man in the Middle Attack API Sniffing

Previously, I published an article titled “The APP uses HTTPS transmission, but the data was still stolen.” The principle is similar, but this time I used Proxyman instead of mitmproxy; both are free, but Proxyman is easier to use.

• Go to the official website https://proxyman.io/ to download the Proxyman tool

• After downloading, launch Proxyman and install the Root certificate (to perform a Man-in-the-Middle attack to decrypt HTTPS traffic).

“Certificate” -> “Install Certificate On this Mac” -> “Installed & Trusted”

After installing the computer’s Root certificate, switching to the phone:

“Certificate” -> “Install Certificate On iOS” -> “Physical Devices…”

Set up the proxy on your phone as instructed, then complete the certificate installation and activation.

• Open the app on your phone whose API transmission you want to sniff.

At this point, Proxyman on the Mac will show the sniffed traffic. Click on the device IP and then the APP API domain you want to check; the first time you view it, you need to click “Enable only this domain” before the traffic can be unpacked.

After enabling “Enable only this domain,” you can see the newly intercepted traffic with the original Request and Response information:

We use this method to sniff which API endpoint the app calls during check-in and what data it sends. We record this information to later simulate the request directly using Python.

⚠️ Note that some APP token information may change, causing future Python simulated requests to fail. It is important to understand how APP token exchange works.

⚠️ If you are sure Proxyman is working properly, but the app cannot send requests while using Proxyman, it means the app may have SSL Pinning; currently, there is no solution and you have to give up.

⚠️App developers who want to learn how to prevent sniffing can refer to the previous article.

Assuming the information we have is as follows:

POST /usercenter HTTP/1.1
Host: zhgchg.li
Content-Type: application/x-www-form-urlencoded
Cookie: PHPSESSID=dafd27784f94904dd586d4ca19d8ae62
Connection: keep-alive
Accept: */*
User-Agent: (iPhone12,3;iOS 14.5)
Content-Length: 1076
Accept-Language: zh-tw
Accept-Encoding: gzip, deflate, br
AuthToken: 12345

2. Write a Python script to forge APP API requests (simulate check-in actions)

Before writing the Python script, we can use Postman to test the parameters and identify which are necessary or time-sensitive; copying them directly is also acceptable.

checkIn.py:

import requests
import json

def main(args):
    results = {}
    try:
      data = { "action" : "checkIn" }
      headers = { "Cookie" : "PHPSESSID=dafd27784f94904dd586d4ca19d8ae62", 
      "AuthToken" : "12345",
      "User-Agent" : "(iPhone12,3;iOS 14.5)"
      }
      
      request = requests.post('https://zhgchg.li/usercenter', data = data, headers = headers)
      result = json.loads(request.content)
      if result['status_code'] == 200:
        return "CheckIn Success!"
      else:
        return result['message']
    except Exception as e:
      return str(e)

⚠️ The purpose of args in main(args) will be explained later. For local testing, just use main(True).

Use the Requests package to perform HTTP Requests for us. If the following occurs:

ImportError: No module named requests

Please install the package first using pip install requests.

Add Execution Result Linebot Notification:

This part is very simple; it is for reference only and just for notifying myself.

• Go to & activate Line Developers Console

• Create a Provider

• Select “Create a Messaging API channel”

Next, after filling in the basic information, click “Create” to submit and create.

• After setting up, locate the “Your user ID” section under the first “Basic settings” tab. This is your User ID.

• After setting up, select the “Messaging API” tab and scan the QR code to add the bot as a friend.

• Scroll down to the “Channel access token” section and click “Issue” to generate the token.

• The copied Token allows us to send messages to users.

With the User Id and Token, we can now send messages to ourselves.

Since no other functions are required, there is no need to install the Python Line SDK; simply send HTTP requests directly.

After connecting with the previous Python script…

checkIn.py:

import requests
import json

def main(args):
    results = {}
    try:
      data = { "action" : "checkIn" }
      headers = { "Cookie" : "PHPSESSID=dafd27784f94904dd586d4ca19d8ae62", 
      "AuthToken" : "12345",
      "User-Agent" : "(iPhone12,3;iOS 14.5)"
      }
      
      request = requests.post('https://zhgchg.li/usercenter', data = data, headers = headers)
      result = json.loads(request.content)
      if result['status_code'] == 200:
        sendLineNotification("CheckIn Success!")
        return "CheckIn Success!"
      else:
        sendLineNotification(result['message'])
        return result['message']
    except Exception as e:
      sendLineNotification(str(e))
      return str(e)
      
def sendLineNotification(message):
    data = {
        "to" : "Insert your User ID here",
        "messages" : [
            {
                "type" : "text",
                "text" : message
            }
        ]
    }
    headers = {
        "Content-Type" : "application/json",
        "Authorization" : "Insert channel access token here"
    }
    request = requests.post('https://api.line.me/v2/bot/message/push',json = data, headers = headers)

Test if the notification was sent successfully:

Success!

A brief side note: I originally planned to use Gmail SMTP to send emails for notifications, but after moving to Google Cloud, I found it was not possible…

3. Moving the Python Script to Google Cloud

The basics have been covered, now let’s move on to the main part of this article: deploying the Python script to the cloud.

At first, I considered using Google Cloud Run, but found it too complicated and didn’t want to spend time researching it since my needs were minimal and didn’t require many features; so I used Google Cloud Function, a serverless solution. It is commonly used to build serverless web services.

• If you haven’t used Google Cloud before, please first go to the Console to create a new project and set up your billing information.

• On the project console homepage, click “Cloud Functions” under Resources.

• Select “Create Function” above

• Enter Basic Information

⚠️ Note down the “Trigger URL”

Selectable Region:

• US-WEST1, US-CENTRAL1, and US-EAST1 regions offer a free tier for Cloud Storage services.

• asia-east2 (Hong Kong) is closer to us but requires a small Cloud Storage fee.

⚠️When creating Cloud Functions, Cloud Storage is required to host the code.

⚠️ For detailed pricing, please refer to the end of the article.

Trigger Condition: HTTP

Verification: According to the requirement, I want to execute the script by clicking an external link, so I choose “Allow unauthenticated calls”; if authentication is required, the Scheduler service must also be configured accordingly.

Variables, network, and advanced settings can be configured in Variables for Python use (so you don’t need to modify the Python code when parameters change):

How to call in Python:

import os

def main(request):
  return os.environ.get('test', 'DEFAULT VALUE')

No other settings need to be changed, just click “Save” -> “Next”.

• Set the runtime to “Python 3.x” and paste the written Python script. Change the entry point to “main”.

Add main(args). As mentioned earlier, this service is mainly used for serverless web; thus, args is actually a Request object. You can get the HTTP GET query and HTTP POST body data from it, as shown below:

Get GET Query information:
request_args = args.args

example: ?name=zhgchgli => request_args = [“name”:”zhgchgli”]

Get POST Body data:
request_json = request.get_json(silent=True)

example: name=zhgchgli => request_json = [“name”:”zhgchgli”]

If using Postman to test POST, remember to use “Raw + JSON” for the POST data; otherwise, nothing will be sent:

• After the code section is done, switch to “requirements.txt” and enter the required package dependencies:

We use the “request” package to call APIs. This package is not included in the native Python library, so we need to add it here:

requests>=2.25.1

Here, the version is specified as ≥ 2.25.1, but you can also omit the version and just enter requests to install the latest version.

• Once everything is OK, click “Deploy” to start the deployment.

It takes about 1 to 3 minutes to complete the deployment.

• After deployment, you can use the previously noted “Trigger URL” to execute and verify if it runs correctly, or use “Actions” -> “Test Function” to perform a test.

If a 500 Internal Server Error occurs, it means there is a program error. You can click the name to view the “Logs” and find the cause there:

UnboundLocalError: local variable 'db' referenced before assignment

• Click the name to enter, then click “Edit” to modify the script content.

The test went smoothly and is now complete! We have successfully migrated the Python script to the cloud.

Additional Information About Variables

According to our requirements, we need a place to store and read the token for the check-in app; since the token may expire, it needs to be requested again and saved for use in the next execution.

To dynamically pass variables from outside into the script, you can use the following methods:

• [Read Only] Runtime environment variables mentioned above

• [Temp] Cloud Functions provides a /tmp directory for writing and reading files during execution, but it is deleted after completion. For details, please refer to the official documentation.

• [Read Only] GET/POST Data Transmission

• [Read Only] Attach Additional File

Using the relative path ./ in the program allows reading files, but only for reading, not for dynamic modification; to modify, you must do so in the console and redeploy.

To read and dynamically modify, you need to connect to other GCP services, such as Cloud SQL, Google Storage, Firebase Cloud Firestore…

• [Read & Write] Here, I chose Firebase Cloud Firestore because it is currently the only option with a free usage tier.

After following the Getting Started steps and creating a Firebase project, go to the Firebase console:

In the left menu, find “Cloud Firestore” -> “Add Collection”

Enter collection ID.

Input data content.

A collection can have multiple documents, and each document can have its own field content; this offers great flexibility in usage.

Usage in Python:

Please go to GCP Console -> IAM & Admin -> Service Accounts and follow the steps below to download the authentication private key file:

First, select an account:

Below “Add Key” -> “Create New Key”

Select “JSON” to download the file.

Place this JSON file in the same project directory as the Python file.

In Local Development Environment:

pip install --upgrade firebase-admin

Install the firebase-admin package.

In Cloud Functions, you need to add firebase-admin to the requirements.txt file.

After setting up the environment, we can now read the data we just added:

firebase_admin.py:

import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore

if not firebase_admin._apps:
  cred = credentials.Certificate('./authentication.json')
  firebase_admin.initialize_app(cred)
# Initializing app multiple times causes the following error:
# providing an app name as the second argument. In most cases you only need to call initialize_app() once. But if you do want to initialize multiple apps, pass a second argument to initialize_app() to give each app a unique name.
# So to be safe, check if initialized before calling initialize_app

db = firestore.client()
ref = db.collection(u'example')  # collection name
stream = ref.stream()
for data in stream:
  print("id:"+data.id+","+data.to_dict())

If you are using Cloud Functions, you can upload the authentication JSON file together, or modify the connection syntax as follows when using it:

cred = credentials.ApplicationDefault()
firebase_admin.initialize_app(cred, {
  'projectId': project_id,
})

db = firestore.client()

If you see Failed to initialize a certificate credential., please check if the authentication JSON is correct.

For adding and deleting more operations, please refer to the official documentation.

4. Set Up Automatic Scheduling in Google Cloud

After having the script, the next step is to automate its execution to achieve our ultimate goal.

• Go to the Google Cloud Scheduler console homepage

• Click “Create Task” above

• Enter basic job information

Execution Frequency: Same as crontab input format. If you are not familiar with crontab syntax, you can directly use the amazing website crontab.guru:

It can directly translate the actual meaning of the syntax you set. (Click next to see the next execution time)

Here I set 15 1 * * * because the check-in only needs to run once a day, scheduled at 1:15 AM daily.

URL part: Enter the previously noted “trigger URL”

Time Zone: Enter “Taiwan” and select Taipei Standard Time

HTTP Method: According to the previous Python code, we can just use Get.

If you have set up “Verification” earlier, remember to expand “SHOW MORE” to configure the verification settings.

After filling in all the fields, click “Create”.

• After successfully creating, you can choose “Run Now” to test if it works properly.

• You can view the execution results and the last execution date.

⚠️ Please note that a “failure” result only applies when the web status code is 400–500 or there is an error in the Python program.

All done!

We have achieved the goal of uploading routine Python scripts to the cloud and setting up automated scheduling for execution.

Pricing Method

Another important aspect is the pricing method; Google Cloud and Linebot are not entirely free services, so understanding their fees is crucial. Otherwise, spending too much money on a small script isn’t worth it—you might as well just leave your computer running.

Linebot

Refer to the official pricing information, with up to 500 messages free per month.

Google Cloud Functions

Refer to the official pricing information, which includes 2 million invocations per month, 400,000 GB-seconds and 200,000 GHz-seconds of compute time, and 5 GB of internet egress traffic.

Google Firebase Cloud Firestore

Refer to the official pricing information: 1 GB storage, 10 GB monthly bandwidth, 50,000 reads per day, 20,000 writes/deletes per day; enough for light use!

Google Cloud Scheduler

Refer to the official pricing information, each account can set up 3 free jobs.

The free quota above is more than enough for the script!

Google Cloud Storage Conditional Free Tier

Hiding here and there, you still can’t avoid services that may charge fees.

After creating Cloud Functions, two Cloud Storage entities are automatically created:

If you selected US-WEST1, US-CENTRAL1, or US-EAST1 for Cloud Functions, you are eligible for the free usage quota:

I chose US-CENTRAL1, that’s correct. You can see the first Cloud Storage instance is located in US-CENTRAL1, but the second one is labeled multiple regions in the US; I personally estimate this will incur charges.

Refer to the official pricing information; prices vary depending on the host region.

The code isn’t much, so I estimate the minimum monthly charge is around 0.0X0 yuan (?

⚠️The above information was recorded as of 2021/02/21. Actual prices may vary and are for reference only.

Pricing and Budget Control Notification

just in case… If the usage exceeds the free limit and charges begin, I want to receive a notification; this will prevent unexpected billing spikes caused by possible program errors from going unnoticed.

• Go to the Console

• Find the “Billing Feature” Card:

Click “View detailed deduction records” to enter.

• Expand the left menu and go to the “Budget & Alerts” feature

• Click on “Set Budget” above

• Enter Custom Name

Next step.

• Amount, enter “Target Amount”, you can input $1, $10; we don’t want to spend too much on small things.

Next step.

You can set the action to trigger a notification when the budget reaches a certain percentage.

Check “Send alerts via email to billing administrators and users” so that notifications are received immediately when conditions are met.

Click “Done” to submit and save.

When the budget is exceeded, we can immediately know to avoid incurring more costs.

Summary

Human energy is limited. In today’s flood of technology and information, every platform and service tries to drain our limited energy. If we can use some automation scripts to share the load of our daily lives, accumulating little by little, it will save us more energy to focus on what truly matters!

Further Reading

• Slack Builds Fully Automated WFH Employee Health Reporting System

• Crashlytics + Big Query Build a More Real-Time and Convenient Crash Tracking Tool

• Crashlytics + Google Analytics Automatic Query for App Crash-Free Users Rate

• The app uses HTTPS for transmission, but the data was still stolen.

• How to Create an Engaging Engineering CTF Competition

• iOS 14 Clipboard Privacy Concerns: The Dilemma Between Privacy and Convenience

• Forward Gmail Emails to Slack Using Google Apps Script

If you have any questions or feedback, feel free to contact me.

If you have automation-related optimization needs, feel free to contact me. Thank you.

---

This post was originally published on Medium (View original post), and automatically converted and synced by ZMediumToMarkdown.

Improve this page on Github.

Follow Me on Medium 1K+ Followers