Firebase Firestore + Functions｜Build Testable API Services Fast for Push Notification Analytics

点击这里查看本文章简体中文版本。

點擊這裡查看本文章正體中文版本。

This post was translated with AI assistance — let me know if anything sounds off!

---

Quickly Build a Testable API Service Using Firebase Firestore + Functions

When Push Notification Statistics Meet Firebase Firestore + Functions

Photo by Carlos Muza

Preface

Precise Push Notification Statistics Function

Recently, we wanted to add a feature to the app. Before implementation, we could only use the success or failure of sending data from the backend to APNS/FCM as the push notification base and record push notification clicks to calculate the “click-through rate.” However, this method is very inaccurate because the base includes many invalid devices, such as deleted apps (which may not become invalid immediately) and devices with push permissions turned off, all of which still return a success response when posting from the backend.

After iOS 10, you can implement a Notification Service Extension to secretly call an API for statistics when the push notification banner appears. The advantage is high accuracy, as the call only happens when the user actually sees the notification banner. If the app is deleted, notifications are turned off, or banners are disabled, no action occurs. The banner appearance equals a push notification shown, which serves as the base for calculating an accurate click-through rate by comparing it with the number of clicks.

For detailed principles and implementation methods, please refer to the previous article: “i OS ≥ 10 Notification Service Extension Application (Swift)”

Currently, testing shows the app’s loss rate is likely 0%. A common real-world example is Line’s point-to-point message encryption (push notifications are encrypted and only decrypted on the phone before display).

Question

The app side’s workload is actually not that big. Both iOS and Android only need to implement similar functions (but Android is more complicated if targeting the Chinese market, as it requires implementing push notification frameworks for multiple platforms). The bigger challenge lies in backend and server load handling, because sending push notifications simultaneously calls APIs to log records, which can max out the server’s max connections. This issue is even worse if using RDBMS to store records. If there is data loss in statistics, it usually happens at this stage.

Here, you can record logs by writing to files and display statistics when querying.

Also, thinking later about the scenario of going out and coming back at the same time, the volume might not be as large as imagined; because push notifications are not sent all at once in hundreds of thousands or millions, but in batches of a few at a time; as long as it can handle the number of requests sent and received in batches simultaneously!

Prototype

Due to the issues considered earlier, the backend requires significant effort to research and modify, and the market may not necessarily care about the results; so we decided to first use available resources to create a prototype to test the waters.

Here, we chose Firebase services commonly used by apps, including Functions and Firestore features.

Firebase Functions

Functions is a serverless service provided by Google. You only need to write the program logic, and Google automatically handles the servers and runtime environment, so you don’t have to worry about server scaling or traffic issues.

Firebase Functions are actually Google Cloud Functions but can only be written in JavaScript (node.js). I haven’t tried it, but if you use Google Cloud Functions with other languages and also import Firebase services, I think it should work as well.

Used in an API, it means I can write a node.js file to get a physical URL (e.g., my-project.cloudfunctions.net/getUser) and implement the logic to obtain Request information and provide the corresponding Response.

Previously wrote an article about Google Functions: “Using Python + Google Cloud Platform + Line Bot to Automate Routine Tasks”

Firebase Functions require enabling a Blaze project (pay as you go) to use.

Firebase Firestore

Firebase Firestore is a NoSql database used to store and manage data.

By integrating Firebase Functions, you can import Firestore during the request to operate the database, then respond to the user, enabling the creation of a simple Restful API service!

Let’s start hands-on practice!

Installing Node.js Environment

It is recommended to use NVM, a Node.js version management tool, for installation and management (similar to using pyenv for Python).

Go to the NVM Github project and copy the installation shell script:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh \| bash

If an error occurs during installation, please make sure you have a ~/.bashrc or ~/.zshrc file. If not, create one using touch ~/.bashrc or touch ~/.zshrc, then run the install script again.

Next, you can use nvm install node to install the latest version of node.js.

You can run npm --version to verify that npm is installed successfully and check its version:

Deploy Firebase Functions

Install Firebase-tools:

npm install -g firebase-tools

After successful installation, please enter the following for first-time use:

firebase login

Complete Firebase login authentication.

Start Project:

firebase init

Note the path where Firebase init is located:

You're about to initialize a Firebase project in this directory:

Here you can choose which Firebase CLI tools to install. Use the “↑” and “↓” keys to navigate and the “spacebar” to select; you can choose to install only “Functions” or select “Firestore” as well.

=== Functions Setup

• Language Selection: “JavaScript”

• Regarding “use ESLint to catch probable bugs and enforce style” syntax style checking, YES / NO both are acceptable.

• install dependencies with npm? YES

===Emulators Setup

You can test Functions and Firestore features and settings locally without affecting usage limits and without waiting for deployment to go live.

Install according to personal needs. I installed it but didn’t use it… because it’s just a minor feature.

Coding!

Go to the path noted above, find the functions folder, and open the index.js file inside with an editor.

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

exports.hello = functions.https.onRequest((req, res) => {
    const targetID = req.query.targetID
    const action = req.body.action
    const name = req.body.name

    res.send({"targetID": targetID, "action": action, "name": name});
    return
})

Paste the above content. We defined a route interface /hello that returns URL Query ?targetID=, POST action, and name parameter information.

After editing & saving, return to the console:

firebase deploy

Remember to run the firebase deploy command after every modification for the changes to take effect.

Start verification & deploy to Firebase…

You might need to wait a moment. After Deploy complete!, your first Request & Response webpage will be ready!

At this point, you can return to the Firebase -> Functions page:

You will see the interface and URL location just written.

Copy the URL below and paste it into PostMan to test:

Remember to select x-www-form-urlencoded for the POST Body.

Success!

Log

We can use in the code:

functions.logger.log("log:", value);

Perform log recording.

You can view the log results in Firebase -> Functions -> Logs:

Example Goal

Create an API for adding, updating, deleting, querying articles, and liking posts

We aim to achieve the functionality design of a Restful API, so we can no longer use the pure Path method from the example above. Instead, we need to use the Express framework to accomplish this.

POST Add New Article

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Insert
app.post('/', async (req, res) => { // The POST here refers to the HTTP Method POST
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;

    if (title == null \|\| content == null \|\| author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author, "created_at": new Date()};
    await admin.firestore().collection('posts').add(post);
    res.status(201).send({"message":"Added successfully!"});
});

exports.post= functions.https.onRequest(app); // The POST here refers to the /post path

Now we switch to using Express to handle network requests. Here, we first add a POST method for the / path. The last line indicates that all paths are under /post. Next, we will add APIs for update and delete.

After successfully deploying with firebase deploy, return to Post Man to test:

After a successful Postman request, you can check Firebase -> Firestore to verify if the data has been correctly written:

PUT Edit Article

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Update
app.put("/:id", async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"}); 
    } else if (title == null \|\| content == null \|\| author == null) {
        return res.status(400).send({"message":"Invalid parameters!"});
    }

    var post = {"title":title, "content":content, "author": author};
    await admin.firestore().collection('posts').doc(req.params.id).update(post);
    res.status(200).send({"message":"Update successful!"});
});

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding; remember to change the Post Man HTTP method to PUT.

DELETE Delete Article

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Delete
app.delete("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection("posts").doc(req.params.id).delete();
    res.status(200).send({"message":"Post deleted successfully!"});
})

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding; remember to change the Post Man HTTP Method to DELETE.

Add, modify, and delete are done, now let’s do the query!

SELECT Query Articles

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Select List
app.get('/', async (req, res) => {
    const posts = await admin.firestore().collection('posts').get();
    var result = [];
    posts.forEach(doc => {
      let id = doc.id;
      let data = doc.data();
      result.push({"id":id, ...data})
    });
    res.status(200).send({"result":result});
});

// Select One
app.get("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Article not found!"});
    }

    res.status(200).send({"result":{"id":doc.id, ...doc.data()}});
});

exports.post= functions.https.onRequest(app);

Deployment & testing method is the same as adding. Remember to change the Post Man HTTP Method to GET and switch the Body back to none.

InsertOrUpdate?

Sometimes we need to update when a value exists and add when it doesn’t. In this case, we can use set with merge: true:

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// InsertOrUpdate
app.post("/tag", async (req, res) => {
    const name = req.body.name;

    if (name == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var tag = {"name":name};
    await admin.firestore().collection('tags').doc(name).set({created_at: new Date()}, {merge: true});
    res.status(201).send({"message":"Successfully added!"});
});

exports.post= functions.https.onRequest(app);

Here, using adding a tag as an example, the deployment & testing method is the same as adding. You can see that Firestore does not repeatedly add new data.

Article Like Counter

Assuming our article data now includes a likeCount field to record the number of likes, how should we proceed?

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Like Post
app.post("/like/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection('posts').doc(req.params.id).set({likeCount: increment}, {merge: true});
    res.status(201).send({"message":"Like successful!"});
});

exports.post= functions.https.onRequest(app);

Using the increment variable allows you to directly perform the action of retrieving the value and adding 1.

High Traffic Article Like Counter

Because Firestore has write speed limits:

A document can only be written once per second, so when there are many likes, simultaneous requests may become very slow.

The official solution “Distributed counters” is not very complex technically. It simply uses multiple distributed likeCount fields to count, then sums them up when reading.

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Distributed counters Like Post
app.post("/like2/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    //1~10
    await admin.firestore().collection('posts').doc(req.params.id).collection("likeCounter").doc("likeCount_"+(Math.floor(Math.random()*10)+1).toString())
    .set({count: increment}, {merge: true});
    res.status(201).send({"message":"Like successful!"});
});


exports.post= functions.https.onRequest(app);

The above method distributes fields to record the Count, avoiding slow writes; however, having too many distributed fields increases read costs ($$), but it should still be cheaper than adding a new record each time a like is made.

Using the Siege Tool for Stress Testing

Install siege using brew

brew install siege

p.s If you get brew: command not found, please install the brew package manager first:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

After installation, you can run:

siege -c 100 -r 1 -H 'Content-Type: application/json' 'https://us-central1-project.cloudfunctions.net/post/like/id POST {}'

Perform stress testing:

• -c 100 ：100 tasks running concurrently

• -r 1 ：Execute 1 request per task run

• -H ‘Content-Type: application/json’ ：Add this when using POST requests

• ‘https://us-central1-project.cloudfunctions.net/post/like/id POST {}’ ：POST URL, Post Body (ex: {“name”:”1234”} )

After execution, you can see the results:

successful_transactions: 100 means 100 transactions were all successful.

You can check Firebase -> Firestore to see if there is any Loss Data:

Success!

Complete Example Code

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

// Insert
app.post('/', async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;

    if (title == null \|\| content == null \|\| author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author, "created_at": new Date()};
    await admin.firestore().collection('posts').add(post);
    res.status(201).send({"message":"Added successfully!"});
});

// Update
app.put("/:id", async (req, res) => {
    const title = req.body.title;
    const content = req.body.content;
    const author = req.body.author;
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"}); 
    } else if (title == null \|\| content == null \|\| author == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var post = {"title":title, "content":content, "author": author};
    await admin.firestore().collection('posts').doc(req.params.id).update(post);
    res.status(200).send({"message":"Updated successfully!"});
});

// Delete
app.delete("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection("posts").doc(req.params.id).delete();
    res.status(200).send({"message":"Deleted successfully!"});
});

// Select List
app.get('/', async (req, res) => {
    const posts = await admin.firestore().collection('posts').get();
    var result = [];
    posts.forEach(doc => {
      let id = doc.id;
      let data = doc.data();
      result.push({"id":id, ...data})
    });
    res.status(200).send({"result":result});
});

// Select One
app.get("/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    res.status(200).send({"result":{"id":doc.id, ...doc.data()}});
});

// InsertOrUpdate
app.post("/tag", async (req, res) => {
    const name = req.body.name;

    if (name == null) {
        return res.status(400).send({"message":"Parameter error!"});
    }

    var tag = {"name":name};
    await admin.firestore().collection('tags').doc(name).set({created_at: new Date()}, {merge: true});
    res.status(201).send({"message":"Added successfully!"});
});

// Like Post
app.post("/like/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    await admin.firestore().collection('posts').doc(req.params.id).set({likeCount: increment}, {merge: true});
    res.status(201).send({"message":"Like successful!"});
});

// Distributed counters Like Post
app.post("/like2/:id", async (req, res) => {
    const doc = await admin.firestore().collection('posts').doc(req.params.id).get();
    const increment = admin.firestore.FieldValue.increment(1)

    if (!doc.exists) {
        return res.status(404).send({"message":"Post not found!"});
    }

    //1~10
    await admin.firestore().collection('posts').doc(req.params.id).collection("likeCounter").doc("likeCount_"+(Math.floor(Math.random()*10)+1).toString())
    .set({count: increment}, {merge: true});
    res.status(201).send({"message":"Like successful!"});
});


exports.post= functions.https.onRequest(app);

Back to the topic, push notification statistics

Back to what we initially wanted to do: push notification statistics.

index.js:

const functions = require('firebase-functions');
const admin = require('firebase-admin');
const express = require('express');
const cors = require('cors');
const app = express();

admin.initializeApp();
app.use(cors({ origin: true }));

const vaildPlatformTypes = ["iOS","Android"]
const vaildActionTypes = ["clicked","received"]

// Insert Log
app.post('/', async (req, res) => {
    const increment = admin.firestore.FieldValue.increment(1);
    const platformType = req.body.platformType;
    const pushID = req.body.pushID;
    const actionType =  req.body.actionType;

    if (!vaildPlatformTypes.includes(platformType) \|\| pushID == undefined \|\| !vaildActionTypes.includes(actionType)) {
        return res.status(400).send({"message":"Parameter error!"});
    } else {
        await admin.firestore().collection(platformType).doc(actionType+"_"+pushID).collection("shards").doc((Math.floor(Math.random()*10)+1).toString())
        .set({count: increment}, {merge: true})
        res.status(201).send({"message":"Record successful!"});
    }
});

// View Log
app.get('/:type/:id', async (req, res) => {
    // received
    const receivedDocs = await admin.firestore().collection(req.params.type).doc("received_"+req.params.id).collection("shards").get();
    var received = 0;
    receivedDocs.forEach(doc => {
      received += doc.data().count;
    });

    // clicked
    const clickedDocs = await admin.firestore().collection(req.params.type).doc("clicked_"+req.params.id).collection("shards").get();
    var clicked = 0;
    clickedDocs.forEach(doc => {
        clicked += doc.data().count;
    });
    
    res.status(200).send({"received":received,"clicked":clicked});
});

exports.notification = functions.https.onRequest(app);

Add Push Notification Record

View Push Notification Statistics

https://us-centra1-xxx.cloudfunctions.net/notification/iOS/1

Also created an interface to track push notification statistics.

Pitfalls

Because I was not very familiar with node.js usage, at first when adding data I forgot to add await. Combined with write speed limits, this caused data loss under high traffic conditions…

Pricing

Don’t forget to refer to the pricing strategies of Firebase Functions & Firestore.

Functions

• https://cloud.google.com/functions/pricing?hl=zh-tw

Processing Time

Internet

Cloud Functions offers a permanent free tier for compute time resources, including GB-seconds and GHz-seconds of compute time. Besides 2 million invocations, the free tier also provides 400,000 GB-seconds and 200,000 GHz-seconds of compute time, along with 5 GB of internet egress per month.

Firestore

• https://cloud.google.com/firestore/pricing?hl=zh-tw

• Calculation Example

Prices may change at any time. Please refer to the official website for the latest information.

Conclusion

As stated in the title, “For Testing Purposes,” “For Testing Purposes,” “For Testing Purposes,” it is not recommended to use the above services in a production environment or as the core of a product launch.

Expensive and Hard to Migrate

I once heard that a fairly large service was built using Firebase. However, as data and traffic grew, the costs skyrocketed. Migrating was also very difficult; the code was manageable, but moving the data was extremely hard. It can only be said that saving a little money early on led to huge losses later, which was not worth it.

For Testing Only

For the above reasons, I personally recommend using Firebase Functions + Firestore API services only for testing or prototype product demonstrations.

More Features

Functions can also integrate Authentication and Storage (file upload), but I haven’t explored these parts.

References

• https://firebase.google.com/docs/firestore/query-data/queries

• https://coder.tw/?p=7198

• https://firebase.google.com/docs/firestore/solutions/counters#node.js_1

• https://javascript.plainenglish.io/firebase-cloud-functions-tutorial-creating-a-rest-api-8cbc51479f80

Further Reading

• Automate Routine Tasks with Python + Google Cloud Platform + Line Bot

• i OS ≥ 10 Notification Service Extension Application (Swift)

• Forward Gmail Emails to Slack Using Google Apps Script

If you have any questions or feedback, feel free to contact me.

---

This post was originally published on Medium (View original post), and automatically converted and synced by ZMediumToMarkdown.

Improve this page on Github.

Follow Me on Medium 1K+ Followers