CI/CD Practical Guide (Part 1): What is CI/CD? How to Build a Stable and Efficient Development Team through CI/CD? Tool Selection?
Using the App (iOS) Team as an example, this guide introduces CI/CD from scratch and the tangible value it can bring after implementation.
ℹ️ℹ️ℹ️ The following content is translated by OpenAI.
Click here to view the original Chinese version. | 點此查看本文中文版
CI/CD Practical Guide (Part 1): What is CI/CD? How to Build a Stable and Efficient Development Team through CI/CD? Tool Selection?
Using the App (iOS) Team as an example, this guide will help you understand CI/CD from scratch and the tangible value it can bring after implementation.
Photo by Leif Christoph Gottwald
Introduction
After going through the experience of setting up App CI/CD in two different development teams, I finally found time to organize my thoughts from “why do it” to “how to do it.” I can’t guarantee that this is the most standard CI/CD workflow, but it is definitely a valuable starting point to help your team begin implementation, improve product stability, and enhance overall development efficiency.
Chapters
This series of articles will start with “What is CI/CD and what value does it bring?” Then, we will walk through the hands-on implementation of “How to set up a CI/CD environment using GitHub Actions + self-hosted Runner” and “Implementing CI and CD using App development as an example.” Finally, we will introduce how to “Use Google Apps Script Web App in conjunction with GitHub Actions to create a convenient app packaging platform for cross-team use.” I hope this series will be helpful to you.
- CI/CD Practical Guide (Part 1): What is CI/CD? How to Build a Stable and Efficient Development Team through CI/CD? Tool Selection?
- CI/CD Practical Guide (Part 2): Comprehensive Guide to Using and Setting Up GitHub Actions and Self-hosted Runner
- CI/CD Practical Guide (Part 3): Implementing CI and CD Workflows for App Projects Using GitHub Actions
- CI/CD Practical Guide (Part 4): Building a Free and Easy-to-Use Packaging Tool Platform Using Google Apps Script Web App and GitHub Actions
Final Results
Without further ado, here are the final results.
CI/CD — All developed using GitHub Actions, easy to maintain and extend.
CI:
- Automatically trigger unit tests upon PR submission
- Execute corresponding tests based on the scope of changed files
- Only merge PR after tests pass
CD:
- Google Apps Script Web App (CD packaging interface) allows engineers, QA, and PMs to package the app via this website on computers or mobile devices
- GitHub Actions Self-hosted Runner uses your own machine for unlimited CI/CD usage
- Integrate with Firebase App Distribution API to directly obtain download links for packaged test versions
Automation:
- Automatically assign self upon PR submission
- Randomly assign a reviewer upon PR submission
- Tag PR with size labels
Demo Web App/Project
Sign in Edit description script.google.com
What is CI/CD?
Story — Development Process Without CI/CD
Before discussing what CI/CD is, let’s set aside the term “CI/CD” and recall how a startup development team without any established workflow would operate. The process can be roughly summarized in the following diagram:
- A product has a bug, Developer T creates a branch fix/bug-c from the main branch to fix it, and after completing the fix, merges it back to the main branch.
- Immediately after, Developer Z creates a branch feature/a from the main branch for Requirement A, but halfway through realizes something is off with the feature. Upon checking, they discover that the current functionality has been broken and the tests are also failing, so they notify Developer T to make the fix.
- Once all development is complete, Developer Z packages the version for QA testing using their computer, going back and forth to fix and package until everything is fine, and then merges the feature back to the main branch.
- As the sprint comes to an end, they need to package for release to users; Developer Z puts aside their current work to assist in packaging from the main branch for QA to conduct regression testing, similarly going back and forth to fix issues and repackage until it is completed and submitted for app review.
- After Apple/Google reviews, it is released to users.
Problems
From the above story, we can identify two major issues.
Question 1: There is no unified checking mechanism for current functional changes.
- Code that does not conform to coding style can still be merged
- Even if the build fails, I can still merge
- Changes can be merged even if basic unit tests and important checks have not passed
- My environment may work correctly, but others may not
- This affects others who are currently developing
Question 2: A significant amount of manpower is wasted on packaging work.
- Packaging requires engineer manpower, interrupting current development work
- The cost of switching between packaging and development is extremely high
- Waiting time for packaging prevents other development work from proceeding
- The time cost of engineers translates to money
- Manual operations may lead to errors
- QA needs to ask engineers to package (back and forth communication)
CI — Continuous Integration
In response to Question 1, “Continuous Integration” aims to ensure that all changes automatically undergo a unified environment’s build and test to ensure that modifications pass all test cases and comply with team standards before entering the production environment — “continuously and automatically ensuring correct code integration into the production environment.”
Additionally, it can include nightly builds and more automated testing phases to ensure stability.
CD — Continuous Delivery / Deployment
In response to Question 2, “Continuous Deployment” aims to ensure that after the CI phase has no issues, the results of changes are automatically packaged and deployed through tedious processes for internal testing (QA, Debug, Staging, Beta…) or external release (Production, Release…).
- Continuous Deployment: Fully automated deployment directly to the Production environment
- Continuous Delivery: Automatically deploys to Staging/Debug environments, requiring manual verification before deploying to the Production environment
In the context of app development, it leans more towards Continuous Delivery, as we want to ensure that the app is fully verified and confirmed to be error-free by humans before release, ensuring the accuracy of release timing and functionality.
Story — Building a Stable and Efficient Development Team through CI/CD
Looking back at our story, after implementing CI/CD:
- CI All adjustments must pass automated testing verification before entering the main branch, and nightly builds are added to enhance stability.
- CD Everyone uses CD for packaging, allowing Developer T and Developer Z to fully focus on business development, reducing manual communication and operational errors.
Team work efficiency and product stability 🚀🚀🚀🚀🚀
The Value of CI/CD
Combining the core concept of agile development “small steps, quick iterations,” CI/CD provides the foundation for stability and work efficiency during “frequent continuous feature iterations.”
Automated Unified Verification of Iteration Results
- Ensures all adjustments meet the correct expected results, do not affect other functionalities, and do not impact other team members
Automated Execution of Tedious Deployment Processes
- Allows team members to focus on primary business development, reducing manual operational errors
The Effectiveness of CI/CD
Reflecting on the 2021 talk at Pinkoi, “2021 Pinkoi Tech Career Talk — Secrets of High-Efficiency Engineering Teams,” the content essentially revolves around “automation, reducing reliance on individuals, and focusing on primary business.” Implementing CI/CD aligns perfectly with these three directions, allowing us to estimate effectiveness using the same methods.
Another point that needs to be highlighted is the cost of switching tasks:
When we continuously engage in work for a period, we enter a “flow” state, where our thoughts and productivity peak, enabling us to produce the most effective output. However, if interrupted, it takes time to return to that flow, which can be estimated at around 30 minutes.
In a scenario without CI/CD, it might look like this: Spending a lot of time only to discover that something was broken and then going back to communicate adjustments (CI), and asking engineers to help package the test version of the app (CD).
Estimating CI/CD Effectiveness
Team size of 6 people / Monthly calculation
Here, we use a monthly basis, assuming that without CI/CD processes, there would be 4 instances of accidentally breaking the main branch each month, leading to subsequent correction and communication costs, totaling about 720 minutes; plus the time spent packaging test versions, official versions, and the potential errors caused by manual operations, summing up to approximately 1,010 minutes; with an engineer’s monthly salary of 80,000, this results in about 13,000 in wasted costs each month.
CI/CD Implementation Costs
- Labor Costs: Based on the construction outlined in this series of articles, it is estimated that 1 person would need to invest 10 days = 4,800 minutes to complete it. (~= NT$36,384 )
- Equipment and Execution Costs: Using GitHub Actions self-hosted Runner only requires the procurement of 1–2 Mac Mini or directly using existing replaced MacBook Pro to serve as CI/CD Runner. Taking the example of purchasing a brand new Mac Mini for a 6-person team: 32G RAM M4 Mini (= NT$40,900 )
The total cost is approximately NT$80,000, allowing for completion of the setup, with benefits starting to be enjoyed after about six months.
Disclaimer: This is merely a proposed method for calculating benefits and may not be the most accurate; it is intended to provide a concept for extending, allowing management decision-makers to see the benefits of CI/CD and subsequently authorize the promotion of the entire workflow.
Tool Selection for CI/CD
Cloud Services Bitrise / XCode Cloud
- Bitrise: The earliest cloud service focused on providing App CI/CD. My first encounter with CI/CD was also through Bitrise, which offers a user-friendly step editing tool that allows for quick setup of App CI/CD processes. Cons: Initially, it was $99 for unlimited usage, and when the Apple M series processors were first released, they switched to usage-based billing ( a trap ). At that time, it was estimated that the team’s usage would cost at least $500 per month, so we migrated to GitHub Actions. However, I recently checked their website, and they now offer 1 App / 1 Concurrent / unlimited usage for $89 per month.
- XCode Cloud: 100 hours / 1 month / $50, with the advantage of being highly integrated with XCode and app development; however, it also has the drawback of not supporting Android and some custom steps may be difficult; but if it’s a small pure iOS app, I would reconsider using it directly.
I am really afraid of cloud services being a trap, and I hope to have control in my hands, so I am considering on-premises services.
On-Premises Services Jenkins / GitHub Actions / Gitlab CI/CD
- Gitlab CI/CD: Launched earlier than GitHub Actions and has more complete features, but since our project is hosted on GitHub, we won’t consider using Gitlab CI/CD; however, the functionalities are similar, and this series of articles will use GitHub Actions as an example.
- GitHub Actions Launched by GitHub in 2018, this CI/CD service is directly tied to GitHub projects and has continuously updated and improved features over the years, with many pre-packaged steps available in the ( Marketplace ) for direct use; it supports self-hosted runners that allow unlimited use of your own machines. ( essentially a hybrid cloud )
- Jenkins: An open-source free tool specifically for handling CI/CD, it is old but powerful; from application layer task design, permission management to underlying service dispatch execution, Jenkins covers it all; it also has Plugins that can be used directly, making it an essential tool for early DevOps CI/CD.
Jenkins vs. GitHub Actions
TL;DR
For an App Team without dedicated DevOps, having app developers set up and maintain a Jenkins environment from scratch is too high a barrier, and there are not many who know how to do it, which can also lead to network security issues; it is better to choose GitHub Actions directly, allowing app developers to focus solely on designing the CI/CD process, and with a quick review of the official documentation on how to write and start the Runner, they can quickly set up a free, stable, and secure CI/CD service.
The following comparison is based solely on setting up App CI/CD and does not apply to all technical scenarios.
Ease of Setup and Maintenance Jenkins »> GitHub Actions
Here’s a not-so-professional structural diagram to illustrate the differences between the two. As mentioned earlier, Jenkins encompasses all functionalities from top to bottom, making self-hosting much more complex. In contrast, GitHub Actions only requires you to write a YAML workflow on GitHub; your local machine just needs to register a GitHub self-hosted Runner (which can be done in 5 commands), and GitHub will automatically dispatch tasks to your local machine for execution. Other aspects, such as GitHub Actions/Runner version upgrades or task dispatch issues, are maintained by GitHub, so we don’t need to handle them.
Another point of complexity is that Jenkins operates independently of Git, requiring communication through APIs (e.g., GitHub API/WebHook), which adds to the setup complexity.
I previously conducted a survey among about 30 iOS developers I know, and only a handful (2) understood Jenkins, while over 10 were using GitHub Actions, as it simply involves writing YAML to complete CI/CD tasks.
Learning Difficulty Jenkins »> GitHub Actions
Similarly, you only need to refer to the official documentation to learn GitHub Actions, including the YAML commands and how to set up your own Runner locally.
Stability GitHub Actions > Jenkins
In this regard, I believe GitHub Actions slightly outperforms Jenkins.
Jenkins can potentially crash due to system upgrades or conflicts with installed plugins (though if it’s running smoothly and you don’t touch it, there shouldn’t be any issues).
GitHub Actions is limited by the GitHub service status (if GitHub is down, it will also be down), but this occurs infrequently, maintaining an average uptime of 99.9%. If issues do arise, there’s no need for us to handle them; we can just wait for a fix.
Security GitHub Actions > Jenkins
Considering that the GitHub Actions/Runner service is maintained and automatically updated by GitHub, it may be more secure than Jenkins, which requires manual updates.
Additionally, Jenkins requires opening API/WebHook ports for communication with GitHub, which is relatively risky. In contrast, GitHub integrates seamlessly with GitHub Actions, and the communication between GitHub Actions and self-hosted Runners follows an observer pattern, where the self-hosted Runner requests tasks from GitHub, meaning it doesn’t need to open external interfaces.
However, in a fully closed network environment, Jenkins may be more secure than GitHub Actions.
Permission Control Jenkins »> GitHub Actions
This point needs to be highlighted for comparison. Jenkins allows for separate account login permissions for control, while GitHub Actions is directly tied to the GitHub Repo, meaning only those with Repo permissions can use it.
*This is why the following articles will use GAS Web App to build a cross-team operational platform.
Usage Scope Jenkins »> GitHub Actions
In a complete DevOps team, it’s undeniable that Jenkins would still be the preferred choice, as it has been running the longest in other fields (such as Web, backend, Java, etc.), with the most plugins available and the best usability. It allows for a unified CI/CD service that all teams can use for easier management, or for complex CI/CD scenarios like automatically deploying the frontend after backend deployment.
*GitHub Actions later also supported cross-repo Actions/Runners.
Richness of Third-Party Plugins Jenkins > GitHub Actions
In terms of quantity, GitHub Actions surpasses Jenkins, but Jenkins offers deeper and more powerful CI/CD functionalities, while many GitHub Actions features are merely automation functions.
Depth of Functionality Jenkins »> GitHub Actions
This is not comparable; Jenkins has been around for nearly 20 years, while GitHub Actions still has many features to catch up on, such as permission management, secret management (currently limited to plain text; key files must be converted to plain text), and cache/artifact support (currently only for the cloud), among others.
In terms of expansion, GitHub Self-hosted Runner also supports Docker or k8s .
Customization Depth Jenkins »> GitHub Actions
Jenkins allows for complete control over customization, with greater permissions that can affect the entire system. GitHub Actions can only customize different steps at the application layer.
For example, the built-in artifacts in GitHub Actions do not support self-hosted, so you can only change the step to sh copy
to another directory, without the ability to customize artifacts implementation.
*App CI/CD scenarios do not require overly complex functionalities.
Usability GitHub Actions »> Jenkins
In terms of interface, GitHub Actions is a newer tool that is easier to use than Jenkins; in terms of script settings, Jenkins uses Pipeline Script stored on Jenkins, while GitHub Actions uses YAML files managed with Git, making it easier to set up than Jenkins.
Cost Risk Jenkins > GitHub Actions
Jenkins is fully open-source and free, giving you complete control. GitHub Actions is partially open-source, but task dispatch and execution are part of GitHub’s closed SaaS service. Currently, the policy is that GitHub Actions is completely free, but using GitHub Runner incurs costs (for private repos), while using self-hosted runners is free.
Purpose of Google Apps Script Web App and Why It Was Chosen
Another tool choice is Google Apps Script Web App. The reason for needing this is that the form functionality provided by GitHub Actions is too basic (the interface is too engineering-focused and only static), and the execution permissions are tied to the GitHub Repo, making it very cumbersome if we need to provide it for other functional partners.
As follows:
For CD packaging, we would like the operator to fill in some information, such as Release Notes.
Thus, we need a “interface” tool to provide for other partners or even for our engineers to use more conveniently.
Required Scenario:
Fill in the necessary information on this more user-friendly “interface,” connect to project management tools (e.g., Jira, Asana) to retrieve tasks or directly get the PR list from GitHub, select from a dropdown menu, and submit, then trigger GitHub Actions for packaging via the GitHub API.
Slack
When we first introduced CI/CD, we chose to integrate the Slack API to achieve effects similar to the following:
https://slack.com/intl/zh-tw/blog/productivity/workflow-builder-tools-automation-examples
- Partners can directly fill out forms in Slack, trigger CD packaging, and receive Slack notifications.
The operation was smooth and unified within daily office tools (SSOT), requiring no retraining. However, the underlying issue is that the development and maintenance costs are extremely high, one reason being that the Slack Outgoing-Webhook API has very high response requirements (it needs to respond within 3 seconds), effectively ruling out the use of FAAS services for simple integration (e.g., Cloud Functions, GAS, Lambda…).
Previously, one partner who was very interested in automation and backend developed a complete backend service using Kotlin+ktor, and then set up a server on GCP for Slack integration.
The development and maintenance costs are extremely high and difficult to hand over.
Google Apps Script — Web App
I previously shared “Using Google Apps Script Web App Forms to Integrate GitHub Action CI/CD Work”:
The advantages of using Google Apps Script — Web App are:
- Web-based
- Permission management with Google Workspace enterprise accounts, allowing access only to Google accounts within the organization
- Completely free
- Function as a Service without the need to set up and maintain servers
- Easier to maintain and hand over
- Usable on mobile devices
- AI Can Help! Whether it’s ChatGPT or other AI tools, they are very familiar with GAS and can directly assist us in creating packaging forms and integrating with the GitHub API.
- Can also integrate with Jira, Asana, and Slack notification APIs.
For the second promotion, I switched to using GAS Web App for partners, which received very positive feedback. The only difference from Slack is that you need to bookmark one more URL and open it to operate the packaging from the web form when needed.
Complete Tool Workflow for App CI/CD
Here’s the complete workflow. The next article will gradually introduce how to use and integrate each tool.
Tool Roles:
- GitHub Actions: CI/CD logic script code
- GitHub Actions — Self-hosted Runner: The actual execution location for CI/CD, using a self-hosted Runner to execute tasks without limits on usage, only incurring the cost of purchasing the machine.
- Google Apps Script Web App: Since packaging may not always be the responsibility of engineers, a platform is needed for cross-functional partners to use; GAS Web App can quickly create a web tool and share the URL for others to operate.
- Asana/Jira: Project management tools that can integrate with GAS Web App, allowing QA/PM to directly select tasks to be packaged.
- Slack: Responsible for receiving execution result notifications.
Scenarios:
- End-User (QA/PM/PD/Developer): Uses GAS Web App to submit the packaging form (fetching the corresponding branch for Jira or Asana task) -> GAS calls GitHub API -> triggers CD packaging in GitHub Actions <- GitHub self-hosted runner listens for the task and pulls it back to the machine for execution -> Upon completion, Slack notifies and updates the GAS Web App packaging status.
- End-User (Developer): Opens PR, pushes new commits to PR -> triggers CI testing process <- GitHub self-hosted runner listens for the task and pulls it back to the machine for execution -> Upon completion, comments on test results and updates checks.
Conclusion
This article primarily aims to give you a preliminary understanding of what CI/CD is and the benefits it brings. The next article will delve into the technical aspects, guiding you step-by-step to understand and implement GitHub Actions CI/CD to achieve the final results discussed earlier.
Series Articles:
- CI/CD Practical Guide (1): What is CI/CD? How to Build a Stable and Efficient Development Team Through CI/CD? Tool Selection?
- CI/CD Practical Guide (2): Comprehensive Use and Setup of GitHub Actions and Self-hosted Runners
- CI/CD Practical Guide (3): Implementing CI and CD Workflows for App Projects Using GitHub Actions
- CI/CD Practical Guide (4): Using Google Apps Script Web App to Integrate GitHub Actions to Build a Free and User-Friendly Packaging Tool Platform
Buy me a coffee
If you have any questions or suggestions, feel free to contact me.
```
This article was first published on Medium ➡️ Click Here
Automatically converted and synchronized using ZMediumToMarkdown and Medium-to-jekyll-starter.