Skip to content

Overview of automated testing

Introduction

How to let 0-level students master the automated test?

This is what the Airtest team has been working on for five years: continuously improving the tool chain to allow users to perform automate tests with a very low threshold.

Whether you are a newcomer to the workplace or a test engineer just starting to learn automation, I believe this tutorial can help you clarify the context of automated testing and master the learning path of automated testing. We also hope to help you get started with automated testing with hands-on tutorials from Airtest.

After learning, we hope you can master the following:

  • Learn about mainstream frameworks and tools for automation
  • Getting started with Airtest and Poco
  • Perform automated testing on Android devices
  • Perform automated testing on IOS devices
  • Automated testing of WeChat applets and games
  • How to write strong compatibility automation scripts
  • ......

It should be noted that testing and automation in this tutorial refer to software testing and software automation, and testing and automation in hardware and industry are not included in the scope of this tutorial.

Are you ready? Let's start together.

Let's talk about testing

Software testing can be understood as a process for identifying software compliance with requirements and defects. According to different dimensions, software testing can be subdivided into different test types.

If divided according to the test method, it can be divided into:
  • Black box testing: The most basic functional test, it does not care about the internal code implementation, but only verifies the correctness of the input and output.
  • White box testing: Based on logic-driven or code-based testing, it will open the internal implementation of the code to study the correctness of the interface or specific implementation in the source code.
  • Gray box test: A test between the two tests above.
According to the test objectives, it can be divided into:
  • Functional test: test each function of the product and module
  • Performance test: test the performance indicators of the system
  • Stress test: test the load capacity of the software or system and discover hidden dangers
  • Compatibility test: test the compatibility between the product and software or hardware, such as software compatibility on various Android models.
  • Security testing: Use different methods to find software security issues, such as information leakage, illegal use, malicious destruction, and so on.
  • Other special tests: such as weak network test, power consumption test, fluency test, etc.
If divided according to the software development stages, each stage can do:
  • Unit tests: Perform white-box tests on independent modules in a program. The purpose is to verify the correctness of the basic components of the software
  • Integration test: Perform combination tests on unit modules to verify that the interfaces between unit modules are correct
  • System test: conduct a complete test of the entire system to verify the correctness and compliance of the entire system
  • Regression test: When the software changes, verify the functional modules that may be affected by the change
  • Acceptance test: the final stage of testing, ensuring software quality before software release or launch
Other commonly used testing concepts:

  • Smoke test: The smoke test is a simple test of the most basic functions of the software. The smoke test can be used to determine whether the software is measurable at low cost.

    The smoke test is derived from the hardware test. When the circuit board is ready, it will first be powered up. If there is no smoke, the next test will be started. Otherwise, the product does not meet the basic quality requirements and needs to be remade.

  • Exploratory testing: Exploratory depends more on the personal experience or expertise of the tester, and on the subjective initiative of the tester.

    The importance of exploratory testing can be referred to the field of game testing. Millions of players will play games in unexpected ways in various unexpected environments.Therefore, game testers must not only master systematic test methodology and advanced test tools, but also have rich game experience and exploring test thinking.

Having said so much, it is a simple and primitive way to complete all the tests by hand. But because there will be a lot of repetitive work during the test, such as regression testing (that needs to be executed after each change), such as Compatibility testing (that needs to perform the same use case on hundreds of mobile phones), so in the process of work, we hope to be able to continue to use code to help us complete the test, that is, automated testing.

Introduction to mainstream automated testing frameworks

Automation for software test can generally be divided into 3 layers * Unit testing at the code level * Integration testing at the interface layer * Testing at the UI layer

Code-level automation

The automation of the code layer generally refers to unit testing of the code, and commonly used unit testing frameworks such as Java's Junit, Python's PyUnit, and so on. Since this part is not the focus of this tutorial, it will not be expanded here in detail, and students who need it can learn by yourselves.

Interface layer automation

The automatic test at the interface layer is mainly to test the interface between the system and the component. The main goal is to verify the exchange of data and business processes. The interface test can test functions, performance, pressure and security.Because the interface is much more stable than the code unit, the maintenance cost of automation script is lower, the benefit is greater, and it has a good cost performance. The commonly used testing tools are the following:

Jmeter: Java-based interface testing, stress testing, and performance testing tools developed by the Apache organization. At first it was designed for web testing, but later it gradually expanded to other areas, which can be used to test static or dynamic resources.

LoadRunner: A performance testing and stress testing tool provided by HP, which can test system performance by simulating thousands of users to implement concurrent operations. And it has detailed test results analysis, which is a good choice for performance testing and stress testing.

Robot Framework: An open source automated testing framework with good scalability. The framework is written in python and also provides cross-platform support.

Postman: Simple, convenient and powerful interface debugging tool, API debugging is preferred.

UI layer automation

The automation testing framework based on the UI layer is much more complicated. In terms of platform types, there are Windows, Linux, Android, Ios, Web, and the latest applets. The following will briefly walk you through the principles, architecture, and cross-platform capabilities of the mainstream UI layer automation framework.

Appium

Appium is an open source automated testing tool that supports IOS, Android, Windows and Mac applications.

Cross-platform

Appium can run on OSX, Windows, and Linux desktops.

Cross language

Appium uses the C/S design pattern and extends the WebDriver protocol, so Client is implemented in various languages such as Python, Java, Js/Nodejs, Ruby, OC, C# and so on.

Principle introduction

At the core of Appium is a web server that adheres to the REST design style. It will be used to accept client connections and instructions. Due to the unified interface design, the client can be implemented in multiple languages. So testers can implement test cases in their favorite language.

After receiving the test instruction, the server sends it to the device. At the device layer, the native test framework provided by the device vendor is used, such as XCUITest Driver and UIAutomation Driver of IOS, UIAutomator and UIAutomator2 of Android, and so on.

appium.png

Appium's official website： http://appium.io/

Homepage of Appium Github： https://github.com/appium/appium

Selenium

Selenium is an open source automation testing tool for web application. It can run directly in multiple browser platforms, just like the actual operation of the user.

Cross-platform

同样，Selenium也可以在OSX，Windows以及Linux桌面上运行。

Browser support

Firefox,Chrome,IE,Edge,Opera,Safari

Principle introduction

selenium.png

Selenium's official website：http://seleniumhq.org/

Homepage of Selenium Github：https://github.com/SeleniumHQ/selenium

Introduction for Airtest Project

Airtest Project is an open source automated testing framework developed by NetEase . Compared with other automated testing frameworks, the Airtest Project has the following two advantages:

  • Dramatically reduce the cost of writing and maintaining automated scripts

    The Airtest Project hopes to complete the recording of the script in a WYSIWYG manner. Even if the testers do not know the script, they can automatically record the script through normal click and drag operations, which can greatly reduce the automatic maintenance costs of the enterprise and the project.

  • Solve the pain points of game testing

    Airtest Project hopes to become a truly cross-engine and cross-platform automated testing platform by supporting different game engines.

Architecture diagram

airtest.png

As you can see, the main testing frameworks at the bottom are Airtest and Poco. The differences between the two are:

  • Airtest: It is a Python-based, cross-platform UI automated testing framework based on the principle of image recognition and suitable for games and apps.
  • Poco: An automated testing framework based on UI control search. Its core advantage is that it supports games in addition to Android and IOS. It also supports WeChat applets, WeChat mini games, and H5 applications.

The entire framework is equipped with a very practical IDE. With AirtestIDE, you can easily complete the recording of scripts, the execution of test tasks, and the generation of final test reports.

For the complete process, please watch the demo video:

In the following chapters, we will explain the automated testing technology based on Airtest and Poco on Android, IOS and other platforms.

Airtest Project's official website: http://airtest.netease.com/

Homepage of Airtest Github：https://github.com/AirtestProject/Airtest