Skip to content

Home

Jump to bottom
deniseAngilica edited this page Oct 17, 2023 · 2 revisions

Contributors Forks Stargazers Issues MIT License


Logo

BrainyBot

BrainyBot is a robot capable of solving the Candy Crush Saga and Ball Sort Puzzle video games using touching a phone screen. Based on the great TapsterBot design from Jason Huggins (https://github.com/tapsterbot/tapsterbot, https://tapster.io/)
Explore the docs »

Video Demo 1 (Italian captions) · Video Demo 2 . Report Bug · Request Feature

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Roadmap
  5. Contributing
  6. License
  7. Contact
  8. Acknowledgments

About The Project

BrainyBot was born as a framework to make a delta robot capable of playing match-3 games and ball-sorting puzzles by acting on mobile phones. As today, the framework can be used in a headless mode, where no physical robot is needed. The framework recognizes objects of different colors and shapes through a vision module, is capable of making strategic decisions based on declarative models of the game’s rules and of the game playing strategy, and can execute moves on physical or virtual devices. Our solution integrates multiple AI methods, including vision processing and answer set programming. Helpful and reusable infrastructure is provided: the vision task is facilitated, while actuation is facilitated either on robot motion control using a delta robot layout, or acting directly on the device at hand via ADB commands. The figure above shows the operating workflow of an instance of a BrainyBot.

(back to top)

How it works

The main hardware parts of a BrainyBot are a mobile device PH, a computer C and, optionally, the robotic effector E controlled using an Arduino board. Software components are placed respectively on PH or on C, which in turn controls all the parts of the system. A game G of choice runs on PH. BrainyBot cyclically processes information taken from PH’s display, then decides and executes arm moves on the touch display itself. More in detail, in each iteration, the Sense-Think-Act workflow is executed.

Built With

  • Python3
  • Javascript (Node.js)
  • Answer Set Programming

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Required hardware and software

This is a list of prerequisites for using the project:

  • PH: an Android mobile phone. This is the device where the game will be physically played on. In order to take screenshots from PH, there are two alternatives:

    • Install our ScreenshotServer application. You can find a pre-built apk for the ScreenshotServer in the ScreenshotServer folder. Just push it to PH, install it and start the server. Take note of the value of PH IP address.
    • Put PH in developer mode and connect it to the Linux host C. Screenshots will be taken via ADB commands.

  • C: a Linux host. The Linux host will host the Tapsterbot server commanding the robot, will collect screenshots from PH and run the AI module.

  • TP(Optional): an assembled Tapsterbot. This robotic arm can programmatically perform actions on a given touch screen (taps, swipes, etc.). If PH is in developer mode, one can switch to the headless mode that does not require TP.

  • npm: https://docs.npmjs.com/downloading-and-installing-node-js-and-npm

  • pipenv: https://pypi.org/project/pipenv/

  • ADB: https://developer.android.com/tools/adb

Installation and set up

You will need to set up some modules which are briefly described next:

Sensing: ScreenshotServer or ADB

The ScreenshotServer is an Android application which opens an HTTP server on PH. You can manually HTTP GET screenshots on demand from the Screenshotserver this way:

curl http://<PHone-IP>:5432/?name=requestimage --output screen.png

You can find a pre-built apk for the ScreenshotServer in the ScreenshotServer folder. Just push it to PH, install it and start the server. Take note of the value of PHone-IP.

Alternatively, you can put PH in developer mode and connect it to the Linux host. Screenshots will be obtained via the shell command:

adb exec-out screencap -p > screenshot.png

The two ways of getting screenshots can be selected by changing the USE_ADB variable in the Brain/AI/src/constants.py file (default: USE_ADB=True).

Acting

The acting step can be implemented in two distinct ways:

  • using a Tapsterbot (details in the following);
  • using ADB to act directly on the devices without external actuators. In this case, connect PH to C via USB and start up the server as npm start -- --hardware=headless.

Tapsterbot

The robotic arm needs to be 3D-printed and assembled as described in several tutorials online like: https://www.instructables.com/Tapsterbot-20-Servo-Arm-Assembly/. The design we opted for, is Tapster-2 whose 3D models can be found in the tappy-client/hardware/tapster-2 folder. The software on the embedded Arduino board must be the Standard Firmata script: from the Arduino IDE find and upload the "Firmata" script:

File -> Open -> Examples > Firmata > StandardFirmata

Calibration and testing of the robot can be done by following the installation guide of the Tappy server: https://github.com/DeMaCS-UNICAL/tappy-original

Tappy server

You find a fork of the tappy project under the tappy-server submodule. The Tappy server is expected to run on some system on port 8000. The config.js file under the tappy-server folder allows to customize the listening port and other physical parameters of the robot. Recall you can use nvm for managing the required node.js version, (currently it is needed node 16.*). If your current user has no access rights to serial ports, recall to use sudo npm start when starting the tappy server instead of a plain npm start, and ensure the correct version of node is available also under sudo privileges (the default node might differ when sudo-ing).

Python client

This python client for controlling tappy server from a client is located under the tappy-client/clients/python folder. You will possibly need to tweak IP and listening port of the Tappy server and other stuff in the config3.py file. Example usage:

python3 client3.py --url http://127.0.0.1 --light 'swipe 325 821 540 821'

Detailed documentation for the python client can be found in the README of the https://github.com/DeMaCS-UNICAL/tapsterbot-original

Basic installation instructions

  1. Clone the repo

    git clone https://github.com/DeMaCS-UNICAL/BrainyBot.git

  2. Don't forget to pull submodules

    git submodule update --init --recursive

  3. If needed, keep submodules up to date

    git pull --recurse-submodules

  4. Set up a pipenv environment after cd-ing in BrainyBot/Brain. The command will download all required libraries in a new virtual environment:

    cd BrainyBot/Brain
pipenv install

    We assume you have Python 3.10 on computer C (Python 3.9 should work too by modifying the internal Pipfile).

  5. Move to the BrainyBot/tappy-server folder and install the server contained in that folder (tested with node.js lts/gallium (16.13.2)):

    npm install

    For more information check the submodule repository: https://github.com/DeMaCS-UNICAL/tappy-original/

(back to top)

Using already implemented games

These are the steps to follow to use and make the project work properly:

  1. Open two terminal windows, T1 and T2.

  2. Ensure the robot is connected via USB to C, and on T1 navigate to the tappy-server folder and issue the following command:

     sudo npm start [-- --hardware=headless]

    The --hardware option allows to switch to the headless implementation: make sure to enable developer options on PH in this case.
    Write down the IP and port address of the Linux host server C. If you opted for the physical robot, you can also perform PH display calibration at this point. Depending on the size of your PH, it could be necessary to adjust the min z value in config.js. Remember to refresh the calibration page in your browser whenever restarting the server. For more information check the official repository for the Tappy Server Module: https://github.com/DeMaCS-UNICAL/tappy-original/

  3. If you don't want to/can't connect PH to C via ADB, run the ScreenshotServer application on PH, and tap its start button. Write down the IP address of PH.

  4. Change the IP addresses of the ScreenshotServer and the tappy-server in the constant.py file contained in the folder Brain/AI/src

  5. Open the Candy Crush Saga or Ball Sort Puzzle game on the PH and start a game on a level of choice. Unlike Candy Crush Saga, there are different implementations of the Ball Sort Puzzle game. BrainyBot is tested on Ball Sort - Color Sorting Game

  6. In T2, navigate to the Brain folder. You can run the AI either by first activating a pipenv shell

cd Brain
export PYTHONPATH=.
pipenv shell

and then issuing

python3 AI/src/main.py -g game_name

Alternatively, you can just cd to the Brain folder and issue

pipenv run play -g game_name

-g/--game can be followed by:

  • ball_sort if you want to run the AI module related to the ball sort puzzle game
  • candy_crush if you want to run the AI module related to the candy crush game

For example:

python3 main.py -g ball_sort

This will take a screenshot, then perform computer vision and decision making on it. The results of this process, i.e. the coordinates (x, y) representing the moves to be performed on the PH, are automatically communicated by a python client to the tappy-server, which in turn will command the robotic effector E who will execute the moves on PH.

The ball_sort AI will execute a whole level, while the candy_crush AI will decide and execute one move at a time.

For more examples, please refer to the Documentation

(back to top)

Implementing you own game AI workflow

In order to facilitate tests and debugging you can use other options for the main.py script:

  • -dV/--debugVision enable debugging on the vision phase and can be combined with
    • -t/--test image_prefix will perform accuracy tests on the vision phase, it will look for images located in the default screenshot folder and whose name starts with image_prefix;
    • -s/--screenshot image_name will perform vision recognition on the image named image_name and located in the default screenshot folder. Concerning the vision phase, you can easly recognize balls or template images by mean of our objectsFinder module. Then, you can abstract the information obtained using the abstraction module to identify stacks and grids of objects in the level. Currently, grids abstraction supports only vertical and horizontal connections between elements.

Vision: objectsFinder module

ObjectsFinder(screenshot_path,color=None,debug=False,threshold=0.8)

In order to create a new objectsFinder's instance you need the screenshot path. Optionally, you can choose a color conversion to apply when loading the image (like cv2.COLOR_BGR2RBG) and a threshold that will be used in the template matching function (see below).

find_circles(balls_min_distance, balls_min_radius, balls_max_radius)

The ObjectsFinder allows to identify circles (balls) in the screenshot: you need to specify the minimum distance between balls and the minimum and max radius of the circles to be searched.

find_one_among(elements_to_find:{}, request_regmax=True)
find_one_among_gray_scale(elements_to_find:{}, request_regmax:True)
find_all(elements_to_find:{}, request_regmax=True)
find_all_gray_scale(elements_to_find:{}, request_regmax:True)

Concerning the template images search, you can search either for a specific template, or one among a number of templates or for all the templates provided possibly on the gray-scaled image. All search functions of the ObjectsFinder but the "find_all" return a list of identified objects. The "find_all" functions return a dictionary where for each template matched there is a list of objects found. At this point, there could still be some false positive matches that can be cleared by the abstraction module.

Abstraction: abstraction module

Stack(elements: list,tolerance=50,max_distance=150, min_number_elements=4)

The objects identified by the ObjectsFinder can be abstracted either in stacks or grids. The Stack function needs a list of elements to be separeted into stacks. Some configuration property can be specified: the maximum (vertical) distance between elements of the same stack, the minimum number of elements contained in the same stack and the horizontal tolerance for element in the same stack. This function returns a list of stack where false positive matches have been removed.

ToGraph(elements:dict, distance:())

The ToGraph function needs a dictionary of elements where the keys are the labels of the nodes in the graph and the values corresponds to the coordinates of the node itself. You can specify which are the distances on both vertical and horizontal axis for each element in the grid. This function returns an ObjectGraph.

Thinking

For each object type in the game that you need to use in your ASP program, you need to create a class that extends the Predicate class with a global field predicate_name whose value is the name of the predicate that will be used in the ASP encoding. The super class needs to be initialized with the names of the instance-fields to be serialized: for those instance-fields the class needs to expose getters and setters. You can look at the AI.src.ball_sort.dlvsolution.helpers.Color class as an example. For further details, please check the EmbASP documentation.

Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

Also, if you want to add your favorite game to the project, the steps to follow are these:

  1. Fork the Project
  2. Create your Game Branch (git checkout -b yourGame)
  3. Commit your Changes (git commit -m 'Add some AmazingFeature')
  4. Push to the Branch (git push origin yourGame)
  5. Open a Pull Request

The code to modify for adding a new game is roughly the following:

  1. Create your own package renaming it with the name of your game and add it in the AI/src folder. In this main package add some subpackages trying to keep the current structure of the project:

    • A package renamed detect for analyzing game screenshots using your favorite computer vision algorithms
    • A package abstract to translate the information extrapolated from the analysis of the screenshot into python objects representing the entities of the game
    • A package renamed dlvsolution for translating python objects into logical predicates and for finding a game solution through ASP and the EmbASP framework
    • A package renamed resources in which to insert the sprites to be used for the analysis of the screenshot (if necessary) and the ASP encoding for the resolution of the game
    • A constant.py script where you specify the path to the various elements you need in the code
    • A helper.py script in which you call the methods of your objects in the right order in order to find the solution to the game. Finally in this script the identified moves to be performed to solve the game are communicated to the tappy-server

  2. In the AI/src/main.py script add an argument to be used at runtime to choose the game you have added and the code to conveniently call the main method present in your AI/src/yourGame/helper.py script. For example:

    python3 main.py -g yourGame

(back to top)

License

Distributed under the APACHE-2.0 License. See LICENSE.txt for more information.

(back to top)

The team

  • General integration of modules, Hardware, Calibration, Actuation modules, AI Tweaking, whatever other kick is needed to keep everything alive and working: Giovambattista Ianni a.k.a. Agelin Bee (@iannigb)
  • General Maintainance & Bug fixing : Denise Angilica (@deniseAngilica) and Francesco Pacenza (@fpacenza)
  • Screenshotserver, Candy Crush Vision & Candy Crush AI : Mario Avolio (@MarioAvolio) @linkedin - [email protected]
  • Ball Sort Vision & Ball Sort AI : Giovanni Beraldi - @linkedin - [email protected]

(back to top)

Clone this wiki locally