From 19270f0a55d52cd18f745a85c38825b68f350411 Mon Sep 17 00:00:00 2001 From: "Win Cowger, PhD" Date: Mon, 28 Nov 2022 09:41:44 +0100 Subject: [PATCH] Manuscript (#69) * my workflow * example templates for manuscript. * manuscript info * update citations * cleanup text * cite * update readme * contribute * add website badge * update web link * Update README.md * Update README.md * add video link * auto update * utility files device specific * move the pdf generator to workflows * test error in paper.bib * test citation style * add citations * add a few more citations * try without space * add all * no spaces allowed in citation names * add demo images * Update paper.md * give images some space * image captions * add mention of wade * update authors and acknowledgements * Updated with Dan's Recs * Mary Comments https://github.com/code4sac/trash-ai/pull/69#issuecomment-1257554556 * SteveO comments https://github.com/code4sac/trash-ai/pull/69#pullrequestreview-1119350215 * Walter's comments https://github.com/code4sac/trash-ai/pull/69#pullrequestreview-1119351559 * Create config.yml * Create bug.yml * Create feature.yml * hyperlink * Update bug.yml * Update feature.yml * Added in Kristiina and Kris's comments * Update README.md * https://github.com/code4sac/trash-ai/pull/69#discussion_r1006489065 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1006514485 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1006517045 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1006519572 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1006536610 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1007668618 * add kris's comments * add WSL2 link * Update paper.md * add acknowledgements * https://github.com/code4sac/trash-ai/pull/69#discussion_r1033145379 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1033146187 * https://github.com/code4sac/trash-ai/pull/69#discussion_r1024726101 --- .github/ISSUE_TEMPLATE/bug.yml | 91 ++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 1 + .github/ISSUE_TEMPLATE/feature.yml | 48 +++++++ .github/workflows/draft-pdf.yml | 23 ++++ .gitignore | 9 ++ README.md | 35 ++++-- docs/localdev.md | 16 ++- paper.bib | 196 +++++++++++++++++++++++++++++ paper.md | 113 +++++++++++++++++ 9 files changed, 515 insertions(+), 17 deletions(-) create mode 100644 .github/ISSUE_TEMPLATE/bug.yml create mode 100644 .github/ISSUE_TEMPLATE/config.yml create mode 100644 .github/ISSUE_TEMPLATE/feature.yml create mode 100644 .github/workflows/draft-pdf.yml create mode 100644 paper.bib create mode 100644 paper.md diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..e7eb59c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,91 @@ +name: Bug report +description: Report a problem you encountered +title: "[Bug]: " +labels: ["bug"] +body: + - type: markdown + attributes: + value: Thanks for taking the time to complete this bug report! + - type: checkboxes + id: terms + attributes: + label: Guidelines + description: By submitting this issue, you agree to follow our [Contributing Guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). + options: + - label: I agree to follow this project's Contributing Guidelines. + required: true + - type: input + id: project + attributes: + label: Project Version + description: Which project versions are affected? + placeholder: "1.0, 2.2, 3.5.1" + validations: + required: false + - type: input + id: platform + attributes: + label: Platform and OS Version + description: Which platforms or operating systems are affected? + placeholder: "macOS 10.15.1, Ubuntu 20.04" + validations: + required: false + - type: input + id: existing-issues + attributes: + label: Existing Issues + description: Are there any similar existing issues? + placeholder: "#42" + validations: + required: false + - type: textarea + id: what-happened + attributes: + label: What happened? + description: Clearly and concisely describe the bug. + placeholder: Tell us what happened. + validations: + required: true + - type: textarea + id: repro-steps + attributes: + label: Steps to reproduce + value: | + 1. + 2. + 3. + ... + validations: + required: true + - type: textarea + id: expected-behavior + attributes: + label: Expected behavior + description: What should have happened? + placeholder: What did you expect to happen? + validations: + required: true + - type: textarea + id: attachments + attributes: + label: Attachments + description: Please include code snippets, stack traces, or compiler errors. + placeholder: Paste code snippets, stack traces, and compiler errors here + validations: + required: false + - type: textarea + id: screenshots + attributes: + label: Screenshots or Videos + description: Add screenshots, gifs, or videos to help explain your problem. + placeholder: Upload screenshots, gifs, and videos here + validations: + required: false + - type: textarea + id: additional + attributes: + label: Additional Information + description: Add any other useful information about the problem here. + placeholder: Is there any additional helpful information you can share? + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..0086358 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1 @@ +blank_issues_enabled: true diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml new file mode 100644 index 0000000..c2b1b36 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -0,0 +1,48 @@ +name: Feature request +description: Suggest an idea for this project +title: "[Feature]: " +labels: ["feature request", "enhancement", "feature"] +body: + - type: markdown + attributes: + value: Thanks for taking the time to complete this form! + - type: checkboxes + id: terms + attributes: + label: Guidelines + description: By submitting this issue, you agree to follow our [Contributing Guidelines](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). + options: + - label: I agree to follow this project's Contributing Guidelines. + required: true + - type: textarea + id: description + attributes: + label: Description + description: Clearly and concisely describe what you would like to change, add, or implement. + placeholder: Tell us your idea. + validations: + required: true + - type: textarea + id: problem + attributes: + label: Problem + description: Is your feature request related to a problem? + placeholder: What problem will this solve? + validations: + required: true + - type: textarea + id: solution + attributes: + label: Proposed Solution + description: How should this be solved? + placeholder: How do you think this should be implemented? + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Alternatives Considered + description: Are there other possible approaches? + placeholder: Can you think of any other options? + validations: + required: true diff --git a/.github/workflows/draft-pdf.yml b/.github/workflows/draft-pdf.yml new file mode 100644 index 0000000..61ae864 --- /dev/null +++ b/.github/workflows/draft-pdf.yml @@ -0,0 +1,23 @@ +on: [push] + +jobs: + paper: + runs-on: ubuntu-latest + name: Paper Draft + steps: + - name: Checkout + uses: actions/checkout@v2 + - name: Build draft PDF + uses: openjournals/openjournals-draft-action@master + with: + journal: joss + # This should be the path to the paper within your repo. + paper-path: paper.md + - name: Upload + uses: actions/upload-artifact@v1 + with: + name: paper + # This is the output path where Pandoc will write the compiled + # PDF. Note, this should be the same directory as the input + # paper.md + path: paper.pdf \ No newline at end of file diff --git a/.gitignore b/.gitignore index 82c26f0..678e1f1 100644 --- a/.gitignore +++ b/.gitignore @@ -11,9 +11,18 @@ data # other .DS_Store +#Google drive +*.ini + +# R Studio +*.Rproj + **/*.pyc .python-version **/*.zip pyrightconfig.json archive/ tmp/ +.Rproj.user +delete.bat +.Rhistory diff --git a/README.md b/README.md index e4857b1..a64499f 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,38 @@ -## AI for Litter Detection (Web Application) +# Trash AI: Web application for serverless image classification of trash +[![Website](https://img.shields.io/badge/Web-TrashAI.org-blue)](https://www.trashai.org) + ### Project Information -- Sponsor: Win Cowger, UC Riverside - Trash Data Projects -- Meeting Times: Wednesdays at 6:30pm (Weekly Brigade Meetings) +- Sponsor: Win Cowger, [Moore Institute for Plastic Pollution Research](https://mooreplasticresearch.org/) +- Meeting Times: Wednesdays at 6:30pm PT [Weekly Brigade Meetings](https://www.meetup.com/code4sac/) ### Project Summary -- Description: Develop web application where users can upload photos of street litter which will be labeled using computer vision to detect and categorize litter type. -- Benefit: Enhance abilities of researchers to quickly label photos and categorize types. +Trash AI is a web application where users can upload photos of litter, which will be labeled using computer vision to detect and categorize litter in the image by type. Early inspiration from [WADE AI](https://github.com/letsdoitworld/wade-ai) streamlined this development. Trash AI will enhance the abilities of researchers to quickly label trash in photos. + +#### Demo +[![image](https://user-images.githubusercontent.com/26821843/188515526-33e1196b-6830-4187-8fe4-e68b2bd4019e.png)](https://youtu.be/HHrjUpQynUM) + +## Deployment + +You can simply go to www.trashai.org to start using the tool or deploy it yourself. Current self-deployment options are local deployment with docker to remote on Amazon Web Services (AWS). ### [Local Development](./docs/localdev.md) -Run the environment live with localstack, and docker +- Run the environment live with localstack and docker. + +### [AWS Deployment](./docs/git-aws-account-setup.md) -### [AWS Deployments](./docs/git-aws-account-setup.md) +- Instructions on bringing up a new AWS deployment. -Instructions on bringing up a new deployment +#### [Continuous Integration and Continuous Delivery (CI/CD) - Github Actions](./docs/github-actions.md) -### [CI/CD - Github Actions](./docs/github-actions.md) +- Mostly CD at this point. -Mostly CD at this point. +#### [Github Actions AWS Deployment Role](./docs/github-actions-deployment-role.md) -### [Github Actions AWS Deployment Role](./docs/github-actions-deployment-role.md) +- Runs the complex stuff so you don't have to. -Runs the complex stuff so you don't have to +## Contribute +We welcome contributions of all kinds. To get started, open an [issue](https://github.com/code4sac/trash-ai/issues) or [pull request](https://github.com/code4sac/trash-ai/pulls). Here are some ideas on [How to Contribute](https://opensource.guide/how-to-contribute/). Please adhere to this project's [Code of Conduct](https://www.contributor-covenant.org/version/2/1/code_of_conduct/). diff --git a/docs/localdev.md b/docs/localdev.md index df1a184..c070449 100644 --- a/docs/localdev.md +++ b/docs/localdev.md @@ -2,7 +2,7 @@ When doing local development, you can run this stack in the background and edit files in the `/backend` and `/frontend` directories and the environment -with automatically update. +with automatic update. The listening port for the web frontend defaults to `http://localhost:5150`, The backend is exposed via `http://localhost:4000` by default. @@ -13,17 +13,23 @@ These values can be adjusted by editing the localdev env file [.env](../localdev ## _*IMPORTANT*_ -Pushing / merging PR's to any branches with a prefix of `aws/` will -trigger deployment actions, when developing locally, create a new branch -and submit a pull request to `aws/trashai-staging` +It's suggested you work in branch `local` by creating your own local branch when developing +Pushing / merging PR's to any branches with a prefix of `aws/` will trigger deployment actions +For full functionality you will want to get a Google Maps API key and name it VITE_GOOGLE_MAPS_API_KEY, but it is not required +======= + --- # Set up ## Operating System Requirements - Linux -- MacOS +- MacOS (testing) - Windows using [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) + - WSL Integration on Ubuntu must be enabled in Docker options -> Resources -> WSL Integration + - Repo must be inside Ubuntu (i.e. ~/code/trash-ai) + - Make local must be run from a WSL (Ubuntu) terminal + - Take note of docker-compose installation (installing through apt may not install the correct version) ## 1. Install Required Software - docker desktop (or configure docker engine and docker-compose another way) diff --git a/paper.bib b/paper.bib new file mode 100644 index 0000000..562d126 --- /dev/null +++ b/paper.bib @@ -0,0 +1,196 @@ + +@software{Jocher:2020, + author = {Glenn Jocher and + Alex Stoken and + Jirka Borovec and + NanoCode012 and + ChristopherSTAN and + Liu Changyu and + Laughing and + tkianai and + Adam Hogan and + lorenzomammana and + yxNONG and + AlexWang1900 and + Laurentiu Diaconu and + Marc and + wanghaoyang0106 and + ml5ah and + Doug and + Francisco Ingham and + Frederik and + Guilhen and + Hatovix and + Jake Poznanski and + Jiacong Fang and + Lijun Yu 于力军 and + changyu98 and + Mingyu Wang and + Naman Gupta and + Osama Akhtar and + PetrDvoracek and + Prashant Rai}, + title = {{ultralytics/yolov5: v3.1 - Bug Fixes and + Performance Improvements}}, + month = oct, + year = 2020, + publisher = {Zenodo}, + version = {v3.1}, + doi = {10.5281/zenodo.4154370}, + url = {https://doi.org/10.5281/zenodo.4154370} +} + +@TECHREPORT{Moore:2020, + title = "California Trash Monitoring Methods and Assessments Playbook", + author = "Moore, Shelly and Hale, Tony and Weisberg, Stephen B and + Flores, Lorenzo and Kauhanen, Pete", + institution = "San Francisco Estuary Institute", + year = 2020 +} + +@ARTICLE{Hapich:2022, + title = "Trash Taxonomy Tool: harmonizing classification systems used to + describe trash in environments", + author = "Hapich, Hannah and Cowger, Win and Gray, Andrew and Tangri, Neil + and Hale, Tony and Magdy, Amr and Vermilye, Antoinette and Yu, + Walter and Ayres, Dick and Moore, Charles and Vermilye, John and + Singh, Samiksha and Haiman, Aaron N K and Youngblood, Kathryn and + Kang, Yunfan and McCauley, Margaret and Lok, Trevor and Moore, + Shelly and Baggs, Eric and Lippiatt, Sherry and Kohler, Peter and + Conley, Gary and Taing, Janna and Mock, Jeremiah", + abstract = "Despite global efforts to monitor, mitigate against, and prevent + trash (mismanaged solid waste) pollution, no harmonized trash + typology system has been widely adopted worldwide. This impedes + the merging of datasets and comparative analyses. We addressed + this problem by 1) assessing the state of trash typology and + comparability, 2) developing a standardized and harmonized + framework of relational tables and tools, and 3) informing + practitioners about challenges and potential solutions. We + analyzed 68 trash survey lists to assess similarities and + differences in classification. We created comprehensive + harmonized hierarchical tables and alias tables for item and + material classes. On average, the 68 survey lists had 20.8\% of + item classes in common and 29.9\% of material classes in common. + Multiple correspondence analysis showed that the 68 surveys were + not significantly different regarding organization type, + ecosystem focus, or substrate focus. We built the Trash Taxonomy + Tool (TTT) web-based application with query features and open + access at openanalysis.org/trashtaxonomy. The TTT can be applied + to improve, create, and compare trash surveys, and provides + practitioners with tools to integrate datasets and maximize + comparability. The use of TTT will ultimately facilitate + improvements in assessing trends across space and time, + identifying targets for mitigation, evaluating the effectiveness + of prevention measures, informing policymaking, and holding + producers responsible.", + journal = "Microplastics and Nanoplastics", + volume = 2, + number = 1, + pages = "15", + month = jun, + year = 2022 +} + +@misc{Waterboards:2018, + author = {Waterboards}, + title = {Trash Tracker}, + year = {2018}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/CAWaterBoardDataCenter/Trash-Tracker} +} + +@article{vanLieshout:2020, + author = {van Lieshout, Colin and van Oeveren, Kees and van Emmerik, Tim and Postma, Eric}, + title = {Automated River Plastic Monitoring Using Deep Learning and Cameras}, + journal = {Earth and Space Science}, + volume = {7}, + number = {8}, + pages = {e2019EA000960}, + keywords = {plastic pollution, object detection, automated monitoring, deep learning, artificial intelligence, river plastic}, + doi = {https://doi.org/10.1029/2019EA000960}, + url = {https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019EA000960}, + eprint = {https://agupubs.onlinelibrary.wiley.com/doi/pdf/10.1029/2019EA000960}, + note = {e2019EA000960 10.1029/2019EA000960}, + abstract = {Abstract Quantifying plastic pollution on surface water is essential to understand and mitigate the impact of plastic pollution to the environment. Current monitoring methods such as visual counting are labor intensive. This limits the feasibility of scaling to long-term monitoring at multiple locations. We present an automated method for monitoring plastic pollution that overcomes this limitation. Floating macroplastics are detected from images of the water surface using deep learning. We perform an experimental evaluation of our method using images from bridge-mounted cameras at five different river locations across Jakarta, Indonesia. The four main results of the experimental evaluation are as follows. First, we realize a method that obtains a reliable estimate of plastic density (68.7\% precision). Our monitoring method successfully distinguishes plastics from environmental elements, such as water surface reflection and organic waste. Second, when trained on one location, the method generalizes well to new locations with relatively similar conditions without retraining (≈50\% average precision). Third, generalization to new locations with considerably different conditions can be boosted by retraining on only 50 objects of the new location (improving precision from ≈20\% to ≈42\%). Fourth, our method matches visual counting methods and detects ≈35\% more plastics, even more so during periods of plastic transport rates of above 10 items per meter per minute. Taken together, these results demonstrate that our method is a promising way of monitoring plastic pollution. By extending the variety of the data set the monitoring method can be readily applied at a larger scale.}, + year = {2020} +} + +@ARTICLE{Lynch:2018, + title = "{OpenLitterMap.com} -- Open Data on Plastic Pollution with + Blockchain Rewards (Littercoin)", + author = "Lynch, Se{\'a}n", + abstract = "OpenLitterMap rewards users with Littercoin for producing open + data on litter. Open data on the geospatial characteristics of + litter provide means of invoking and evaluating responses to + plastic pollution. OpenLitterMap currently works as a web app on + all devices with native mobile apps in development. The stack + includes the integration of the Laravel PHP Framework on the + backend; Vue for frontend reactivity; NativeScript-Vue for mobile + apps; Bulma for CSS; Leaflet for web-mapping; Turf.js for + geospatial analysis; the Ethereum Blockchain for tokenization; + Stripe; ChartJS; AWS; and more. Anywhere from a single cigarette + butt to the contents of an entire beach or street clean can be + logged in a single geotagged photo. Alternatively, a simple index + may be used if litter is incalculable. The open data includes an + increasing 100+ pre-defined types of litter; 20+ corporate + brands; verification status; coordinates; timestamp; phone model; + the latest OpenStreetMap address at each location; and the litter + presence as a Boolean. To date, 100\% of all submitted data (~ + 8200 photos, ~ 28,000 litter from over 150 contributors) has been + manually verified which is being used to develop machine learning + algorithms.", + journal = "Open Geospatial Data, Software and Standards", + volume = 3, + number = 1, + pages = "6", + month = jun, + year = 2018 +} + + + +@misc{WADEAI:2020, + author = {K. Kerge, W. Cowger, K. Haamer, K. Ehala, K. Kivistik, T. Tammiste, M. Vares}, + title = {WADE AI Trash Detection}, + year = {2020}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/letsdoitworld/wade-ai} +} + +@misc{Wuu:2018, + author = {S. Wuu}, + title = {Litter Detection Tensorflow}, + year = {2018}, + publisher = {GitHub}, + journal = {GitHub repository}, + url = {https://github.com/isaychris/litter-detection-tensorflow} +} + + +@article{Majchrowska:2022, +title = {Deep learning-based waste detection in natural and urban environments}, +journal = {Waste Management}, +volume = {138}, +pages = {274-284}, +year = {2022}, +issn = {0956-053X}, +doi = {https://doi.org/10.1016/j.wasman.2021.12.001}, +url = {https://www.sciencedirect.com/science/article/pii/S0956053X21006474}, +author = {Sylwia Majchrowska and Agnieszka Mikołajczyk and Maria Ferlin and Zuzanna Klawikowska and Marta A. Plantykow and Arkadiusz Kwasigroch and Karol Majek}, +keywords = {Object detection, Semi-supervised learning, Waste classification benchmarks, Waste detection benchmarks, Waste localization, Waste recognition}, +abstract = {Waste pollution is one of the most significant environmental issues in the modern world. The importance of recycling is well known, both for economic and ecological reasons, and the industry demands high efficiency. Current studies towards automatic waste detection are hardly comparable due to the lack of benchmarks and widely accepted standards regarding the used metrics and data. Those problems are addressed in this article by providing a critical analysis of over ten existing waste datasets and a brief but constructive review of the existing Deep Learning-based waste detection approaches. This article collects and summarizes previous studies and provides the results of authors’ experiments on the presented datasets, all intended to create a first replicable baseline for litter detection. Moreover, new benchmark datasets detect-waste and classify-waste are proposed that are merged collections from the above-mentioned open-source datasets with unified annotations covering all possible waste categories: bio, glass, metal and plastic, non-recyclable, other, paper, and unknown. Finally, a two-stage detector for litter localization and classification is presented. EfficientDet-D2 is used to localize litter, and EfficientNet-B2 to classify the detected waste into seven categories. The classifier is trained in a semi-supervised fashion making the use of unlabeled images. The proposed approach achieves up to 70% of average precision in waste detection and around 75% of classification accuracy on the test dataset. The code and annotations used in the studies are publicly available online11https://github.com/wimlds-trojmiasto/detect-waste..} +} + +@misc{Proença:2020, + doi = {10.48550/ARXIV.2003.06975}, + url = {https://arxiv.org/abs/2003.06975}, + author = {Proença, Pedro F and Simões, Pedro}, + keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences}, + title = {TACO: Trash Annotations in Context for Litter Detection}, + publisher = {arXiv}, + year = {2020}, + copyright = {arXiv.org perpetual, non-exclusive license} +} + diff --git a/paper.md b/paper.md new file mode 100644 index 0000000..57d97f6 --- /dev/null +++ b/paper.md @@ -0,0 +1,113 @@ +--- +title: 'Trash AI: A Web GUI for Serverless Computer Vision Analysis of Images of Trash' +tags: + - tensorflow.js + - IndexDB + - Plastic Pollution + - Trash + - Litter + - AI + - Image Classification + - Serverless + - vue + - vuetify + - vite + - pinia +authors: + - name: Win Cowger + orcid: 0000-0001-9226-3104 + affiliation: 1 # (Multiple affiliations must be quoted) + - name: Steven Hollingsworth + corresponding: true # (This is how to denote the corresponding author) + affiliation: 2 + - name: Day Fey + affiliation: 2 + - name: Mary C Norris + affiliation: 2 + - name: Walter Yu + affiliation: 3 + - name: Kristiina Kerge + affiliation: 4 + - name: Kris Haamer + affiliation: 4 + - name: Gina Durante + affiliation: 2 + - name: Brianda Hernandez + affiliation: 2 +affiliations: + - name: Moore Institute for Plastic Pollution Research, USA + index: 1 + - name: Code for Sacramento and Open Fresno, USA + index: 2 + - name: California Department of Transportation, USA + index: 3 + - name: Let's Do It Foundation, Estonia + index: 4 +date: 1 September 2022 +bibliography: paper.bib + +--- + +# Summary +Although computer vision classification routines have been created for trash, they have not been accessible to most researchers due to the challenges in deploying the models. Trash AI is a web GUI (Graphical User Interface) for serverless computer vision classification of individual items of trash within images, hosted at www.trashai.org. With a single batch upload and download, a user can automatically describe the types and quantities of trash in all of their images. + +# Statement of need +Trash in the environment is a widespread problem that is difficult to measure. Policy makers require high quality data on trash to create effective policies. Classical measurement techniques require surveyors with pen and paper to manually quantify every piece of trash at a site. This method is time-consuming. Scientists are actively trying to address this issue by using imaging to better understand the prevalence and distribution of trash in an `efficient yet effective manner` [@Majchrowska:2022; @Proença:2020; @Moore:2020; @vanLieshout:2020; @WADEAI:2020; @Lynch:2018; @Wuu:2018; @Waterboards:2018]. An app-based reporting of trash using cell phones, laptops, and other devices has been a `valuable solution` [@Lynch:2018]. Applications for AI in detecting trash currently include: images from `bridges` [@vanLieshout:2020], `drone imaging` [@Moore:2020], cameras on `street sweepers` [@Waterboards:2018], and cell phone app based reporting of `trash` [@Lynch:2018]. Although there are many artificial intelligence algorithms developed for trash classification, none are readily accessible to the average litter researcher. The primary limitation is that artificial intelligence (AI) algorithms are primarily run through programming languages (not graphic user interfaces), difficult to deploy without AI expertise, and often live on a server (which costs money to host). New developments in browser-side AI (e.g., tensorflow.js) and serverless architecture (e.g., AWS Lambda) have created the opportunity to have affordable browser-side artificial intelligence in a web GUI, alleviating both obstacles. We present Trash AI, an open source service for making computer vision available to anyone with a web browser and images of trash. + +# Demo +We have a full video tutorial on [Youtube](https://youtu.be/HHrjUpQynUM) + +## Basic workflow: +### 1. + +![Upload images by dragging onto the screen.\label{fig:example1}](https://user-images.githubusercontent.com/26821843/188520590-86d7b0b3-1b40-4ce5-8fb0-1be54b2de20e.png) + +### 2. + +![View results while images are processing.\label{fig:example2}](https://user-images.githubusercontent.com/26821843/188520700-43f4c964-c430-4a78-843b-68ae7aae2ba2.png) + +### 3. + +![View summary results of detected trash.\label{fig:example3}](https://user-images.githubusercontent.com/26821843/188520723-92b50200-d568-4953-aa26-fbcbbd965a38.png) + +### 4. + +![View results mapped if your images have location stamp.\label{fig:example4}](https://user-images.githubusercontent.com/26821843/188520745-65ef3270-6093-488a-b501-305ecb436bc1.png) + +### 5. + +![Click download all to extract a zip folder with labeled images and metadata.\label{fig:example5}](https://user-images.githubusercontent.com/26821843/188520813-f9169ba9-14d9-4f11-bf53-a6fd8e379cdf.png) + +### 6. + +![View labeled images from downloaded results.\label{fig:example6}](https://user-images.githubusercontent.com/26821843/188520833-d313279d-b2d0-4d37-ac0b-670ce3252540.png) + +### 7. + +![View metadata for each image using "image_hash.json" (using https://jsoneditoronline.org/).\label{fig:example7}](https://user-images.githubusercontent.com/26821843/188520860-629c529d-dc5e-4e93-9beb-b65e4560bc13.png) + +### 8. + +![View metadata for all images in "summary.json" (using https://jsoneditoronline.org/).\label{fig:example8}](https://user-images.githubusercontent.com/26821843/188520906-3061ecce-cb0e-4c76-9b81-303731110380.png) + +# Method + +## Workflow Overview +Trash AI is trained on the [TACO dataset](http://tacodataset.org/) using [YOLO 5](pytorch.org). Trash AI stores images in [IndexDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) to keep the data primarily browser side and uses [tensorflow.js](https://www.tensorflow.org/js) to keep analysis browser side too. When images are uploaded to the browser, Trash AI provides the prediction of the model as a graphical output. The raw data from the model and labeled images can be downloaded in a batch download to expedite analyses. Any data uploaded to the platform may be automatically saved to an [S3 bucket](https://aws.amazon.com/s3/), which we can use to improve the model over time. + +## AI Training +The AI model was developed starting with the TACO dataset, which was available with a complimentary Jupyter Notebook on [Kaggle](https://www.kaggle.com/datasets/kneroma/tacotrashdataset). An example notebook was referenced, which used the default `YOLO v5 model` [@Jocher:2020] as the basic model to begin transfer learning. Next, transfer learning was completed using the entire TACO dataset to import the image classes and annotations in the YOLO v5 model. + +## Limitations +From our experience, the accuracy of the model varies depending on the quality of the images and their context/background. "Trash" is a word people use for an object that lacks purpose, and the purpose of an object is often not obvious in an image. Trash is a nuanced classification because the same object in different settings will not be considered trash (e.g., a drink bottle on someone's desk vs in the forest laying on the ground). This is the main challenge with any image-based trash detection algorithm. Not everything that LOOKS like trash IS trash. This and other complexities to trash classification make a general trash AI a challenging (yet worthwhile) long-term endeavor. The algorithm is primarily trained on single pieces of trash in the image, with the trash laying on the ground. Thus, model class prediction of trash in these kinds of images will generally be better than trash appearing in aerial images, for example. Additionally, user feedback has shows that the distance of trash from the camera is a critical aspect. The model performs ideally with single pieces of trash in an image less than 1 m away. The model performs less accurately on images when trash which is farther away such as when taken from a vehicle. This is likely due to the training data, TACO dataset, which consists primarily of images of trash close to the camera. + +# Availability +Trash AI is hosted on the web at www.trashai.org. The source code is [available on github](https://github.com/code4sac/trash-ai) with an [MIT license](https://mit-license.org/). The source code can be run offline on any machine that can install [Docker and Docker-compose](www.docker.com). [Documentation](https://github.com/code4sac/trash-ai#ai-for-litter-detection-web-application) is maintained by Code for Sacramento and Open Fresno on Github and will be updated with each release. [Nonexhaustive instructions for AWS deployment](https://github.com/code4sac/trash-ai/blob/manuscript/docs/git-aws-account-setup.md) is available for anyone attempting production level deployment. The image datasets shared to the tool are in an S3 Bucket that needs to be reviewed before being shared with others due to security and moderation concerns but can be acquired by [contacting the repo maintaniers](https://github.com/code4sac/trash-ai/graphs/contributors). + +# Future Goals +This workflow is likely to be highly useful for a wide variety of computer vision applications and we hope that people reuse the code for applications beyond trash detection. We aim to increase the labeling of images by creating a user interface that allows users to improve the annotations that the model is currently predicting by manually restructuring the bounding boxes and relabeling the classes. We aim to work in collaboration with the TACO development team to improve our workflow integration to get the data that people share to our S3 bucket into the [TACO training dataset](http://tacodataset.org/) and trained model. Future models will expand the annotations to include the `Trash Taxonomy` [@Hapich:2022] classes and add an option to choose between other models besides the current model. + +# Acknowledgements +Code for Sacramento and Open Fresno led the development of the software tool. The Moore Institute advised on priorities and led the drafting of this manuscript. Let's Do It Foundation assisted with original products leading up to trash AI in the development of WADE AI. We acknowledge the work of the Code for Sacramento and Open Fresno team, brigades of Code for America, without whom this project would not have been possible, and acknowledge the input of the California Water Monitoring Council Trash Monitoring Workgroup. In particular, we would like to acknowledge Kevin Fries, J.Z. Zhang, Joseph Falkner, Democracy Lab, Brad Anderson, Jim Ewald, Don Brower, and University of Houston. We acknowledge financial support from McPike Zima Charitable Foundation. + +# References