Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Improve documentation #31

Merged
merged 10 commits into from
Feb 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
274 changes: 42 additions & 232 deletions README.md

Large diffs are not rendered by default.

27 changes: 27 additions & 0 deletions TODO
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
- [ ] An easy setup script / deploy my own Ollama server to replace OpenAI [ticket #6](https://github.com/adamcohenhillel/AdDeus/issues/6)
- [ ] Add How-to for Ollama setup ([ticket #9](https://github.com/adamcohenhillel/AdDeus/issues/9))

### Known Bugs:

- [ ] Whisper tends to generate YouTube-like text when the audio is unclear, so you can get noise data in the database like "Thank you for watching", and "See you in the next video," even though it has nothing to do with the audio ([ticket #7](https://github.com/adamcohenhillel/AdDeus/issues/7))

- [ ] Currently it is using Wi-Fi, which makes it not-so mobile. An alternative approach would either be: ([ticket #8](https://github.com/adamcohenhillel/AdDeus/issues/8))

- Bluetooth, pairing with the mobile device
- Sdd a 4G card that will allow it to be completly independent

- [ ] Sometimes when loading from scratch, it takes some time (2-3 curl requests) until it resolves the DNS of the Supabase instance ([ticket #8](https://github.com/adamcohenhillel/AdDeus/issues/12))

#### Backend:

- The RAG (Retrieval-Augmented Generation) can be extremely improved:
- [ ] Need to process the audio not only into "embeddings" but also run an LLM on it to generate some context ([ticket #1](https://github.com/adamcohenhillel/AdDeus/issues/1))
- [ ] Need to query the RAG more efficiently, maybe with timestamp as well, etc. - not only embeddings (relates to the processing part) ([ticket #2](https://github.com/adamcohenhillel/AdDeus/issues/2))
- [ ] Improve security - currently I didn't spent too much time making the Supabase RLS really work (for writing data) ([ticket #3](https://github.com/adamcohenhillel/AdDeus/issues/3))

#### Hardware / On-device:

- [ ] Run on a Rasberry Pi Pico / Zero, as it is much much cheaper, and should do the work too ([ticket #4](https://github.com/adamcohenhillel/AdDeus/issues/4))
- [ ] Currently the setup is without battery, need to find the easiest way to add this as part of the setup ([ticket #5](https://github.com/adamcohenhillel/AdDeus/issues/5))

#### UX and Onboarding
178 changes: 178 additions & 0 deletions docs/archive/launch_readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# ADeus: Open-Source AI Wearable Device

In the upcoming world of AI devices like [Tab](https://mytab.ai/), [Pin](https://hu.ma.ne/aipin), [Rewind](https://www.rewind.ai/pendant), that are with us all the time, **literally** listening to everything we say, and know much about us as our closest friends - it is crucial to be able to own this setup, to own our data, to have this completely open source, managed by the user itself.

**This is Adeus, the Open Source AI Wearble device -** and in this repo, you will be guided on how to set up your own! From buying the hardware (~$100, and will be cheaper once we finish the Raspberry PI Zero version) to setting up the backend, the software, and start using your wearable!

- [Introduction](#introduction)
- [Setup](#setup)
- [Prerequisite](#prerequisite-110)
- [Supabase](#setup-supabase)
- [Web / Mobile App](#setup-app-web)
- [Hardware (Coral AI)](#setup-hardware---coral-ai-device)
- [Hardware (Rasberry Pi Zero)](#setup-hardware---rasberry-pi-zero-w)
- [Run with Ollama](#setup-run-with-ollama)
- [Areas to Contribute](#areas-to-contribute)
- [CTA for the Community](#cta-for-the-community)

![ADeus v0.1](docs/images/adeus_01.jpeg)

> p.s. any contribution would be amazing, whether you know how to code, and want to jump straight in to the codebase, a hardware person who can help out, or just looking to support this project financially (can literally be $10) - please reach out to me on X/Twitter [@adamcohenhillel](https://twitter.com/adamcohenhillel)

### Introduction:

Adeus consists of 3 parts:

1. **A mobile / web app:**
an interface that lets the user to interact with their assistant and data via chat.

2. **Hardware device (Currently Coral AI, but soon a Rasberry-Pi Zero W worth $15):** this will be the wearable that will record everything, and send it to the backend to be processed
3. **Supabase :** Our backend, and datavase, where we will process and store data, and interact with LLMs.
Supabase is an open source Firebase alternative, a "backend-as-a-service" - which allows you to setup a Postgres database, Authentication, Edge Functions, Vector embeddings, and more - for free (at first) and at extreme ease!
- [!!] But more importantly - **it is open source, and you can choose to deploy and manage your own Supabase instance** - which us crucial for our mission: A truly open-source, personal AI.

This will look something like:
![ADeus diagram](docs/images/adeus_diagram.png)

## Setup

> Note: I'm working on an easy setup.sh file that will do everything here more or less automatically, but it is still in the making

A'ight, let's get this working for you!

#### Prerequisite: ($110)

1. [Dev Board Micro](https://coral.ai/products/dev-board-micro/) ($80)
2. [Wireless/Bluetooth Add-on](https://coral.ai/products/wireless-add-on/) ($20)
3. [A case](https://coral.ai/products/dev-board-micro-case/) (Optional, 10$)
4. Either an OpenAI key, or a Ollama server running somewhere you can reach via internet

> Note: We are working on a version of this working with Raspberry PI Zero W, which will cost ~$20, stay tuned

First - cloning the repo:

```bash
git clone https://github.com/adamcohenhillel/ADeus
```

#### Setup: Supabase

Check out

#### Setup: App (Web)

check out

### Setup: Hardware - Coral AI device

First, to learn more about the device, it is good to check out the [official docs](https://coral.ai/docs/dev-board-micro/get-started/). Our project is using [out-of-tree setup](official) with a [Wireless Add-on](https://coral.ai/docs/dev-board-micro/wireless-addon/).

Here is quick video showing how to "connect" the hardware pieces together, and install the software:

[![set up device video](docs/images/thumbnail_2.png)](https://youtu.be/_2KRSlpnXrA)

In the root folder of this repository, run the following commands, (which will download the Coral AI Micro Dev dependencies to your computer - note that it might take a few minutes):

```bash
git submodule add https://github.com/google-coral/coralmicro devices/coralai/coralmicro
```

```bash
git submodule update --init --recursive
```

Then, when it is finished, CD to the `devices/coralai` folder:

```bash
cd devices/coralai
```

And run the setup script, which will make sure your computer can compile the code and pass it on to the device:

> Note that if you're using Apple Silicon Mac, you might need to change the `coralmicro/scripts/requirements.txt` file, making the version of the package `hidapi==0.14.0` (see [issue](https://github.com/google-coral/coralmicro/pull/98))

```bash
bash coralmicro/setup.sh
```

```bash
export SUPABASE_URL"<YOUR_SUPABASE_URL"
```

> Note: Security RLS best practices is still WOP! (see [ticket #3](https://github.com/adamcohenhillel/AdDeus/issues/3))

Once the setup has finished running, you can connect your device via a USB-C, and run the following to create a build:

```bash
cmake -B out -S .
```

```bash
make -C out -j4
```

And then, flash it to your device with WIFI_NAME and WIFI_PASSWORD: (Bluetooth pairing is coming soon, see [ticket][https://github.com/adamcohenhillel/AdDeus/issues/8])

```bash
python3 coralmicro/scripts/flashtool.py --build_dir out --elf_path out/coralmicro-app --wifi_ssid "<WIFI_NAME>" --wifi_psk "<WIFI_PASSWORD>"
```

To debug the device, you can connect to it serial-y via the USB-C.

First, find the serial id
On Linux:

```bash
ls /dev/ttyACM*
```

On Mac:

```bash
ls /dev/cu.usbmodem*
```

Then run the [checkOutput.py](/devices/coralai/checkOutput.py) script:

```python
python3 checkOutput.py --device "/dev/cu.usbmodem101"
```

(replace the `/dev/cu.usbmodem*` with whatever you got in the `ls` command)

> Note: It might fail for the first few CURL requests, until it resolves the DNS

### Setup: Hardware - Rasberry Pi Zero W

SOON! (cost $15, but need to solder a microphone)

### Setup: Run with Ollama

How-to-Guide will be written here soon, but it should be fairly simple with [Ollama](https://ollama.ai/) serve and `ngrok http 11434`

```
brew install ngrok/ngrok/ngrok
```

## How to Contribute:

As people will soon notice, my C++ skills are limited, as well as my React and hardware skills :P - any help would be amazing! Contributions are more than welcomed. This should be maintained by us, for us.

Build it for yourself, and build it for others. This can become the Linux of the OS, the Android of the mobile. It is raw, but we need to start from somewhere!

- [ ] An easy setup script / deploy my own Ollama server to replace OpenAI [ticket #6](https://github.com/adamcohenhillel/AdDeus/issues/6)
- [ ] Add How-to for Ollama setup ([ticket #9](https://github.com/adamcohenhillel/AdDeus/issues/9))

## CTA for the Community:

A lot of companies and organizations are now after building the "Personal AI" - the one that will be a companion for individuals. This will be a paradigm shift of the way we all experince the digital (and physical) realms. Interacting with our AI that knows a lot about us, and help us navigate the world.

The problem with all these initiatives, is that they don't really provide you with your own personal AI. It’s not private, you don’t own it. As long as you don’t have a way to opt out, and take your so-called personal AI elsewhere, it’s not yours, you merely renting it from somewhere.

**Personal AI should be like a personal computer, connected to the internet.**

The pioneers of the personal computers, the internet, they all knew it - and that what made it great, a period of possibilities. But since, as we all know, things had drifted. You don’t own things, merely renting them. You can’t take it elsewhere - and therefore the free-market forces of capitalism can’t be easily integrated into the digital realm.

Check out the Intro video:

[![set up device video](docs/images/thumbnail_1.png)](https://youtu.be/4CqEC2yLGQU)
33 changes: 33 additions & 0 deletions docs/how-to/setup_app.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
---
title: Setting up the mobile / web app
description: add description
---

Now that you have a Supabase instance that is up and running, you can technically start chatting with your assistant, it just won't have any personal data it.

To try it out, you can either use the deployed version of the web app here: [adeusai.com](https://adeusai.com) - which will ask you to connect to your own Supabase instance (it is only a frontend client).

Or you can deploy the app yourself somewhere - the easiest is Vercel, or locally:

from the root folder:

```bash
cd ./app
```

npm install and run:

```bash
npm i
npm run dev
```

Once you have an app instance up and running, head to its address `your-app-address.com/`, and you should see the screen:

<img src="docs/images/login_screenshot.png" width="150">

Enter the four required details, which you should've obtained in the Supabase setup: `Supabase URL`, `Supabase Anon API Key`, `email` and `password`.

And you should be able to start chatting!

Now - let's configure our hardware device, so we could start provide crucial context to our personal AI!
92 changes: 92 additions & 0 deletions docs/how-to/setup_coralai.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
---
title: Setting up the CoralAI hardware device
description: add description
---

#### Prerequisite: ($110)

1. [Dev Board Micro](https://coral.ai/products/dev-board-micro/) ($80)
2. [Wireless/Bluetooth Add-on](https://coral.ai/products/wireless-add-on/) ($20)
3. [A case](https://coral.ai/products/dev-board-micro-case/) (Optional, 10$)
4. Either an OpenAI key, or a Ollama server running somewhere you can reach via internet

> Note: We are working on a version of this working with Raspberry PI Zero W, which will cost ~$20, stay tuned

### Setup: Hardware - Coral AI device

First, to learn more about the device, it is good to check out the [official docs](https://coral.ai/docs/dev-board-micro/get-started/). Our project is using [out-of-tree setup](official) with a [Wireless Add-on](https://coral.ai/docs/dev-board-micro/wireless-addon/).

Here is quick video showing how to "connect" the hardware pieces together, and install the software:

[![set up device video](docs/images/thumbnail_2.png)](https://youtu.be/_2KRSlpnXrA)

In the root folder of this repository, run the following commands, (which will download the Coral AI Micro Dev dependencies to your computer - note that it might take a few minutes):

```bash
git submodule add https://github.com/google-coral/coralmicro devices/coralai/coralmicro
```

```bash
git submodule update --init --recursive
```

Then, when it is finished, CD to the `devices/coralai` folder:

```bash
cd devices/coralai
```

And run the setup script, which will make sure your computer can compile the code and pass it on to the device:

> Note that if you're using Apple Silicon Mac, you might need to change the `coralmicro/scripts/requirements.txt` file, making the version of the package `hidapi==0.14.0` (see [issue](https://github.com/google-coral/coralmicro/pull/98))

```bash
bash coralmicro/setup.sh
```

```bash
export SUPABASE_URL"<YOUR_SUPABASE_URL"
```

> Note: Security RLS best practices is still WOP! (see [ticket #3](https://github.com/adamcohenhillel/AdDeus/issues/3))

Once the setup has finished running, you can connect your device via a USB-C, and run the following to create a build:

```bash
cmake -B out -S .
```

```bash
make -C out -j4
```

And then, flash it to your device with WIFI_NAME and WIFI_PASSWORD: (Bluetooth pairing is coming soon, see [ticket][https://github.com/adamcohenhillel/AdDeus/issues/8])

```bash
python3 coralmicro/scripts/flashtool.py --build_dir out --elf_path out/coralmicro-app --wifi_ssid "<WIFI_NAME>" --wifi_psk "<WIFI_PASSWORD>"
```

To debug the device, you can connect to it serial-y via the USB-C.

First, find the serial id
On Linux:

```bash
ls /dev/ttyACM*
```

On Mac:

```bash
ls /dev/cu.usbmodem*
```

Then run the [checkOutput.py](/devices/coralai/checkOutput.py) script:

```python
python3 checkOutput.py --device "/dev/cu.usbmodem101"
```

(replace the `/dev/cu.usbmodem*` with whatever you got in the `ls` command)

> Note: It might fail for the first few CURL requests, until it resolves the DNS
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ Download the Raspberry Pi OS Lite (64-bit) image from the following link:
- Add WiFi settings.
- Write the image to the SD card.

<img src="images/imagerMenu.png" width="200" alt="Home">
<img src="images/osSelect.png" width="200" alt="Home">
<img src="../images/imagerMenu.png" width="200" alt="Home">
<img src="../images/osSelect.png" width="200" alt="Home">

<img src="images/enableSettings.png" width="200" alt="Home">
<img src="images/enableSSH.png" width="200" alt="Home">
<img src="../images/enableSettings.png" width="200" alt="Home">
<img src="../images/enableSSH.png" width="200" alt="Home">

### 4. Boot Raspberry Pi

Expand All @@ -51,6 +51,7 @@ Download the Raspberry Pi OS Lite (64-bit) image from the following link:
### 5. Connect via SSH

Open a terminal on your computer and connect to your Raspberry Pi using SSH:

```
ssh [email protected]
```
Expand Down Expand Up @@ -90,6 +91,7 @@ Enter the `SUPABASE_URL` and `AUTH_TOKEN` when prompted.
### 8. Reboot

Reboot your Raspberry Pi:

```
sudo reboot
```
Expand All @@ -111,10 +113,11 @@ chmod +x compile.sh
```

```
gdb ./main
gdb ./main
```

Optional: Start the application with GDB for debugging

Use `run` within gdb to start the application in debug mode.

Congratulations! You have successfully set up your Raspberry Pi Zero W for development with your project.
Congratulations! You have successfully set up your Raspberry Pi Zero W for development with your project.
Loading