Adding various Canvas functions #33

star-nox · 2023-08-10T22:06:14Z

added function for getting canvas user emails

Here's our development course: https://canvas.illinois.edu/courses/37348

lintrule-review · 2023-08-10T22:06:16Z

Lintrule is disabled.

  You can fix that by putting in a card [here.](https://lintrule.com/dashboard)

railway-app · 2023-08-10T22:06:18Z

This PR is being deployed to Railway 🚅

flask: ◻️ REMOVED

KastanDay

I didn’t test it, but the code structure & quality looks very nice.

KastanDay · 2023-08-14T22:29:01Z

How do we know we were successful?

All documents on Canvas are uploaded to course, with diff mechanism. Try to make this diff mechanism reusable for @jkmin3 web scraper.
All students in the course are auto-added to course_users. Email the UIUC help desk. We want to talk tot the canvas people about creating a student that profs can add to their course. Should be a regular student. ✉️ [email protected]
Diff mechanism: Input:
1. course_metadata object containing course_name & base_url
2. A path to a folder, pointing to a collection of NEWLY scraped documents

Auth?

How do professors auth? Can they add a special UIUC-chat canvas user as a student? Or provider their own API keys.

No event listeners on Canvas, so use cron job to do DAILY uploads.

Diff mechanism (no file types considerations).
Just ask chatGPT to do it...

import hashlib

def generate_checksum(file_path):
    sha256_hash = hashlib.sha256()
    with open(file_path, "rb") as file:
        # Read and update the hash string value in blocks
        for byte_block in iter(lambda: file.read(4096), b""):
            sha256_hash.update(byte_block)
    return sha256_hash.hexdigest()

gitguardian · 2023-08-15T19:42:27Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	Secret	Commit	Filename
7724259	Supabase Service Role JWT	`deceb15`	ai_ta_backend/nomic.ipynb	View secret
7724259	Supabase Service Role JWT	`07238a2`	ai_ta_backend/nomic.ipynb	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!}

ai_ta_backend/data_logging.py

KastanDay · 2023-08-21T22:20:18Z

We also want to grab all "page content" which is defined as Syllabus page, Rubrics page, maybe others too. Only if they exist.
We want these pages if they're helpful.

Remember the "black hole" ingest philosophy.

KastanDay · 2023-08-22T00:58:09Z

Adding a link to the current Canvas apps on Illinois Canvas@Illinois, General LTI Integrations (uillinois.edu)

ai_ta_backend/update_materials.py

star-nox · 2023-08-25T19:21:18Z

Added uiuc.chat as a Teacher to the dev course and replaced access token. Student doesn't have access to user IDs, so can't implement add_users() with a Student profile.
Need to pass the collected user emails to the function which adds users to uiuc.chat.
Ingest and Update functions for Canvas files are done. Looking into scraping the page content next.

star-nox · 2023-08-25T22:57:05Z

We are now able to scrape all of the content in Canvas (syllabus, pages, etc.)!

KastanDay · 2023-08-29T01:31:54Z

Wow fantastic, can we do a live demo on next meeting?

KastanDay · 2023-08-29T01:33:33Z

Make sure to finish any details necessary so professors can start using it. Just LMK if you have any questions, the more questions the better.

star-nox · 2023-09-18T22:15:35Z

Current progress on the feature: Currently it exports all course content irrespective of whether anything is unpublished. I am working on creating a list of published content first and then passing it to the content export API, so only those files are exported.

KastanDay · 2023-09-19T02:06:04Z

Shouldn’t it be fine? If the “student user” can see it, then it’s meant to be seen?

…

On Mon, Sep 18, 2023 at 3:15 PM Asmita Dabholkar ***@***.***> wrote: Current progress on the feature: Currently it exports all course content irrespective of whether anything is unpublished. I am working on creating a list of published content first and then passing it to the content export API, so only those files are exported. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADH2CNOVIDJ7FXD4FT4SC4DX3DBZDANCNFSM6AAAAAA3MDQF5A> . You are receiving this because you commented.Message ID: ***@***.***>

rohan-uiuc · 2023-09-19T21:25:01Z

These might be helpful @star-nox
https://community.canvaslms.com/t5/Canvas-Question-Forum/Use-API-to-export-only-pages/m-p/460735
https://community.canvaslms.com/t5/Canvas-Developers-Group/Using-Content-Migrations-API-to-copy-courses-w-o-Announcements/m-p/470539

star-nox · 2023-09-19T21:39:05Z

These might be helpful @star-nox https://community.canvaslms.com/t5/Canvas-Question-Forum/Use-API-to-export-only-pages/m-p/460735 https://community.canvaslms.com/t5/Canvas-Developers-Group/Using-Content-Migrations-API-to-copy-courses-w-o-Announcements/m-p/470539

Selective content export yes! I was trying exactly that before Kastan suggested the individual files route. I have uiuc.chat as a user in the course. I have the bearer token set. But the content_exports API call returns an authorization error as a student, but works as a teacher.

star-nox · 2023-09-19T22:13:46Z

Yeah but we need to use the student profile, not teacher. We verified that this should work before... please try to fix.

Don't you remember how we debated endlessly between:

❌ Forcing course creators to supply their Canvas API key and

✅ Using a student account that professors invite into their course? We need to use the student account.

Oh, by teacher I don't mean the course creators. I am referring to using the UIUC Chat account which professors will add to their course. Instead of adding it as a student, I was adding it as a teacher.

star-nox · 2023-09-21T23:02:07Z

@KastanDay I have updated the ingest and add_users functions here.
I have modified the workflow of ingest such that it takes a dictionary containing boolean values for what content to ingest and acts accordingly. I am currently working on modifying the update function to reflect this workflow.

KastanDay · 2023-09-22T17:42:35Z

Hi @star-nox, do you think any of this is ready for testing & review? I'd love to get an MVP live ASAP so I can give it to George to better understand what users will want. Thanks.

KastanDay

Needs a minor change to update_files(), see below

ai_ta_backend/vector_database.py

ai_ta_backend/update_materials.py

star-nox · 2023-09-22T17:54:43Z

Hi @star-nox, do you think any of this is ready for testing & review? I'd love to get an MVP live ASAP so I can give it to George to better understand what users will want. Thanks.

I was facing some JSON error when I passed this dictionary as a parameter in the API call. It is supposed to come from the front-end. For now I have hardcoded it into the function - it ingests all of the below things. It works well for now though, you can test it.

            # a dictionary of all contents we want to ingest - files, pages, modules, syllabus, assignments, discussions.
            content_to_ingest = {
                'files': True,
                'pages': True,
                'modules': True,
                'syllabus': True,
                'assignments': True,
                'discussions': True
            }

star-nox · 2023-09-22T20:03:51Z

@KastanDay update_files() is also ready for review.

KastanDay · 2023-09-29T22:06:07Z

ai_ta_backend/update_materials.py

+
+
+    # Access checksum of s3 files
+    s3_client = boto3.client('s3', aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),


Hmm, this is really expensive to download all files from S3.

In the end, it's the exact same as using the Document Contexts from Supabase SQL. I think we have to re-implement this with Supabase instead of S3. It'll be hundreds of times faster and cheaper and probably conceptually simpler too.

Oh okay. I thought get_object() just reads the files without downloading them. I use it to access the ETag attribute which is the md5 hash of the file.
Won't the Supabase contexts give us a different checksum than the original non-manipulated file?

ai_ta_backend/main.py

KastanDay

Changes requested in main & update_materials.

star-nox · 2023-10-02T23:44:37Z

@KastanDay I have separated the file update functions into a different PR. This one now contains the addCanvasUsers and ingestCanvas APIs which are ready for testing and demo. I have added these APIs to our Postman workspace too.

KastanDay

Fully working in my end-to-end testing. Merging!

added the add_users() for Canvas

c42f606

star-nox had a problem deploying to lintrule August 10, 2023 22:06 — with GitHub Actions Failure

added canvas course ingest

6854205

star-nox had a problem deploying to lintrule August 13, 2023 21:55 — with GitHub Actions Failure

updated requirements

54e3fb0

star-nox had a problem deploying to lintrule August 13, 2023 21:58 — with GitHub Actions Failure

KastanDay assigned star-nox Aug 14, 2023

KastanDay approved these changes Aug 14, 2023

View reviewed changes

star-nox added 2 commits August 15, 2023 14:25

added .md ingest and fixed .py ingest

07238a2

deleted test ipynb file

deceb15

added nomic viz

27383e1

KastanDay reviewed Aug 21, 2023

View reviewed changes

ai_ta_backend/data_logging.py Outdated Show resolved Hide resolved

added canvas file update function

6f08340

KastanDay reviewed Aug 22, 2023

View reviewed changes

ai_ta_backend/update_materials.py Outdated Show resolved Hide resolved

completed update function

34cbbdc

star-nox added 2 commits August 25, 2023 17:51

updated course export to include all contents

efd9048

modified to handle diff file structures of downloaded content

bf3726b

modified canvas update

93646ac

root and others added 2 commits September 21, 2023 17:32

modified add_users() and ingest_course_content() functions

05ab444

modified ingest function

f5655ab

KastanDay requested changes Sep 22, 2023

View reviewed changes

ai_ta_backend/vector_database.py Show resolved Hide resolved

ai_ta_backend/update_materials.py Outdated Show resolved Hide resolved

star-nox added 2 commits September 22, 2023 14:58

modified update_files() for file replacement

6f80b96

removed the extra os.remove()

0223a22

star-nox requested review from KastanDay September 22, 2023 20:04

fix underscore to dash in for pip

2e10cc8

KastanDay requested changes Sep 29, 2023

View reviewed changes

star-nox added 3 commits October 2, 2023 12:33

removed json import and added abort to canvas functions

a38fb90

Merge branch 'main' into canvas

79142c5

removed file update functions from this PR

7a9c21d

star-nox requested a review from KastanDay October 2, 2023 23:46

Maxwell-Lindsey and others added 5 commits October 23, 2023 12:05

added ability to receive canvas checkbox data

65b2ba1

Merge branch 'main' into canvas

acd61e7

removed unused import

7c4a839

added comment for debugging

8de8900

removing pypandoc because it appears to be unused

46646c7

KastanDay approved these changes Oct 23, 2023

View reviewed changes

KastanDay merged commit 89f61aa into main Oct 23, 2023
1 check failed

KastanDay deleted the canvas branch October 23, 2023 20:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding various Canvas functions #33

Adding various Canvas functions #33

star-nox commented Aug 10, 2023 •

edited by KastanDay

Loading

lintrule-review bot commented Aug 10, 2023

railway-app bot commented Aug 10, 2023 •

edited

Loading

KastanDay left a comment

KastanDay commented Aug 14, 2023 •

edited

Loading

gitguardian bot commented Aug 15, 2023 •

edited

Loading

KastanDay commented Aug 21, 2023

KastanDay commented Aug 22, 2023

star-nox commented Aug 25, 2023

star-nox commented Aug 25, 2023

KastanDay commented Aug 29, 2023

KastanDay commented Aug 29, 2023

star-nox commented Sep 18, 2023

KastanDay commented Sep 19, 2023 via email

rohan-uiuc commented Sep 19, 2023

star-nox commented Sep 19, 2023

star-nox commented Sep 19, 2023

star-nox commented Sep 21, 2023

KastanDay commented Sep 22, 2023

KastanDay left a comment •

edited

Loading

star-nox commented Sep 22, 2023

star-nox commented Sep 22, 2023

KastanDay Sep 29, 2023

star-nox Oct 2, 2023

KastanDay left a comment

star-nox commented Oct 2, 2023

KastanDay left a comment



		# Access checksum of s3 files
		s3_client = boto3.client('s3', aws_access_key_id=os.getenv('AWS_ACCESS_KEY_ID'),

Adding various Canvas functions #33

Adding various Canvas functions #33

Conversation

star-nox commented Aug 10, 2023 • edited by KastanDay Loading

lintrule-review bot commented Aug 10, 2023

Lintrule is disabled.

railway-app bot commented Aug 10, 2023 • edited Loading

KastanDay left a comment

Choose a reason for hiding this comment

KastanDay commented Aug 14, 2023 • edited Loading

gitguardian bot commented Aug 15, 2023 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

KastanDay commented Aug 21, 2023

KastanDay commented Aug 22, 2023

star-nox commented Aug 25, 2023

star-nox commented Aug 25, 2023

KastanDay commented Aug 29, 2023

KastanDay commented Aug 29, 2023

star-nox commented Sep 18, 2023

KastanDay commented Sep 19, 2023 via email

rohan-uiuc commented Sep 19, 2023

star-nox commented Sep 19, 2023

star-nox commented Sep 19, 2023

star-nox commented Sep 21, 2023

KastanDay commented Sep 22, 2023

KastanDay left a comment • edited Loading

Choose a reason for hiding this comment

star-nox commented Sep 22, 2023

star-nox commented Sep 22, 2023

KastanDay Sep 29, 2023

Choose a reason for hiding this comment

star-nox Oct 2, 2023

Choose a reason for hiding this comment

KastanDay left a comment

Choose a reason for hiding this comment

star-nox commented Oct 2, 2023

KastanDay left a comment

Choose a reason for hiding this comment

star-nox commented Aug 10, 2023 •

edited by KastanDay

Loading

railway-app bot commented Aug 10, 2023 •

edited

Loading

KastanDay commented Aug 14, 2023 •

edited

Loading

gitguardian bot commented Aug 15, 2023 •

edited

Loading

KastanDay left a comment •

edited

Loading