Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor slot and professors spiders in light of sigarra schedule html page changes #112

Merged
merged 6 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 4 additions & 12 deletions src/Makefile
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
.PHONY: all clean
FILES = $(shell ls)

all: info faculties courses course_units course_metadata classes slots slot_professor professors
# all: info faculties courses course_units course_metadata classes slots slot_professor professors
all: info faculties courses course_units course_metadata slots


faculties:
Expand All @@ -16,18 +17,9 @@ course_units:
course_metadata:
scrapy crawl course_metadata

classes:
scrapy crawl classes

slots:
scrapy crawl slots

slot_professor:
scrapy crawl slot_professor

professors:
scrapy crawl professors

info:
python ./scrapper/info.py

Expand All @@ -43,9 +35,9 @@ upload:

convert_mysql:
@echo "Converting dump to mysql..."
@bash ./scripts/sqlite3-to-mysql.sh ./scripts/dump/data/dump_sqlite3.sql > ./scripts/dump/data/dump_mysql.sql
@bash ./scripts/sqlite3-to-mysql.sh ./scripts/dump/data/dump_sqlite3.sql > ./scripts/dump/data/01_data.sql
@echo "Convertion completed!"

clean:
@echo "Removing database.db"
@rm ./scrapper/database/dbs/database.db
@rm ./scrapper/database/dbs/database.db
9 changes: 7 additions & 2 deletions src/scrapper/items.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@

import scrapy


class Faculty(scrapy.Item):
acronym = scrapy.Field()
name = scrapy.Field()
Expand Down Expand Up @@ -37,18 +38,22 @@ class CourseUnit(scrapy.Item):
classes_url = scrapy.Field()
last_updated = scrapy.Field()


class CourseMetadata(scrapy.Item):
course_id = scrapy.Field()
course_unit_id = scrapy.Field()
course_unit_year = scrapy.Field()
ects = scrapy.Field()


class Class(scrapy.Item):
course_unit_id = scrapy.Field()
name = scrapy.Field() # 1MIEIC01
last_updated = scrapy.Field()



class Slot(scrapy.Item):
id = scrapy.Field()
lesson_type = scrapy.Field() # T, TP, PL, etc.
day = scrapy.Field() # 0 = monday, 1 = tuesday, .., 5 = saturday (no sunday)
start_time = scrapy.Field() # At what time the lesson starts
Expand All @@ -64,8 +69,8 @@ class SlotProfessor(scrapy.Item):
slot_id = scrapy.Field()
professor_id = scrapy.Field()


class Professor(scrapy.Item):
id = scrapy.Field()
professor_acronym = scrapy.Field()
professor_name = scrapy.Field()

133 changes: 0 additions & 133 deletions src/scrapper/spiders/class_spider.py

This file was deleted.

98 changes: 0 additions & 98 deletions src/scrapper/spiders/professor_spider.py

This file was deleted.

Loading