Refactor slot and professors spiders in light of sigarra schedule html page changes #112
Conversation
tomaspalma
force-pushed
the
refactor/use-sigarra-api-link-on-slot-fetching
branch
from
4103cb6
to
b343501
Compare
tomaspalma
force-pushed
the
refactor/use-sigarra-api-link-on-slot-fetching
branch
from
bde541a
to
b975076
Compare
tomaspalma
force-pushed
the
refactor/use-sigarra-api-link-on-slot-fetching
branch
from
b975076
to
0bf499b
Compare
src/scrapper/spiders/slot_spider.py
Outdated
| (sigarra_id, name) = self.get_professor_info( | ||
| teacher) # (sigarra_id | None, teacher_name) | ||
|
|
||
| yield SlotProfessor( | ||
| slot_id=current_class["sigarra_id"] | ||
| professor_id=teacher["sigarra_id"] | ||
| slot_id=current_class_id, | ||
| professor_id=sigarra_id |
There was a problem hiding this comment.
Is None a possible value? What happens in these cases?
There was a problem hiding this comment.
None was a possible value in the first version of the get_professor_info method at the time that comment was written. Now that you pointed it out, I removed that outdated comment (# (sigarra_id | None, teacher_name))
In the current version, None is not a possible value. When the teacher["name"] does not respect the format <professor_id> - <professor_name>, instead of retrieving the professor_id on the left side of the -, it justs uses the teacher["sigarra_id"] number from the values returned from sigarra api.
To clarify, the teacher["sigarra_id"] field is not really the id that appears on the link of the page of the teacher, which is extremely weird but not up to us since we didn't developed that API.
There was a problem hiding this comment.
But the professor_id is used in the link, right?
There was a problem hiding this comment.
Ok, I just see that there is also a sigarra_url, which redirect to teacher profile. Can we save that? I think it could be usefull
There was a problem hiding this comment.
Yes, the professor_id is used in the link.
We can save that, good point!
There was a problem hiding this comment.
I'm not entirely sure I understood what you suggested
Do you mean retrieving the professor_id from the sigarra_url?
There was a problem hiding this comment.
No, save the sigarra_url in the database as, for example, teacher_link
There was a problem hiding this comment.
I think that maybe it is probably best to do it in a separate issue from this PR, as it is a lower priority issue and this PR should be merged as soon as possible.
Closes #110
Currently the sigarra schedule page has an API link which is present in an attribute of an HTML element, from which we are able to fetch the classes, slots and professors.
Due to this, the following spiders were removed:
class_spiderslot_professor_spiderprofessor_spiderThey were merged into the
slot_spideras to avoid to make requests to the same API link more than one time unecessarily