feat: job board scraping pipeline + email-sync lambda scaffold - #3
Open
gersondiaz12 wants to merge 8 commits into
Open
feat: job board scraping pipeline + email-sync lambda scaffold#3gersondiaz12 wants to merge 8 commits into
gersondiaz12 wants to merge 8 commits into
Conversation
Adds a new jobsync Lambda (EventBridge, every minute) that scrapes new-grad and internship postings from the SpeedyApply and SimplifyJobs GitHub READMEs and upserts them into a new `jobs` Postgres table, keyed by a stable hash of company+position+link so re-scrapes update in place instead of duplicating rows (and so a future email-sync pass can match a job by that same ID). The api Lambda serves them via GET /jobs, rendered on a new Job Board dashboard card/modal with search and section filtering. Also stages the emailsync Lambda scaffold (Gmail/Outlook OA + interview + rejection detection stub) and its terraform wiring, laying the groundwork to attach application-status updates to jobs on the dashboard.
gersondiaz12
commented
Aug 10, 2026
| db, err := openDB(ctx) // connects to Postgres using PG_DSN | ||
| if err != nil { | ||
| return fmt.Errorf("open db: %w", err) // if it fails, return an error to Lambda so it can retry later | ||
| } |
Collaborator
Author
There was a problem hiding this comment.
Use the search bar for finding specific emails related to job applications.
GET /jobs now returns pages via a cursor (the last row's first_seen_at + id) instead of a flat LIMIT. jobsync writes to the jobs table every minute, so plain OFFSET pagination would let a newly-inserted row shift every later page by one slot, skipping or duplicating jobs between loads - anchoring to an actual row instead of a row count avoids that. JobBoardModal gets a Load more button that fetches and appends pages; JobBoardCard now asks the API for exactly 5 rows instead of fetching everything and slicing client-side.
Reverses the jobs table/DB approach in favor of one JSON file in S3 that jobsync overwrites every minute and the api Lambda reads on every GET /jobs request. Pagination changes from SQL keyset cursors to plain offset slicing over the in-memory list, since a single S3 read is already a stable snapshot for the life of one request - unlike the Postgres table, which was being written to every minute. jobsync no longer touches Postgres at all; adds a jobscache S3 module and S3 IAM support in the lambda module.
Fixes alignment in the scheduler module block - this is what terraform fmt -check -recursive was failing CI on.
Adds src/lib/jobsCache.ts, a localStorage cache (2 min TTL) in front of GET /jobs, so reopening the modal or the card re-mounting within a session doesn't always refetch. JobBoardCard and JobBoardModal both route through it now, and each gets its own bounded, independently-scrolling job list - matching the h-72 scroll-box pattern CloudCard/NetworkingCard/GenAICard already use - instead of growing the whole card/modal as more jobs load. Also adds scripts/mock-jobs-server.mjs, a local-only fake API server for testing this without a real deployed backend.
sah-rohan
reviewed
Aug 14, 2026
Reverses d6f573b. The job data is derived from two public GitHub READMEs rather than owned by us, so it can be re-fetched at any moment - which makes any server-side copy a cache rather than a record, and this feature already had a cache in the browser (localStorage, src/lib/jobsCache.ts). GET /jobs now calls the scrapers directly and keeps the result in the api Lambda's process memory for 5 minutes. Package-level vars survive between invocations on a warm container, so this needs no storage service at all. The browser's 2-minute localStorage cache sits in front of it. Removes the jobsync Lambda and its every-minute EventBridge schedule, the jobs S3 bucket and its Terraform module, the S3 IAM wiring in the lambda module, and jobs.ReadCache/WriteCache/CachePayload. The api Lambda gains GITHUB_TOKEN via SSM, since it is now the one talking to GitHub. Two behavior changes worth knowing: the first request against a cold Lambda container now scrapes inline (~1-3s) instead of doing one fast S3 read, and a total scrape failure returns 500 rather than an empty 200. Note: applying the Terraform destroys the jobs bucket and jobsync Lambda. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
newID hashed only lightly-normalized fields, so a trailing slash or a utm_ parameter produced a different ID for the same job. Worse, nothing ever removed duplicates: identical IDs still rendered as separate rows, with duplicate React keys. Identity is now URL-first. When a posting has a link, its canonical URL alone decides identity, which sidesteps title wording entirely - no fuzzy matching anywhere. Linkless rows (SimplifyJobs renders closed applications as a lock emoji) fall back to company + position + location. Location is new to that key and fixes a live bug where every closed role sharing a company and title merged into one entry regardless of city. canonical.go normalizes URLs (scheme, host, www, trailing slash, fragment, tracking parameters, sorted query), company names (Unicode folding, punctuation, legal suffixes) and titles (deliberately minimal - no SWE/Software Engineer expansion, since that risks hiding real postings). A Workday rule handles the largest real cluster: one requisition is published through several career sites and gains a -1/-2/-3 disambiguator, so both the site segment and that suffix are dropped. Verified against live data - collapses Cadence 4 to 1 and 3 to 1, and Salesforce 2 to 1, while leaving RTX's three genuinely distinct requisitions apart. dedupe.go merges records sharing an ID in first-seen order. The ordering is deliberate: Go randomizes map iteration, and GET /jobs pages with plain offsets, so a list reshuffled between scrapes would make readers skip and repeat jobs across pages. Merging fills blanks from duplicates, and a posting counts as closed only when every source agrees - biasing toward showing jobs rather than hiding them. Live result: 303 scraped, 297 after dedupe; human-visible duplicates down from 15 rows to 9. The remaining 9 are distinct requisition IDs within one ATS, which no URL rule can resolve. Adds the repo's first tests - 108 subtests, including must-not-merge cases that guard against over-merging.
Consolidates the per-scraper const blocks into sources.go so one file answers 'what do we scrape?'. Names unchanged, no behavior change. Also adds per-section counts to scrapetest, which is how a silently-renamed upstream heading would show up.
sah-rohan
reviewed
Aug 19, 2026
Comment on lines
+132
to
+142
| module "emailsync" { | ||
| source = "./modules/lambda" | ||
| name = "${var.project}-emailsync" | ||
| zip_path = var.emailsync_zip | ||
| ssm_parameter_arns = local.secret_arns | ||
| environment = { | ||
| DATABASE_URL_SSM = module.ssm.name | ||
| LEETCODE_SESSION_SSM = data.aws_ssm_parameter.leetcode_session.name | ||
| } | ||
| } | ||
|
|
Owner
There was a problem hiding this comment.
I do not believe there is a reason to start making microservices. Instead let's add the end points to the already made api lambda. Just update that
Comment on lines
150
to
159
| module "scheduler" { | ||
| source = "./modules/scheduler" | ||
| name = var.project | ||
| sync_function_arn = module.sync.arn | ||
| sync_function_name = module.sync.function_name | ||
| enrich_function_arn = module.enrich.arn | ||
| enrich_function_name = module.enrich.function_name | ||
| source = "./modules/scheduler" | ||
| name = var.project | ||
| sync_function_arn = module.sync.arn | ||
| sync_function_name = module.sync.function_name | ||
| enrich_function_arn = module.enrich.arn | ||
| enrich_function_name = module.enrich.function_name | ||
| emailsync_function_arn = module.emailsync.arn | ||
| emailsync_function_name = module.emailsync.function_name | ||
| } |
Owner
There was a problem hiding this comment.
yeah so again do not have the email sync lambda here because microservices do not make sense for an application of this scale. Research lambda cold starts
Comment on lines
+2
to
+76
| import { Briefcase } from "lucide-react"; | ||
| import { Card } from "../components/Card"; | ||
| import { useData } from "../data/source"; | ||
| import { type ApiJob } from "../lib/api"; | ||
| import { fetchJobsPage } from "../lib/jobsCache"; | ||
|
|
||
| // How many jobs to show in the card's own scrollable preview - bigger than | ||
| // a plain top-5 slice (matching CloudCard/NetworkingCard/GenAICard's | ||
| // pattern of a scrollable list right on the dashboard tile, before you even | ||
| // open the full modal), but still well short of the modal's full paginated | ||
| // list. | ||
| const PREVIEW_COUNT = 20; | ||
|
|
||
| // JobBoardCard is the dashboard tile for the Job Board feature. It shows a | ||
| // scrollable preview of new-grad/internship postings that our GET /jobs | ||
| // route scrapes from public GitHub job-list repos on demand - this | ||
| // component never talks to GitHub itself, only to our own API. | ||
| // Clicking the card opens JobBoardModal for the full, searchable list. | ||
| export function JobBoardCard({ onOpen }: { onOpen: () => void }) { | ||
| const { getToken } = useData(); | ||
| const [jobs, setJobs] = useState<ApiJob[]>([]); | ||
| const [loading, setLoading] = useState(true); | ||
|
|
||
| useEffect(() => { | ||
| fetchJobsPage(getToken, PREVIEW_COUNT, 0) | ||
| .then((page) => setJobs(page.jobs ?? [])) | ||
| .catch(() => setJobs([])) | ||
| .finally(() => setLoading(false)); | ||
| }, [getToken]); | ||
|
|
||
| return ( | ||
| <Card className="lg:col-span-1 h-full" onClick={onOpen}> | ||
| <div className="flex items-center justify-between"> | ||
| <div className="text-[15px] font-medium">Job Board</div> | ||
| <Briefcase className="h-4 w-4 text-coral" /> | ||
| </div> | ||
| <p className="mt-1 text-xs text-muted-foreground"> | ||
| New grad & internship roles, scraped from GitHub every minute. | ||
| </p> | ||
| <div className="mt-3 flex items-center justify-between text-[11px] font-medium uppercase tracking-wide text-muted-foreground"> | ||
| <span>Job Board</span> | ||
| <span>{jobs.length} shown</span> | ||
| </div> | ||
| <ul className="modal-scroll mt-3 h-72 space-y-2 overflow-y-auto pr-1"> | ||
| {loading && ( | ||
| <li className="rounded-2xl border border-dashed border-border px-4 py-8 text-center text-sm text-muted-foreground"> | ||
| Loading… | ||
| </li> | ||
| )} | ||
| {!loading && jobs.length === 0 && ( | ||
| <li className="rounded-2xl border border-dashed border-border px-4 py-8 text-center text-sm text-muted-foreground"> | ||
| No jobs yet — check back soon. | ||
| </li> | ||
| )} | ||
| {jobs.map((j) => ( | ||
| <li | ||
| key={j.id} | ||
| className="flex items-center gap-3 rounded-2xl border border-border px-4 py-3" | ||
| > | ||
| <div className="min-w-0 flex-1"> | ||
| <div className="truncate text-sm font-medium"> | ||
| {j.company} — {j.position} | ||
| </div> | ||
| <div className="truncate text-[11px] text-muted-foreground"> | ||
| {j.location} | ||
| </div> | ||
| </div> | ||
| {j.age && ( | ||
| <span className="shrink-0 rounded-full bg-sky px-2.5 py-1 text-[11px] font-medium text-sky-foreground"> | ||
| {j.age} | ||
| </span> | ||
| )} | ||
| </li> | ||
| ))} | ||
| </ul> |
Owner
There was a problem hiding this comment.
send me a picture of how the UI for this component looks please in the GitHub pr description
Owner
There was a problem hiding this comment.
I need to see a picture of this component in the GitHub pr description
Comment on lines
+236
to
+275
| func classifyEmail(email EmailMessage) string { | ||
| text := strings.ToLower(email.Subject + " " + email.Body) | ||
|
|
||
| if strings.Contains(text, "online assessment") || strings.Contains(text, "oa") { | ||
| return StatusOaReceived | ||
| } | ||
| if strings.Contains(text, "interview") || strings.Contains(text, "panel") { | ||
| return StatusInterviewInvite | ||
| } | ||
| if strings.Contains(text, "rejected") || strings.Contains(text, "regret") || strings.Contains(text, "not selected") { | ||
| return StatusRejection | ||
| } | ||
| return StatusOther | ||
| } | ||
|
|
||
| // extractCompany uses a simple heuristic to guess the company name. | ||
| // It checks the sender address and the subject line. | ||
| func extractCompany(email EmailMessage) string { | ||
| cleanedFrom := strings.TrimSpace(strings.Split(email.From, "<")[0]) | ||
| if cleanedFrom != "" && cleanedFrom != email.From { | ||
| return cleanedFrom | ||
| } | ||
|
|
||
| subject := strings.ToLower(email.Subject) | ||
| patterns := []string{" at ", " from ", " for "} | ||
| for _, pat := range patterns { | ||
| if idx := strings.Index(subject, pat); idx != -1 { | ||
| candidate := strings.TrimSpace(email.Subject[idx+len(pat):]) | ||
| if candidate != "" { | ||
| return candidate | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Fallback to the raw sender or subject when we cannot parse a company. | ||
| if email.From != "" { | ||
| return email.From | ||
| } | ||
| return email.Subject | ||
| } |
Owner
There was a problem hiding this comment.
same thing please test and write clean logic that is well tested with a bunch of your emails regarding job applications. I would also like to say that if you have the trend/patterns of emails changed in future recruiting what will you do. Please review this very crucial logic
Comment on lines
+277
to
+296
| var datePattern = regexp.MustCompile(`(?i)(\b\d{1,2}/\d{1,2}/\d{2,4}\b)|(\b\d{4}-\d{1,2}-\d{1,2}\b)`) // MM/DD/YYYY or YYYY-MM-DD | ||
|
|
||
| // extractDueDate looks for a simple date pattern in the email text. | ||
| // It returns sql.NullTime so we can store NULL in the database when no date is found. | ||
| func extractDueDate(email EmailMessage) (sql.NullTime, error) { | ||
| text := email.Subject + " " + email.Body | ||
| match := datePattern.FindString(text) | ||
| if match == "" { | ||
| return sql.NullTime{Valid: false}, nil | ||
| } | ||
|
|
||
| layouts := []string{"1/2/2006", "01/02/2006", "2006-01-02"} | ||
| for _, layout := range layouts { | ||
| if due, err := time.Parse(layout, match); err == nil { | ||
| return sql.NullTime{Time: due, Valid: true}, nil | ||
| } | ||
| } | ||
|
|
||
| return sql.NullTime{Valid: false}, nil | ||
| } |
Owner
There was a problem hiding this comment.
yeah this is not what it usually is. The date pattern is usually like next two weeks. REWRITE!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new jobsync Lambda (EventBridge, every minute) that scrapes new-grad and internship postings from the SpeedyApply and SimplifyJobs GitHub READMEs and upserts them into a new
jobsPostgres table, keyed by a stable hash of company+position+link so re-scrapes update in place instead of duplicating rows (and so a future email-sync pass can match a job by that same ID). The api Lambda serves them via GET /jobs, rendered on a new Job Board dashboard card/modal with search and section filtering.Also stages the emailsync Lambda scaffold (Gmail/Outlook OA + interview + rejection detection stub) and its terraform wiring, laying the groundwork to attach application-status updates to jobs on the dashboard.