Skip to content

Python project that uses Beautiful Soup to scrape tech news and articles, storing them in a MongoDB database.

Notifications You must be signed in to change notification settings

garciaagui/tech-news

Repository files navigation

Tech News 📰

🇧🇷 Clique aqui para acessar a versão em português.

Summary

  1. Description
  2. Technologies
  3. Features
  4. How to Run
  5. About Trybe
  6. Contact

Description

31st project of the Trybe Web Development course.

Tech News is a Python project that uses Beautiful Soup to perform web scraping on Trybe's blog and fetch tech news and articles, storing them in a MongoDB database.

ℹ️ I wrote tests for the functions implemented by Trybe. These tests can be found in the tests subdirectory.


Technologies

The project was developed in Python and tested with Pytest. The database used was MongoDB and the connection and interaction to it was established through the PyMongo library. Flake8 ensured code quality while Beautiful Soup performed web scraping.

Python MongoDB PyMongo Beautiful Soup Pytest Flake8


Features

  • Collect news and articles from Trybe's blog using web scraping and store them in a MongoDB database.
  • Search news by title.
  • Search news by date.
  • Browse news by category.
  • List the top 5 categories with the most news.

How to Run

To run the project locally, follow the steps below.

  1. Clone the repository.
git clone [email protected]:garciaagui/tech-news.git
  1. Navigate to the root of the project.
cd tech-news/
  1. Create the virtual environment.
python3 -m venv .venv
  1. Activate the virtual environment.
source .venv/bin/activate
  • Note that at the beginning of the terminal line there will be (.venv), as in the example below.
(.venv) gui@gui-desktop:~/Trybe/tech-news$
  • To deactivate the virtual environment, run the command deactivate. Remember to activate it again when you return to the project.
  1. Install dependencies in the virtual environment.
python3 -m pip install -r dev-requirements.txt
  1. If you don't have MongoDB installed locally, launch it via Docker.
docker-compose up -d mongodb
  • ℹ️ Database name is tech_news.
  1. Run the following command to access the menu.
tech-news-analyzer
ℹ️ For additional instructions, click here.
  • 🧪 To run all tests, execute the command below.
python3 -m pytest
  • 🧪 To run only one test file, follow the example below.
python3 -m pytest tests/reading_plan/test_reading_plan.py
  • 🧪 To run only one specific test, follow the example below.
python3 -m pytest -k test_reading_plan_group_news_with_valid_input
  • If you wish to manually test directly in the modules where the functions were implemented, follow the example below.
python3 -m tech_news.scraper.py

About Trybe

"Trybe is a future school for anyone who wants to improve their lives and build a successful career in technology, where the person only pays when they get a good job."

"The program features over 1,500 hours of online classes covering introduction to software development, front-end, back-end, computer science, software engineering, agile methodologies, and behavioral skills."


Contact

Project developed by Guilherme Garcia. Below are my social networks and means of contact. 🤘

Gmail Linkedin GitHub Instagram

Back to top

About

Python project that uses Beautiful Soup to scrape tech news and articles, storing them in a MongoDB database.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published