Skip to content

분석하고 싶은 데이터를 수집하고 가공해 설계한 데이터베이스에 저장 후 엘라스틱 서치로 시각화 한 프로젝트입니다.

Notifications You must be signed in to change notification settings

StatPage/data-management-project

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 
 
 

Repository files navigation

Charter


Objectives and project purpose


Objective

Active participation of Danngn Market users in trading.

  • In various regions of Korea, many users are using the Dangn Market app to trade second-hand.
  • But many people have a hard time deciding which of my used products to sell.
  • Because they doubt who will buy the things they use.
  • Also, there are many people who want to buy what they want outside the scope of certification of Dangn Market.
  • We will provide data on which item categories people in the area are most interested in through data.
  • Such a service can induce users to participate in a transaction with their used goods, and can also increase the participation rate of those who are far away.

Project scope

Collect

I will crawl the data from Daangn Market website by using Python. The names of modules are requests and BeautifulSoup of bs4.


Process

All files imported through crawling are saved as CSV file and JSON file. Especially I made some modules for this project. So it doesn't need to import other special packages.


Store

CSV and JSON files are sent to and stored in databases of SQLite and Elasticsearch.

  • JSON schema description for NoSQL
All_Data
{
  local_name: text,
  Item : text,
  Like : number,
  Chat : number,
  Date : date
}
Local_Data
{
  local_name: text,
  Item_name : text,
  Like : number,
  Chat : number,
  Date : date
}
Column Family Name local_name
All_Data
Item Chat
Count Like
Local_Data
Item_name Chat
Like Date

Query

After data is properly stored in the database, all data are accessed and processed through query statements.

About

분석하고 싶은 데이터를 수집하고 가공해 설계한 데이터베이스에 저장 후 엘라스틱 서치로 시각화 한 프로젝트입니다.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages