Skip to content

1Marcuth/deetlist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Deetlist Scraper

The deetlist library is a Web Scraping library that works on the website "https://deetlist.com/dragoncity/" to extract data about Dragon City, such as events, dragons and game releases.


Installation

Using pypi:

pip install deetlist

Using GitHub repository:

pip install git+https://github.com/1marcuth/deetlist.git

Using


Scraping Home page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_home_page()

print(data)

Running the code, you will get something like this:

{'events': {'actives': [{'title': 'Starcrossedlovers Fog Island is now on!', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/fi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/fog/Starcrossedlovers'}, {'title': 'Heroic Race is now on!', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/hr.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/race/'}], 'upcoming': [{'title': 'Star-crossed  Maze Island coming soon', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/mi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/maze/star-crossed-'}, {'title': 'Arctic Tower Island coming soon', 'icon_url': 
'https://www.deetlist.com/dragoncity/img/event/ti.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/tower/Arctic'}, {'title': 'Puzzle Island coming soon', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/pi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/puzzle/star-crossed-'}]}}

Scraping Events page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_events_page()

print(data)

Running the code, you will get something like this:

{'events': {'actives': [{'title': 'Starcrossedlovers Fog Island is now on!', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/fi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/fog/Starcrossedlovers'}, {'title': 'Heroic Race is now on!', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/hr.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/race/'}], 'upcoming': [{'title': 'Star-crossed  Maze Island coming soon', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/mi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/maze/star-crossed-'}, {'title': 'Arctic Tower Island coming soon', 'icon_url': 
'https://www.deetlist.com/dragoncity/img/event/ti.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/tower/Arctic'}, {'title': 'Puzzle Island coming soon', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/pi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/puzzle/star-crossed-'}], 'past': [{'title': 'Runner Island has finished', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/ri.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/runner/Aquatic'}, {'title': 'Sweetrevenge Grid Island has finished', 'icon_url': 'https://www.deetlist.com/dragoncity/img/event/gi.png', 'page_url': 'https://www.deetlist.com/dragoncity/events/grid/Sweetrevenge'}]}}

Scraping All Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_all_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'Nature Dragon', 'page_url': 'https://deetlist.com/dragoncity/dragon/Nature', 'image_url': 'https://deetlist.com/dragoncity/img/dragon/nature.png'}, {'name': 'Firebird Dragon', 'page_url': 'https://deetlist.com/dragoncity/dragon/Firebird', 'image_url': 'https://deetlist.com/dragoncity/img/dragon/firebird.png'}, {'name': 'Mercury Dragon', 'page_url': 'https://deetlist.com/dragoncity/dragon/Mercury', 'image_url': 'https://deetlist.com/dragoncity/img/dragon/mercury.png'}, {'name': 'Gummy Dragon', 'page_url': 'https://deetlist.com/dragoncity/dragon/Gummy', 'image_url': 'https://deetlist.com/dragoncity/img/dragon/gummy.png'}, {'name': 'Lantern Fish Dragon', 'page_url': 'https://deetlist.com/dragoncity/dragon/Lantern%20Fish', 'image_url': 'https://deetlist.com/dragoncity/img/dragon/lantern%20fish.png'}, {'name': 'Tropical Dragon', 'page_url': 'https://deetlist.com/dragonc
...}

See complete example in json file


Scraping New Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_new_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'Larimar Dragon', 'rarity': 'L', 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/larimar.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/larimar', 'released': 1707423}, {'name': 'Lotus Dragon', 'rarity': 'L', 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/lotus.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/lotus', 'released': 1707423}, {'name': 'Voyager Dragon', 'rarity': 'L', 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/voyager.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/voyager', 'released': 
1707423}, {'name': 'High Passion Dragon', 'rarity': 'H', 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high_passion.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/high_passion', 'released': 1707423}, {'name': 'Gigaguard Dragon', 'rarity': 'L', 'image_url': '
...}

See complete example in json file


Scraping Heroic Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_all_heroic_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'High Fenrir Dragon', 'rarity': 'H', 'elements': ['ice', 'pure', 'war', 'electric'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20fenrir.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/high_fenrir'}, {'name': 'High Nucleus Dragon', 'rarity': 'H', 'elements': ['pure', 'metal', 'terra', 'nature'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20nucleus.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/high_nucleus'}, {'name': 'High Tension Dragon', 'rarity': 'H', 'elements': ['electric', 'war', 'metal', 'dark'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20tension.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/high_tension'}, {'name': 'High Guardian Dragon', 'rarity': 'H', 'elements': ['pure', 'ice', 'electric', 'dark'], 'image_url': 'https://www.deetlist.com/dra
...}

See complete example in json file


Scraping Breedable Legendary Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_breedable_legendary_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'Apocalypse Dragon', 'rarity': 'L', 'elements': ['flame', 'ice', 'light', 'dark'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/apocalypse.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/apocalypse'}, {'name': 'Millennium Dragon', 'rarity': 'L', 'elements': ['sea', 'war', 'terra', 'metal'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/millennium.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/millennium'}, {'name': 'Forge Dragon', 'rarity': 'L', 'elements': ['metal', 'terra', 'ice', 'flame'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/forge.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/forge'}, {'name': 'Promethium Dragon', 'rarity': 'L', 'elements': ['light', 'dark', 'nature', 'electric'], 
'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/promethium.png', 'page_
...}

See complete example in json file


Scraping Breedable Category 5 Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_breedable_category_5_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'Dujur Dragon', 'rarity': 'V', 'elements': ['flame', 'electric', 'ice'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/dujur.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/dujur'}, {'name': 'Master Dragon', 'rarity': 'E', 'elements': ['flame', 'sea', 'war'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/master.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/master'}, {'name': 'Greenfluid Dragon', 'rarity': 'E', 'elements': ['nature', 'sea', 'electric'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/greenfluid.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/greenfluid'}, {'name': 'Sylvan Dragon', 'rarity': 'E', 'elements': ['nature', 'war'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/sylvan.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/sylvan'}, {
...}

See complete example in json file


Scraping Breedable Category 9 Dragons page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_breedable_category_9_dragons_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'Decay Dragon', 'rarity': 'L', 'elements': ['dark', 'terra', 'primal', 'metal'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/decay.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/decay'}, {'name': 'Quasar Dragon', 'rarity': 'L', 'elements': ['soul', 'electric', 'light', 'pure'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/quasar.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/quasar'}, {'name': 'Shapeshifter Dragon', 'rarity': 'L', 'elements': ['chaos', 'sea', 'dark', 'flame'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/shapeshifter.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/shapeshifter'}, {'name': 'Morpheus Dragon', 'rarity': 'L', 'elements': ['dream', 'pure', 'ice', 'sea'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/morpheus.png', 'page_url': 'http
...}

See complete example in json file


Scraping Heroic Dragons Categorized by Attacks page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_heroic_dragons_categorized_by_attacks_page()

print(data)

Running the code, you will get something like this:

{'dragons': [{'name': 'High Reborn Dragon', 'rarity': 'H', 'elements': ['pure', 'nature', 'light', 'sea'], 'attack': {'power': 0, 'rank': 1}, 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20reborn.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/High_Reborn'}, {'name': 'High Virago Dragon', 'rarity': 'H', 'elements': ['light', 'war', 'nature', 'electric'], 'attack': {'power': 0, 'rank': 2}, 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20virago.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/High_Virago'}, {'name': 'High Arcane Dragon', 'rarity': 'H', 'elements': ['magic', 'pure', 'electric', 'flame'], 'attack': {'power': 0, 'rank': 3}, 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20arcane.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/High_Arcane'}, {'name': 'High Frozen Dragon', 'rari
...}

See complete example in json file


Scraping Elements page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_elements_page()

print(data)

Running the code, you will get something like this:

{'elements': [{'name': 'terra', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/terra.png'}, {'name': 'flame', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/flame.png'}, {'name': 'sea', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/sea.png'}, {'name': 'nature', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/nature.png'}, {'name': 'electric', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/electric.png'}, {'name': 'ice', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/ice.png'}, {'name': 'metal', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/metal.png'}, {'name': 'dark', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/dark.png'}, {'name': 'light', 'image_url': 'https://www.deetlist.com/dragoncity/img/types/light.png'}, {'name': 'war', 'image_url': 'https://www.deetlist.com/dragoncity/img
...}

See complete example in json file


Scraping Element Tokens page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_element_tokens_page()

print(data)

Running the code, you will get something like this:

{'element_tokens': [{'name': 'terra', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Terra_Token.png'}, {'name': 'flame', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Flame_Token.png'}, {'name': 'sea', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Sea_Token.png'}, {'name': 'nature', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Nature_Token.png'}, {'name': 'electric', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Electric_Token.png'}, {'name': 'ice', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Ice_Token.png'}, {'name': 'metal', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Metal_Token.png'}, {'name': 'dark', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Dark_Token.png'}, {'name': 'light', 'image_url': 'https://www.deetlist.com/dragoncity/img/tokens/Light_Token.png'},
...}

See complete example in json file


Scraping Fog Islands page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_fog_islands_page()

print(data)

Running the code, you will get something like this:

{'duration': 259200, 'pool': {'size': 666, 'time': 28800}, 'dragons': [{'name': 'Equinox Dragon', 'rarity': 'C', 'elements': ['dark', 'light'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/equinox.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/equinox'}, {'name': 'Fairy Dragon', 'rarity': 'R', 'elements': ['nature', 'ice'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/fairy.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/fairy'}, {'name': 'Vibrant Dragon', 'rarity': 'V', 'elements': ['electric', 'sea', 'flame'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/vibrant.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/vibrant'}, {'name': 'Side-t Dragon', 'rarity': 'E', 'elements': ['terra', 'sea', 'pure'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/side-t.png', 'page_url': 'https://www.d
...}

See complete example in json file


Scraping Heroic Races page data:

from deetlist import Deetlist

deetlist = Deetlist()

data = deetlist.scrape_heroic_races_page()

print(data)

Running the code, you will get something like this:

{'duration': 950400, 'dragons': [{'name': 'High Passion Dragon', 'rarity': 'H', 'elements': ['beauty', 'flame', 'electric', 'chaos'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/high%20passion.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/high_passion'}, {'name': 'Magicienne Dragon', 'rarity': 'L', 'elements': ['magic', 'dark', 'sea', 'nature'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/magicienne.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/magicienne'}, {'name': 'Rideable Dragon', 'rarity': 'E', 'elements': ['metal', 'ice', 'electric'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/rideable.png', 'page_url': 'https://www.deetlist.com/dragoncity/dragon/rideable'}, {'name': 'Wave Dragon', 'rarity': 'V', 'elements': ['sea', 'electric', 'light'], 'image_url': 'https://www.deetlist.com/dragoncity/img/dragon/
...}

See complete example in json file


Possible problems

  • Some URLs of dragon images or pages may vary their word separators between _ and %20, as I was unable to find a standard to correct this, it will be your role to carry out this validation when getting the image from the URL: If it doesn't work with one tab, try the other, but please be careful not to overload the Deetlist server!

How to Contribute

If you want to contribute to this project, please follow these steps:

  1. Fork this repository.
  2. Create a branch for your feature (git checkout -b feature/MyFeature).
  3. Commit your changes (git commit -am 'Add new feature').
  4. Push to the branch (git push origin feature/MyFeature).
  5. Create a new Pull Request.

Author

Marcuth @1marcuth

License

This project is licensed under the MIT License.