Skip to content

Latest commit

 

History

History
168 lines (105 loc) · 4.66 KB

README.rst

File metadata and controls

168 lines (105 loc) · 4.66 KB
Raw

ProxyBroker

https://img.shields.io/pypi/v/proxybroker.svg?style=flat-square https://img.shields.io/travis/constverum/ProxyBroker.svg?style=flat-square https://img.shields.io/pypi/wheel/proxybroker.svg?style=flat-square https://img.shields.io/pypi/pyversions/proxybroker.svg?style=flat-square https://img.shields.io/pypi/l/proxybroker.svg?style=flat-square

ProxyBroker is an open source tool that asynchronously finds public proxies from multiple sources and concurrently checks them.

https://raw.githubusercontent.com/constverum/ProxyBroker/master/docs/source/_static/index_find_example.gif

Features

  • Finds more than 7000 working proxies from ~50 sources.
  • Support protocols: HTTP(S), SOCKS4/5. Also CONNECT method to ports 80 and 23 (SMTP).
  • Proxies may be filtered by type, anonymity level, response time, country and status in DNSBL.
  • Work as a proxy server that distributes incoming requests to external proxies. With automatic proxy rotation.
  • All proxies are checked to support Cookies and Referer (and POST requests if required).
  • Automatically removes duplicate proxies.
  • Is asynchronous.

Requirements

Installation

To install last stable release from pypi:

$ pip install proxybroker

The latest development version can be installed directly from GitHub:

$ pip install -U git+https://github.com/constverum/ProxyBroker.git

Usage

CLI Examples

Find

Find and show 10 HTTP(S) proxies from United States with the high level of anonymity:

$ proxybroker find --types HTTP HTTPS --lvl High --countries US --strict -l 10

https://raw.githubusercontent.com/constverum/ProxyBroker/master/docs/source/_static/cli_find_example.gif

Grab

Find and save to a file 10 US proxies (without a check):

$ proxybroker grab --countries US --limit 10 --outfile ./proxies.txt

https://raw.githubusercontent.com/constverum/ProxyBroker/master/docs/source/_static/cli_grab_example.gif

Serve

Run a local proxy server that distributes incoming requests to a pool of found HTTP(S) proxies with the high level of anonymity:

$ proxybroker serve --host 127.0.0.1 --port 8888 --types HTTP HTTPS --lvl High

https://raw.githubusercontent.com/constverum/ProxyBroker/master/docs/source/_static/cli_serve_example.gif

Run proxybroker --help for more information on the options available.
Run proxybroker <command> --help for more information on a command.

Basic code example

Find and show 10 working HTTP(S) proxies:

import asyncio
from proxybroker import Broker

async def show(proxies):
    while True:
        proxy = await proxies.get()
        if proxy is None: break
        print('Found proxy: %s' % proxy)

proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
    broker.find(types=['HTTP', 'HTTPS'], limit=10),
    show(proxies))

loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)

More examples.

Documentation

https://proxybroker.readthedocs.io/

TODO

  • Check the ping, response time and speed of data transfer
  • Check site access (Google, Twitter, etc) and even your own custom URL's
  • Information about uptime
  • Checksum of data returned
  • Support for proxy authentication
  • Finding outgoing IP for cascading proxy
  • The ability to specify the address of the proxy without port (try to connect on defaulted ports)

Contributing

  • Fork it: https://github.com/constverum/ProxyBroker/fork
  • Create your feature branch: git checkout -b my-new-feature
  • Commit your changes: git commit -am 'Add some feature'
  • Push to the branch: git push origin my-new-feature
  • Submit a pull request!

License

Licensed under the Apache License, Version 2.0

This product includes GeoLite2 data created by MaxMind, available from http://www.maxmind.com.