proxyscrape 0.1.0

Collectors

Collectors serve as the interface to retrieving proxies. They are instantiating at module-level and can be retrieved and re-used in different parts of the application (similar to the Python logging library). Collectors can be created and retrieved via the create_collector(…) and get_collector(…) functions.

from proxyscrape import create_collector, get_collector

collector = create_collector('my-collector', ['socks4', 'socks5'])

# Some other section of code
collector = get_collector('my-collector')

Each collector should have a unique name and be initialized only once. Typically, only a single collector of a given resource type should be utilized. Filters can then be applied to the proxies if specific criteria is desired.

When given one or more resources, the collector will use those to retrieve proxies. If one or more resource types are given, the resources for each of the types will be used to retrieve proxies.

Once created, proxies can be retrieved via the get_proxy(…) function. This optionally takes a filter_opts parameter which can filter by the following: - code (us, ca, …) - country (united states, canada, …) - anonymous (True, False) - type (http, https, socks4, socks5, …)

from proxyscrape import create_collector

collector = create_collector('my-collector', 'http')

# Retrieve any http proxy
proxy = collector.get_proxy()

# Retrieve only 'us' proxies
proxy = collector.get_proxy({'code': 'us'})

# Retrieve only anonymous 'uk' or 'us' proxies
proxy = collector.get_proxy({'code': ('us', 'uk'), 'anonymous': True})

Filters can be applied to every proxy retrieval from the collector via apply_filter(…). This is useful when the same filter is expected for any proxy retrieved.

from proxyscrape import create_collector

collector = create_collector('my-collector', 'http')

# Only retrieve 'uk' and 'us' proxies
collector.apply_filter({'code': 'us'})

# Filtered proxies
proxy = collector.get_proxy()

# Clear filter
collector.clear_filter()

Note that some filters may instead use specific resources to achieve the same results (i.e. ‘us-proxy’ or ‘uk-proxy’ for ‘us’ and ‘uk’ proxies).

Blacklists can be applied to a collector to prevent specific proxies from being retrieved. They accept one or more Proxy objects and won’t allow retrieval of matching proxies.

from proxyscrape import create_collector

collector = create_collector('my-collector', 'http')

# Add proxy to blacklist
collector.blacklist_proxy(Proxy('192.168.1.1', '80', None, None, None, 'http', 'my-resource'))

# Blacklisted proxies won't be included
proxy = get_proxy()

# Clear blacklist
collector.clear_blacklist()

Instead of permanently blacklisting a particular proxies, a proxy can instead be removed from internal memory. This allows it to be re-added to the pool upon a subsequent refresh.

from proxyscrape import create_collector

collector = create_collector('my-collector', 'http')

# Remove proxy from internal pool
collector.remove_proxy(Proxy('192.168.1.1', '80', None, None, 'http', 'my-resource'))

Apart from automatic refreshes when retrieving proxies, they can also be forcefully refreshed via the refresh_proxies(…) function.

from proxyscrape import create_collector

collector = create_collector('my-collector', 'http')

# Forcefully refresh
collector.refresh_proxies(force=True)

# Refresh only if proxies not refreshed within `refresh_interval`
collector.refresh_proxies(force=False)

Resources

Resources refer to a specific function that retrieves a set of proxies; the currently implemented proxies are all retrieves from scraping a particular web site.

Additional user-defined resources can be added to the pool of proxy retrieval functions via the add_resource(…) function. Resources can belong to multiple resource types.

from proxyscrape import add_resource

def func():
    return {Proxy('192.168.1.1', '80', 'us', 'united states', False, 'http', 'my-resource'), }

add_resource('my-resource', func, 'http')

As shown above, a resource doesn’t necessarily have to scrape proxies from a web site. It can be return a hard-coded list of proxies, make a call to an api, read from a file, etc.

The set of library- and user-defined resources can be retrieved via the get_resources(…) function.

from proxyscrape import get_resources
resources = get_resources()

Resource Types

Resource types are groupings of resources that can be specified when defining a collector (opposed to giving a collection of resources.

Additional user-defined resource types can be added via the add_resource_type(…) function. Resources can optionally be added to a resource type when defining it.

from proxyscrape import add_resource_type
add_resource_type('my-resource-type')
add_resource_type('my-other-resource-type', 'my-resource')  # Define resources for resource type

The set of library- and user-defined resource types can be retrieved via the get_resource_types(…) function.

from proxyscrape import get_resource_types
resources = get_resource_types()