pyfortified-requests 0.3.2

Extension to Python `requests` functionality.

.. -*- mode: rst -*-

pyfortified-requests
------------------------

**Work In Progress**

Extension of Python HTTP `requests <https://pypi.python.org/pypi/requests>`_ with verbose
logging using `logging-fortified <https://pypi.python.org/pypi/logging-fortified>`_, and robust handling for
Downloading files containing JSON, CSV, and XML data formats and Streaming.

Important Note
^^^^^^^^^^^^^^

This Python project is a refactoring of `requests-mv-integration <https://pypi.org/project/requests-mv-integrations/>`_
for the purpose of general usage intent.

Work In Progress
^^^^^^^^^^^^^^^^

The following still needs to be performed for this Python project:

- Unit-testing: This project will be switching over to using Python native Unit testing framework `unittest <https://docs.python.org/3/library/unittest.html>`_.
- More concise documentation is required.
- Travis CI
- Badges


Badges
------

.. start-badges

.. list-table::
:stub-columns: 1


.. |license| image:: https://img.shields.io/badge/License-MIT-yellow.svg
:alt: License Status
:target: https://opensource.org/licenses/MIT

.. |version| image:: https://img.shields.io/pypi/v/pyfortified_requests.svg?style=flat
:alt: PyPI Package latest release
:target: https://pypi.python.org/pypi/pyfortified_requests

.. |supported-versions| image:: https://img.shields.io/pypi/pyversions/pyfortified-requests.svg?style=flat
:alt: Supported versions
:target: https://pypi.python.org/pypi/pyfortified-requests

.. end-badges


Install
-------

.. code-block:: bash

pip install pyfortified_requests


Architecture
------------

``pyfortified-requests`` is an extension of the `Python package requests <https://pypi.python.org/pypi/requests>`_
and it is used for handling all HTTP requests including APIs in REST and SOAP,
screen scrapping, and handling response downloads in JSON, XML, and CSV.

Usage
-----

.. code-block:: python

URL_REST_COUNTRIES = \
'https://restcountries.eu/rest/v2/all'

from pyfortified_requests import (
RequestsFortifiedDownload,
)
request_download = RequestsFortifiedDownload(logger_level=logging.DEBUG)

result = \
request_download.request(
request_method='GET',
request_url=URL_REST_COUNTRIES,
request_params=None,
request_retry=None,
request_headers=HEADER_CONTENT_TYPE_APP_JSON,
request_label="REST Countries"
)

json_rest_countries = result.json()

pprint(json_rest_countries)


Example
^^^^^^^

.. code-block:: bash

$ python3 examples/example_request.py

======================================================
run-examples pyfortified-requests
======================================================
rm -fR _tmp/*.json
======================================================
'Logger file path: ./tmp/log_1526043820.json'
[
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "NOTE", "name": "pyfortified_requests", "version": "0.1.0", "message": "debug"}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.1.0", "message": "START"}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Start"}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Start: Details", "Disk:": {"path": "/", "total": "232.62 GB", "used": "134.44 GB", "free": "97.94 GB", "percent": 57}, "Mem": {"total": "16.00 GB", "used": "12.50 GB", "free": "147.59 KB", "shared": "0 B", "buffers": "0 B", "cached": "0 B"}, "allow_redirects": true, "build_request_curl": true, "cookie_payload": null, "request_auth": null, "request_cert": null, "request_data": null, "request_headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.1.0, Python/3.6.5)"}, "request_json": null, "request_label": "REST Countries", "request_method": "GET", "request_params": null, "request_retry": {"timeout": 60, "tries": 3, "delay": 10}, "request_url": "https://restcountries.eu/rest/v2/all", "stream": false, "timeout": null, "verify": true}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Start", "request_label": "REST Countries", "request_retry_excps": ["ConnectTimeout", "ReadTimeout", "Timeout"], "request_retry_http_status_codes": [500, 502, 503, 504, 429], "timeout": null}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Attempt", "attempts": 1, "delay": 10, "request_label": "REST Countries", "request_url": "https://restcountries.eu/rest/v2/all", "timeout": 60, "tries": 3}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Session: Details", "cookie_payload": {}, "request_label": "REST Countries"}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Details", "request_data": "", "request_headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.1.0, Python/3.6.5)"}, "request_label": "REST Countries", "request_method": "GET", "request_params": {}, "request_url": "https://restcountries.eu/rest/v2/all", "timeout": 60}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "NOTE", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Curl", "request_curl": "curl --verbose -X GET -H \'Content-Type: application/json\' -H \'User-Agent: (pyfortified-requests/0.1.0, Python/3.6.5)\' --connect-timeout 60 -L \'https://restcountries.eu/rest/v2/all\'", "request_label": "REST Countries", "request_method": "GET"}\n',
'{"asctime": "2018-05-11 06:03:26 -0700", "levelname": "DEBUG", "name": "pyfortified_requests.support.requests_session_client", "version": "0.1.0", "message": "Session Request: Details", "allow_redirects": true, "headers": {"Content-Type": "application/json", "User-Agent": "(pyfortified-requests/0.1.0, Python/3.6.5)"}, "method": "GET", "timeout": 60, "url": "https://restcountries.eu/rest/v2/all", "verify": true}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Response: Details", "http_status_code": 200, "http_status_desc": "Request fulfilled, document follows", "http_status_type": "Successful", "response_headers": {"Date": "Fri, 11 May 2018 13:03:27 GMT", "Content-Type": "application/json;charset=utf-8", "Transfer-Encoding": "chunked", "Connection": "keep-alive", "Set-Cookie": "__cfduid=d46030914d4b01044c167b67bbca43b951526043807; expires=Sat, 11-May-19 13:03:27 GMT; path=/; domain=.restcountries.eu; HttpOnly", "Access-Control-Allow-Origin": "*", "Access-Control-Allow-Methods": "GET", "Access-Control-Allow-Headers": "Accept, X-Requested-With", "Cache-Control": "public, max-age=86400", "Expect-CT": "max-age=604800, report-uri=\\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\\"", "Server": "cloudflare", "CF-RAY": "4194d881fc146bd4-SJC", "Content-Encoding": "gzip"}}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Cookie Payload", "cookie_payload": {"__cfduid": "d46030914d4b01044c167b67bbca43b951526043807"}, "request_label": "REST Countries"}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Try Send Request: Is Return Response: Checking", "request_url": "https://restcountries.eu/rest/v2/all"}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Try Send Request: Is Return Response: Valid", "request_url": "https://restcountries.eu/rest/v2/all"}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Finished", "request_time_msecs": 569}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "DEBUG", "name": "pyfortified_requests", "version": "0.1.0", "message": "REST Countries: Usage", "Disk:": {"path": "/", "total": "232.62 GB", "used": "134.44 GB", "free": "97.94 GB", "percent": 57}, "Mem": {"total": "16.00 GB", "used": "12.50 GB", "free": "145.89 KB", "shared": "0 B", "buffers": "0 B", "cached": "0 B"}}\n',
'{"asctime": "2018-05-11 06:03:27 -0700", "levelname": "INFO", "name": "pyfortified_requests", "version": "0.1.0", "message": "COMPLETED", "connection": "<requests.adapters.HTTPAdapter object at 0x1043b11d0>", "cookies": "<RequestsCookieJar[<Cookie __cfduid=d46030914d4b01044c167b67bbca43b951526043807 for .restcountries.eu/>]>", "elapsed": "0:00:00.286022", "encoding": "utf-8", "headers": "{\'Date\': \'Fri, 11 May 2018 13:03:27 GMT\', \'Content-Type\': \'application/json;charset=utf-8\', \'Transfer-Encoding\': \'chunked\', \'Connection\': \'keep-alive\', \'Set-Cookie\': \'__cfduid=d46030914d4b01044c167b67bbca43b951526043807; expires=Sat, 11-May-19 13:03:27 GMT; path=/; domain=.restcountries.eu; HttpOnly\', \'Access-Control-Allow-Origin\': \'*\', \'Access-Control-Allow-Methods\': \'GET\', \'Access-Control-Allow-Headers\': \'Accept, X-Requested-With\', \'Cache-Control\': \'public, max-age=86400\', \'Expect-CT\': \'max-age=604800, report-uri=\\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\\"\', \'Server\': \'cloudflare\', \'CF-RAY\': \'4194d881fc146bd4-SJC\', \'Content-Encoding\': \'gzip\'}", "history": [], "raw": "<urllib3.response.HTTPResponse object at 0x1043b1dd8>", "reason": "", "request": "<PreparedRequest [GET]>", "status_code": 200, "url": "https://restcountries.eu/rest/v2/all"}\n',
]
'DEBUG'
[
{
'alpha2Code': 'AF',
'alpha3Code': 'AFG',
'altSpellings': ['AF', 'Afġānistān'],
'area': 652230.0,
'borders': ['IRN', 'PAK', 'TKM', 'UZB', 'TJK', 'CHN'],
'callingCodes': ['93'],
'capital': 'Kabul',
'cioc': 'AFG',
'currencies': [
{'code': 'AFN', 'name': 'Afghan afghani', 'symbol': '؋'},
],
'demonym': 'Afghan',
'flag': 'https://restcountries.eu/data/afg.svg',
'gini': 27.8,
'languages': [
{
'iso639_1': 'ps',
'iso639_2': 'pus',
'name': 'Pashto',
'nativeName': 'پښتو',
},
{
'iso639_1': 'uz',
'iso639_2': 'uzb',
'name': 'Uzbek',
'nativeName': 'O\u02bbzbek',
},
{
'iso639_1': 'tk',
'iso639_2': 'tuk',
'name': 'Turkmen',
'nativeName': 'Türkmen',
},
],
'latlng': [33.0, 65.0],
'name': 'Afghanistan',
'nativeName': 'افغانستان',
'numericCode': '004',
'population': 27657145,
'region': 'Asia',
'regionalBlocs': [
{
'acronym': 'SAARC',
'name': 'South Asian Association for Regional Cooperation',
'otherAcronyms': [],
'otherNames': [],
},
],
'subregion': 'Southern Asia',
'timezones': ['UTC+04:30'],
'topLevelDomain': ['.af'],
'translations': {
'br': 'Afeganistão',
'de': 'Afghanistan',
'es': 'Afganistán',
'fa': 'افغانستان',
'fr': 'Afghanistan',
'hr': 'Afganistan',
'it': 'Afghanistan',
'ja': 'アフガニスタン',
'nl': 'Afghanistan',
'pt': 'Afeganistão',
},
},


Classes
-------

- ``class RequestsFortified`` -- Base class using `requests <https://pypi.python.org/pypi/requests>`_ with retry functionality and verbose logging.
- ``class RequestsFortifiedDownload`` -- Download file handling.
- ``class RequestsFortifiedUpload`` -- Upload file handling.

Requirements
------------

``pyfortified-requests`` module is built upon Python 3 and has dependencies upon
several Python modules available within `Python Package Index PyPI <https://pypi.python.org/pypi>`_.

.. code-block:: bash

make install

or

.. code-block:: bash

python3 -m pip uninstall --yes --no-input -r requirements.txt
python3 -m pip install --upgrade -r requirements.txt


Packages
^^^^^^^^

- **beautifulsoup4**: https://pypi.python.org/pypi/beautifulsoup4
- **deepdiff**: https://pypi.python.org/pypi/deepdiff
- **logging-fortified**: ***TBD***
- **pyhttpstatus-utils**: https://pypi.python.org/pypi/pyhttpstatus-utils
- **requests**: https://pypi.python.org/pypi/requests
- **safe-cast**: https://pypi.python.org/pypi/safe-cast