macedon 0.13.0

CLI web service availability verifier

Multi-threaded CLI web service availability verifier. Takes a list of endpoints with optional input dataset, performs series of HTTP requests and displays the results.

Motivation

Necessity to have a fast and configurable endpoint testing tool at fingertips.

Installation

With pipx

$ pipx install macedon

Manually

$ git clone https://github.com/es7s/macedon.git
$ cd macedon
$ python -m venv venv
$ ./venv/bin/pip install .
$ ln -s $(pwd)/run ~/.local/bin/macedon

Basics

In the following example we are telling the application to make 4 sequential requests to 192.168.1.2 on port 80 (GET / by default); the number of threads is determined automatically by the number of logical cores of host CPU (-T option overrides this number).

Options

Usage:

   macedon [OPTIONS] [ENDPOINT_URL]...

Options:
  -T, --threads INTEGER         Number of threads for concurrent request making. Default value depends on number of
                                CPU cores available in the system.  [default: 6]
  -n, --amount INTEGER          How many times each request will be performed.  [default: 1]
  -d, --delay FLOAT             Seconds to wait between requests.  [default: 0]
  -t, --timeout FLOAT           Seconds to wait for the response.  [default: 10]
  -i, --insecure                Ignore invalid/expired certificates when performing HTTPS requests.
  -f, --file FILENAME           Execute request(s) from a specified file, or from stdin, if FILENAME is specified as
                                '-'. The file should contain a list of endpoints in the format '{method} {url}', one
                                per line. Another (partially) supported format is JetBrains HTTP Client format (see
                                below), which additionally allows to specify request headers and/or body. The option
                                can be specified multiple times. Note that ENDPOINT_URL argument(s) are ignored if
                                this option is present.
  -x, --exit-code               Return different exit codes depending on completed / failed requests. With this option
                                exit code 0 is returned if and only if each request was considered successful (1xx,
                                2xx HTTP codes); even one failed request (4xx, timed out, etc) will result in a non-
                                zero exit code. (Normally the exit code 0 is returned as long as the application
                                terminated under normal conditions, regardless of an actual HTTP codes; but it can
                                still die with a non-zero code upon invalid option syntax, etc).
  -c, --color / -C, --no-color  Force output colorizing using ANSI escape sequences or disable it unconditionally. If
                                omitted, the application determines it automatically by checking if the output device
                                is a terminal emulator with SGR support.
  --show-id                     Print a column with request serial number.
  --show-error                  Print a column with network (not HTTP) error messages, when applicable.
  -v, --verbose                 Increase verbosity:
                                    -v for request details and exceptions;
                                   -vv for request/response contents and headers;
                                  -vvv for exception stack traces and thread state transitions.
  -V, --version                 Show the version and exit. Specify twice (-VV) to see interpreter and entrypoint
                                paths. If stdout is not a terminal, print only app version number without labels or
                                timestamps.
  --help                        Show this message and exit.

Headers, body, authorization

Request metadata can be specified using JetBrains HTTP syntax (note that support is very limited) using either:

1. a helper data file (see example.http):

   $ macedon -T1 -vv -f req1.http
   

2. bash and stdin (all three commands are equivalent):

   $ macedon -f - <<<$'GET http://2ip.ru\nUser-Agent: curl/7.68.0'
   
   $ macedon -f - <<<'GET http://2ip.ru
   User-Agent: curl/7.68.0'
   
   $ echo -e 'GET http://2ip.ru \n User-Agent: curl/7.68.0' | macedon -f -
   

HTTP Syntax

Supported features include:

• Method (GET/POST/etc)
• Request headers
• Request body
• # comments

General syntax:

# Request name / description (optional)
Method Request-URI
Header-field: Header-value

Request-Body

Proxy configuration

The application is based on Python requests library that manages all of low-level request handling, which includes proxy support, so that opens up a possibility to test the connectivity to proxies as well. Proxy configuration is done using environment variables. Below is a mini-tutorial on querying the remote server through local SOCKS proxy, but it shall work with regular HTTP/S proxies as well.

First let's create a file with request data, specifically -- with HTTP headers, as a lot of services are suspicious to the requests that came not from a regular browser, but from some custom software, and redirect them to some crazy captchas or just respond with 4XXs.

req1.http

GET http://2ip.ru
User-Agent: curl/7.68.0

Now, let's perform a direct request. Some -v options were added to examine the actual response from the server, which should contain our external IP address.

The main reason of my regular usage of this service is that it's the fastest way to check your real external IP address from literally everywhere with only one requirement (curl), and it's very easy to memorize: curl 2ip.ru

$ macedon -T1 -vv -f req1.http 

[INFO ][macedon:#0](+117ms) Request #1: GET http://2ip.ru
[INFO ][macedon:#0](+338ms) Response #1: HTTP 200 OK
[TRACE][macedon:#0](+339ms) 
# [R/R 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> GET http://2ip.ru
> User-Agent: curl/7.68.0

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
< HTTP 200 OK
< Connection: keep-alive
< Content-Length: 14
< Content-Type: application/octet-stream
< Date: Tue, 28 Nov 2023 18:32:35 GMT
< Server: nginx

185.243.218.152

Direct requests work, yay. Next, to use a proxy server define the environment variables HTTP_PROXY and HTTPS_PROXY in the following format:

$ export HTTP_PROXY="<protocol>://<user>:<pass>@<proxy>:<port>"
$ export HTTPS_PROXY="<protocol>://<user>:<pass>@<proxy>:<port>"

Needless to say that you shall put the real credentials instead of placeholders, but just in case...

Personally I prefer another method: prepending the command with an environment variable, which sets it as well, but for the one command only. Let's try it:

$ HTTP_PROXY=socks5h://localhost:1080 macedon -f req1.http -vv

[INFO ][macedon:#0](+115ms) Request #1: GET http://2ip.ru
[INFO ][macedon:#0](+1.24s) Response #1: HTTP 200 OK
[TRACE][macedon:#0](+1.24s) 
# [R/R 1]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> GET http://2ip.ru
> User-Agent: curl/7.68.0

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
< HTTP 200 OK
< Connection: keep-alive
< Content-Length: 14
< Content-Type: application/octet-stream
< Date: Tue, 28 Nov 2023 18:51:01 GMT
< Server: nginx

23.129.64.132

The service now sees that we made a request from another address and reflects that in his response. Furthermore, the latency drastically increased (from ~200ms to more than 1s), which also indicates that request was sent through the proxy.

(logs context) The delay can be calculated from numbers in (parentheses) displaying elapsed time from the application launch.

Sometimes using SOCKS proxy causes errors (HTTP(S) proxies unaffected), which usually look like this: [ERROR][...] Missing dependencies for SOCKS support (sigh). The solution is to install the missing dependency, namely — requests[socks] which is an optional dependency and thus is not installed by default.

With pipx

$ pipx inject macedon requests[socks]

Manually

$ ./venv/bin/pip install requests[socks]

Changelog

CHANGES.rst