rechat 0.1.6

Caching and replaying man-in-the-middle proxy for OpenAI APIs

Rechat

Rechat is a caching and replaying man-in-the-middle proxy for OpenAI's APIs, it provides inspection and debugging layer, particularly useful for quick inspection of interactions of existing clients, developing multi-request workflows, and benchmarks.

Rechat is for you, if you ever wanted to:

• speed up your code that makes repeated calls to OpenAI APIs
• quickly inspect what is being sent to OpenAI APIs
• emulate an endpoint with pre-recorded (or pre-defined) responses

Quickstart

1. pip install rechat (dev: pip install git+https://gitlab-master.nvidia.com/dchichkov/rechat.git)
2. Run rechat, it will listen on eight-nine-ten port (http://localhost:8910/v1) and use OpenAI's endpoint by default as upstream.
3. Configure your OpenAI client to use it export OPENAI_BASE_URL=http://localhost:8910/v1 and run your requests as usual.

You can specify a different upstream endpoint by providing it as an argument, e.g. rechat https://api.openai.com/v1.

By default, rechat outputs intercepted chat content onto the console:

And it records the session to flows_<timestamp>.dump file in the current directory. During subsequent runs, if a -f/--flow [dump_file] argument is provided, rechat would attempt to load flows_[timestamp].dump files, or the specified dump file. It always tries to use cached responses for any matching requests.

Inspection

Rechat provides http://localhost:8910 web UI for inspecting the current session, with search and filtering capabilities. By default, rechat will output chat content to the console. Use --quiet flag to reduce verbosity.

Additionally, a --diff mode would search for similar requests on cache misses and print difference between the current request and the closest cached request. This is useful for ensuring that different clients or implementations of the same functionality produce the same requests.

Recording and Replaying

Any markdown editor, for example VSCode or GitHub/GitLab web UI, can be used to view and edit the logs, and these modified logs can be loaded into rechat, to emulate model's responses.

Architecture

flowchart TB
    subgraph Client["Client Application"]
        APP[["OpenAI SDK / HTTP Client"]]
    end

    subgraph Rechat["Rechat Proxy (MITM)"]
        PROXY["Proxy Server"]
        INTERCEPT["Request Interceptor"]
        
        subgraph CacheLookup["Cache Lookup"]
            EXACT["Exact Match?"]
            FUZZY["Fuzzy Match\n(similarity search)"]
        end
        
        CACHE[("Cache Storage")]
        REPLAY["Replay Cached Response"]
        DIFF["Print Diff 📋"]
        STORE["Record Response"]
    end

    subgraph OpenAI["OpenAI API"]
        API[["api.openai.com"]]
    end

    APP -->|"HTTPS Request\n(via proxy)"| PROXY
    PROXY --> INTERCEPT
    INTERCEPT --> EXACT
    
    EXACT -->|"yes"| REPLAY
    REPLAY --> RETURN
    
    EXACT -->|"no"| FUZZY
    FUZZY -->|"found similar"| DIFF
    FUZZY -->|"no match"| FORWARD
    DIFF --> FORWARD
    
    CACHE <-->|"lookup"| EXACT
    CACHE <-->|"similarity search"| FUZZY
    
    FORWARD["Forward Request"] --> API
    API -->|"Response"| STORE
    STORE -->|"save"| CACHE
    STORE --> RETURN["Return Response"]
    
    RETURN --> APP

    style Rechat fill:#1a1a2e,stroke:#16213e,color:#eee
    style CacheLookup fill:#0f3460,stroke:#16213e,color:#eee
    style CACHE fill:#e94560,stroke:#16213e,color:#fff
    style PROXY fill:#533483,stroke:#16213e,color:#fff
    style DIFF fill:#f9a825,stroke:#16213e,color:#000
    style REPLAY fill:#4caf50,stroke:#16213e,color:#fff

Rechat Markdown Format

Example markdown snippet, in markdown format. Note <blockquote> tags. See more details in sample.md.

### user
<blockquote>
What is the capital of France?
</blockquote>

### assistant
<blockquote>
Paris.
</blockquote>

Intercepting traffic to existing OpenAI endpoints

Rechat can intercept traffic to existing endpoints, without changing the client code or configuring OPENAI_BASE_URL, by using mitmproxy as a transparent proxy. For example, to use mitmproxy local proxy mode and intercept traffic from a python script (for example python scripts/query.py), use the following command to intercept the traffic from python:

rechat --mode local:python

And run the python script as follows, specifying mitmproxy's CA certificate for SSL interception:

SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem python scripts/query.py

Note: SSL_CERT_FILE environment variable is required for Python clients to trust mitmproxy's root CA certificate. Please refer to mitmproxy's documentation for more details on installing and trusting mitmproxy's root CA certificate on your system.

Routing and Load Balancing

TODO: Rechat supports multiple endpoints, using <endpoint>:[local_port]:[model_name] arguments, e.g. https://api.openai.com/v1:8910:gpt-5. It would route the requests to the appropriate endpoint based on the model name in the request, and it'd balance the load between endpoints for the same model.

Miscellaneous

• Replaying queries against an endpoint
• Multiple responses for the same request