Chrome Lens API for Python

English | Русский

This project provides a powerful, asynchronous Python library and command-line tool for interacting with Google Lens. It allows you to perform advanced Optical Character Recognition (OCR), get segmented text blocks (e.g., for comics), translate text, and get precise word coordinates.

🚀 Quick Start for Windows Users

No Python needed:

irm https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install.ps1 | iex

📸 Automated ShareX Setup

If you use ShareX, you can fully automate the setup with one command:

lens_scan --setup-sharex

This will automatically configure a hotkey (Ctrl + O) and the necessary actions to use Google Lens OCR. Add -t ru (any language code) to make the hotkey translate instead. Your old HotkeysConfig.json is backed up first. Details: docs/sharex.md.

✨ Key Features

• Modern Backend: Utilizes Google's official Protobuf endpoint (v1/crupload) for robust and accurate results.
• Asynchronous & Safe: Built with asyncio and httpx. Includes a built-in semaphore to prevent API abuse and IP bans from excessive concurrent requests.
• Powerful OCR & Segmentation:
  • Extract text from images as a single string.
  • Get text segmented into logical blocks (paragraphs, dialog bubbles) with their own coordinates.
  • Get individual text lines with their own precise geometry.
• Built-in Translation: Instantly translate recognized text into any supported language.
• Versatile Image Sources: Process images from a file path, URL, bytes, PIL Image object, or NumPy array.
• Chromium-Accurate Text Overlay: Repaints translated text in place the way Chrome's own Lens overlay does — erasing the source with the server's inpainted background patch, then drawing the translation in the original colours, angle and line box. Handles right-to-left scripts, vertical CJK, and per-character font fallback.
• Region Queries: Re-read one region of an image at native resolution (process_region), which reads small print far better than the full-image pass.
• Local Daemon: lens_scan --serve keeps a warm process so callers skip the ~1.2 s interpreter start on every invocation. Doubles as a backend for browser userscripts.
• Feature-Rich CLI: A simple yet powerful command-line interface (lens_scan) for quick use.
• Proxy Support: Full support for HTTP, HTTPS, and SOCKS proxies, plus --no-env-proxy when you want to bypass the proxy your environment sets.
• Clipboard Integration: Instantly copy OCR or translation results to your clipboard with the --sharex flag.
• Flexible Configuration: Manage settings via a config.json file, CLI arguments, or environment variables.

📦 Which install to pick

Measured on the same machine, --help only, median of six runs.

The standalone build starts fastest of the three because Nuitka has compiled the imports away; the one-file variant lost to both by unpacking itself first.

Four ways in, differing in what they need from you:

brew install bropines/tap/lens-scan           # macOS / Linux
uv tool install "chrome-lens-py[clipboard]"   # or: pipx install chrome-lens-py
pip install chrome-lens-py                    # if you manage Python yourself

Homebrew

That is one command, not two: brew taps bropines/homebrew-tap on the way past. It installs the prebuilt standalone folder, so it does not care which Python you have, and the formula is bumped automatically on every release.

Published builds cover Apple Silicon and x86_64 Linux. On an Intel Mac or ARM Linux the formula stops and points you at uv, rather than failing to download something that was never built.

One-line install with uv

Installs uv if you do not have it, then installs lens_scan as a uv tool and puts it on PATH. Nothing is frozen into an executable, so there is nothing for antivirus heuristics to react to, and no Python is needed beforehand — uv brings its own.

irm https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install-uv.ps1 | iex

curl -fsSL https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install-uv.sh | sh

Installs the [clipboard] extra too, so --sharex works out of the box.

uv tool upgrade chrome-lens-py      # later
uv tool uninstall chrome-lens-py

If another lens_scan is already on your PATH — a pip install behind a pyenv shim, say, or an older standalone build — that one may come first and keep winning after this install. The installer warns you when it spots this. lens_scan --version prints the path it actually ran from, which settles it.

One-line install of the standalone build

Downloads the latest release, verifies its SHA-256, unpacks it, puts lens_scan on your PATH, and checks it starts before declaring victory.

# Windows
irm https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install.ps1 | iex

# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install.sh | sh

Installs to %LOCALAPPDATA%\Programs\lens-scan or ~/.local/share/lens-scan. Re-running replaces the existing install, unpacking into a staging folder first, so a download that fails verification leaves the old install untouched. Pin a version with LENS_SCAN_VERSION=3.6.0.

Builds: Windows x64 (compiled, plus the portable fallback), Linux x64 and arm64 (built against glibc 2.35, so Ubuntu 22.04 / Debian 12 and newer), macOS Apple Silicon and Intel. Anything else is pointed at uv tool install.

Scoop

scoop bucket add lens https://github.com/bropines/chrome-lens-py
scoop install lens-scan            # or lens-scan-portable

Docker (the daemon)

docker run -d -p 127.0.0.1:8765:8765 -e LENS_TOKEN=change-me ghcr.io/bropines/chrome-lens-py

The container refuses to start without LENS_TOKEN, because inside the container it listens beyond loopback.

Every release publishes a .sha256 next to each archive, and the installers check it before unpacking. Verify one by hand with:

sha256sum -c lens_scan-linux-amd64.zip.sha256

Be clear about what that buys: it catches a corrupted or truncated download and a CDN serving something other than what was uploaded. It is not a signature — anyone able to rewrite the release could rewrite the checksum with it.

For that, every archive also carries a signed build-provenance attestation, proving it was built by this repository's release workflow from a given commit:

gh attestation verify lens_scan-linux-amd64.zip --repo bropines/chrome-lens-py

Not sure which copy you are running? lens_scan --version prints the version along with the path it was loaded from, which settles the usual confusion between a pip install, a uv tool install, an editable checkout and a standalone build all owning the same name.

🚀 Installation

You can install the package using pip:

pip install chrome-lens-py

To enable clipboard functionality (the --sharex flag), install the library with the [clipboard] extra:

pip install "chrome-lens-py[clipboard]"

Or, install the latest version directly from GitHub:

pip install git+https://github.com/bropines/chrome-lens-py.git

🚀 Usage

lens_scan <image_source> [ocr_lang] [options]

lens_scan "path/to/your/image.png" -t en

lens_scan "path/to/manga.jpg" ja -b

lens_scan "path/to/document.png" --output-lines

lens_scan "path/to/diagram.png" --get-coords

lens_scan "path/to/manga.jpg" ja -t ru -to "translated_manga.png" -sx

lens_scan "https://i.imgur.com/VPd1y6b.png" en --ocr-single-line

lens_scan "image.png" --proxy "socks5://127.0.0.1:9050"

import asyncio
from chrome_lens_py import LensAPI

async def main():
    # Initialize the API. You can pass a proxy, region, etc. here.
    # By default, an API key is not required.
    api = LensAPI()

    image_source = "path/to/your/image.png" # Or a URL, PIL Image, NumPy array

    try:
        # Process the image and get a single string of text
        result = await api.process_image(
            image_path=image_source,
            ocr_language="ja",
            target_translation_language="en"
        )

        print("--- OCR Text ---")
        print(result.get("ocr_text"))

        print("\n--- Translated Text ---")
        print(result.get("translated_text"))
        
    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    asyncio.run(main())

from PIL import Image
import numpy as np

# ... inside an async function ...

# From a URL
result_url = await api.process_image("https://i.imgur.com/VPd1y6b.png")

# From a PIL Image object
with Image.open("path/to/image.png") as img:
    result_pil = await api.process_image(img)

# From a NumPy array (e.g., loaded via OpenCV)
with Image.open("path/to/image.png") as img:
    numpy_array = np.array(img)
    result_numpy = await api.process_image(numpy_array)

import asyncio
from chrome_lens_py import LensAPI

async def process_comics():
    api = LensAPI()
    image_source = "path/to/manga.jpg"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='blocks' # Get segmented blocks instead of a single string
    )

    # The result now contains a 'text_blocks' key
    text_blocks = result.get("text_blocks", [])
    print(f"Found {len(text_blocks)} text blocks.")

    for i, block in enumerate(text_blocks):
        print(f"\n--- Block #{i+1} ---")
        print(block['text'])
        # block also contains 'lines' and 'geometry' keys

asyncio.run(process_comics())

import asyncio
from chrome_lens_py import LensAPI

async def process_document_lines():
    api = LensAPI()
    image_source = "path/to/document.png"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='lines' # Get individual lines with their geometry
    )

    # The result now contains a 'line_blocks' key
    line_blocks = result.get("line_blocks", [])
    print(f"Found {len(line_blocks)} lines.")

    for i, line in enumerate(line_blocks):
        print(f"\n--- Line #{i+1} ---")
        print(f"Text: {line['text']}")
        print(f"Geometry: {line['geometry']}")

asyncio.run(process_document_lines())

import asyncio
from chrome_lens_py import LensAPI

async def process_with_details():
    api = LensAPI()
    image_source = "path/to/document.png"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='detailed' # Get the fully nested structure
    )

    # The result now contains a 'detailed_blocks' key
    detailed_blocks = result.get("detailed_blocks", [])
    print(f"Found {len(detailed_blocks)} detailed blocks.")

    for i, block in enumerate(detailed_blocks):
        print(f"\n--- Block #{i+1} ---")
        print(f"  Geometry: {block['geometry']}")
        for j, line in enumerate(block['lines']):
            print(f"    --- Line #{j+1}: '{line['text']}' ---")
            for k, word in enumerate(line['words']):
                 print(f"      - Word: '{word['text']}', Geometry: {word['geometry']}")

asyncio.run(process_with_details())

api = LensAPI(
    api_key: str = "YOUR_API_KEY_OR_DEFAULT",
    client_region: Optional[str] = None,
    client_time_zone: Optional[str] = None,
    proxy: Optional[str] = None,
    timeout: int = 60,
    font_path: Optional[str] = None,
    font_size: Optional[int] = None,
    max_concurrent: int = 5,
    trust_env: bool = True,
)

async with LensAPI() as api:
    ...
# or: api = LensAPI(); ...; await api.aclose()

result: dict = await api.process_image(
    image_path: Any,
    ocr_language: Optional[str] = None,
    target_translation_language: Optional[str] = None,
    source_translation_language: Optional[str] = None,
    output_overlay_path: Optional[str] = None,
    new_session: bool = True,
    ocr_preserve_line_breaks: bool = True,
    output_format: Literal['full_text', 'blocks', 'lines', 'detailed'] = 'full_text',
    # overlay rendering; see the rendering section for what each one is for
    overlay_mode: Literal['chromium', 'legacy'] = 'chromium',
    vertical_text: Literal['keep', 'auto', 'horizontal'] = 'auto',
    erase_mode: Literal['patch', 'hull'] = 'patch',
    hull_padding: float = 0.45,
    outline_scale: float = 1.0,
    min_readable_px: float = 0.0,
    text_align: Literal['auto', 'left', 'center', 'right'] = 'auto',
    manga_mode: bool = False,
    manga_box_growth: float = 1.45,
    include_raw_response: bool = False,
)

result: dict = await api.process_region(
    image_path: Any,
    region: Tuple[float, float, float, float],   # center_x, center_y, w, h
    ocr_language: Optional[str] = None,
    text_query: Optional[str] = None,
    ocr_preserve_line_breaks: bool = True,
)

{
  "proxy": "socks5://127.0.0.1:9050",
  "client_region": "DE",
  "client_time_zone": "Europe/Berlin",
  "timeout": 90,
  "retries": 3,
  "target_lang": "en",
  "overlay_mode": "chromium",
  "manga_mode": false,
  "concurrency": 4,
  "font_path": "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
  "ocr_preserve_line_breaks": true
}

⚡ Daemon / fast mode

A cold lens_scan spends most of its time importing Pillow, protobuf and httpx and opening a fresh TLS connection to Google. A process that is already running has done both. ShareX can only execute a program per screenshot, so lens_scan is now a thin client that hands the command to a warm background daemon by itself - there is nothing to set up.

Measured on Windows 11, Python 3.13, the real API through a local proxy, the same image each time, median of 7:

The last two rows are the floor: this machine's interpreter alone takes 98 ms (an editable-install .pth in site accounts for about half of that), and the client adds roughly 40 ms on top (about 19 ms to import, 10 ms for the first call, in a process that imports no Pillow, protobuf, httpx or rich).

How auto works (the default)

1. lens_scan looks for the daemon's discovery file (below). If a daemon is running and is the same version, it sends it the command line and prints what comes back - same stdout, same stderr, same exit code.
2. If none is running, it does the work in-process exactly as before and starts a detached background daemon for the next call (no console window, nothing inherited from the caller: ShareX can read the output to EOF without waiting for the daemon).
3. The daemon exits by itself after daemon_idle_timeout seconds (default 600) with no requests, and removes its files.

Correctness never depends on the daemon. If anything is off - no daemon, a stale file, a version mismatch, a refused or dropped connection (even mid-request), a reply that does not parse, a timeout - the client silently runs the command in-process instead. A daemon of a different version is asked to exit and is replaced. Several first calls at once start one daemon (a lock file), not several.

What runs in the daemon is the same CLI code, with the client's working directory (so relative paths work), the client's LENS_* variables and config file (so settings behave the same), and --sharex copying to your clipboard from the daemon, which runs in your session. What never runs in the daemon: --help, --version, --setup-sharex, --update-config, --serve, the service and daemon commands, and any command line argparse would reject - they run in your own process, so they behave exactly as without a daemon.

Turning it on, off, or managing it

lens_scan img.png --daemon off          # this call only
export LENS_DAEMON=off                  # this shell / CI job
#   "daemon": "off"                     # in config.json, for good
lens_scan img.png --daemon on           # wait (up to 5 s) for a starting daemon
                                        # instead of running in-process

lens_scan --daemon-status               # pid, version, uptime, transport, requests, idle time left
lens_scan --daemon-stop
lens_scan --daemon-start                # normally unnecessary
lens_scan --daemon-idle-timeout 3600 img.png      # longer-lived daemon (0 = never exits)

Use LENS_DAEMON=off in CI and containers where a process that outlives the job is unwelcome. The daemon's log is daemon.log next to the state files.

Things to know:

• The client forwards LENS_* variables and a few rendering variables (NO_COLOR, FORCE_COLOR, COLORTERM, TERM, XDG_CONFIG_HOME). It does not forward HTTP_PROXY/HTTPS_PROXY/ALL_PROXY: the daemon reads those from the environment it was started in. If you change them, run lens_scan --daemon-stop (or set proxy / LENS_PROXY, which are forwarded).
• -l DEBUG in a forwarded run shows this package's debug log, not httpx's.
• One daemon per user. It keeps one HTTP/2 connection per distinct combination of API key, proxy, timeout, region and the other connection settings.

Where it lives, and who can talk to it

daemon.json holds {version, pid, transport, address, token, started_at}, is written atomically, and is how a client finds the daemon. Every request must carry the random 32-byte token (secrets.token_urlsafe); on POSIX the socket's permissions already keep other users out, and the token is sent regardless. No browser can reach it: the client sends no Origin, and a request with one is refused (see What can reach it). Override the directory with LENS_STATE_DIR (the tests do).

ShareX: two hotkeys

lens_scan --setup-sharex --mode both -t ru      # -t is optional

• Ctrl + Shift + O posts the capture straight to the daemon - no process per capture, the fastest path.
• Ctrl + O runs lens_scan on it - works even when the daemon is down, and brings the daemon back.

--mode uploader installs only the first, --mode program (the default) only the second. The uploader is registered in ShareX and selected by the hotkey itself, so your normal image destination is left as it was.

Uploader and both modes install the daemon as a login service on a fixed port (--port, setting daemon_port, default 8765): on Windows a Startup-folder item running a supervisor that restarts the daemon if it crashes, with no console window; a KeepAlive LaunchAgent on macOS; a systemd --user unit with Restart=on-failure on Linux. --install-service / --uninstall-service manage it on their own. Details: docs/sharex.md.

To keep the auto daemon alive too, set daemon_idle_timeout: 0 (or LENS_DAEMON_IDLE_TIMEOUT=0).

🖥️ Daemon mode

This is the plain HTTP server for your own programs (the fast mode above is the automatic one). Importing Pillow, protobuf and httpx costs roughly a second. If something calls it repeatedly, run it as a daemon and pay that once:

lens_scan --serve                      # http://127.0.0.1:8765
lens_scan --serve --port 9000 --token secret

Every rendering option the CLI has is accepted here under the same name, so --erase-mode hull --outline 3 on the command line is "erase_mode": "hull", "outline_scale": 3 in the body. Requests must be Content-Type: application/json (/v1/ocr/raw takes an image type instead).

What can reach it

This daemon holds an API key and will call Google for anyone who reaches it, so the defaults are deliberately closed:

• No web page can use it. A request carrying an Origin header is refused unless you named that origin with --allow-origin. The check is on the request, not the response, because a page can send a POST it is forbidden to read and still get the side effect it wanted — a Google call billed to your key, or a file opened off your disk.
• --host anywhere but loopback requires --token, and then the daemon accepts only image_b64. A path or a URL is something it would go and open itself, which on a network listener means reading your disk, or fetching a host of the caller's choosing.
• Tokens are compared in constant time.
• /v1/ocr/raw is closed to pages for the same reason as the JSON routes. The four image types it accepts are not ones a page may send without a CORS preflight (only text/plain, form-encoded and multipart are), and the preflight carries an Origin, which is refused. The tests cover both the POST and the preflight.
• The background daemon of fast mode is stricter still: its private socket / ephemeral port is announced only in a file readable by your user, every request needs its random token, and its extra routes (/v1/cli, /v1/status, /v1/shutdown, which can run a lens_scan command on your behalf) exist only on a local listener that has a token.

# let one page you control use it
lens_scan --serve --allow-origin https://yoursite.example

# on the network: token required, pixels only
lens_scan --serve --host 0.0.0.0 --token "$(openssl rand -hex 24)"

The Tampermonkey userscript does not need any of this — it talks to Google directly through GM_xmlhttpRequest, which is not subject to CORS. Opening the daemon to a browser origin is only for pages you write yourself.

Measured here, same work each time:

The binary being the slowest is not a typo. Onefile extracts its payload to a temporary directory on every run, which costs more than importing the modules did: startup alone is 1.67 s against 0.52 s from source. Freezing is for distribution, not for speed - if startup is what you care about, the daemon is the answer, and --standalone rather than --onefile would at least stop the binary making it worse.

🔍 Region queries

The full-image pass downscales anything over 1600px, which is exactly when small print suffers. A region query sends only the cropped pixels, at native scale:

async with LensAPI() as api:
    # (center_x, center_y, width, height), normalized against the full image
    result = await api.process_region("page.png", (0.8, 0.82, 0.35, 0.06))
    print(result["ocr_text"])

lens_scan page.png --region 0.8,0.82,0.35,0.06
lens_scan page.png --region 0.8,0.82,0.35,0.06 --text-query "what does this say"

Geometry comes back in full-image coordinates, so it lines up with process_image output directly.

🎨 Overlay rendering

lens_scan manga.png -t ru -to out.png                      # Chromium-style, default
lens_scan manga.png -t ru -to out.png --vertical-text horizontal
lens_scan page.png  -t ru -to out.png --overlay-mode legacy

--vertical-text controls what happens to top-to-bottom CJK source text:

Making the result readable

Chromium repaints each line into the box the original occupied. That is right for a street sign and wrong for a manga bubble, where the translation is far longer than the Japanese it replaces. These knobs exist for that gap:

lens_scan page.png -t en -to out.png --manga
lens_scan page.png -t en -to out.png --erase-mode hull --outline 3 --min-text-size 18

Right-to-left targets (Arabic, Hebrew, Persian…) need shaping that Pillow's published wheels cannot do, since they are built without libraqm:

pip install "chrome-lens-py[rtl]"      # python-bidi + arabic-reshaper
pip install "chrome-lens-py[fonts]"    # fontTools, for per-character font fallback

Without the fonts extra the renderer still works, but a font lacking glyphs for the target script will draw tofu boxes.

Sharex Integration

Check sharex.md for more information on how to use this library with ShareX.

❤️ Support & Acknowledgments

• OWOCR: Greatly inspired by and based on OWOCR. Thank you to them for their research into Protobuf and OCR implementation.
• Chrome Lens OCR: For the original implementation and ideas that formed the basis of this library. The update with SHAREX support was originally tested and added by me to chrome-lens-ocr, thanks for the initial implementation and ideas.
• AI Collaboration: A significant portion of the v3.0 code, including the architectural refactor, asynchronous implementation, and Protobuf integration, was developed in collaboration with an advanced AI assistant.
• GOOGLE: For the convenient and high-quality Lens technology.
• Support the Author: If you find this library useful, you can support the author - Boosty

Star History

Disclaimer

This project is intended for educational and experimental purposes only. Use of Google's services must comply with their Terms of Service. The author is not responsible for any misuse of this software.