Chrome Lens API for Python

English | Русский

This project provides a powerful, asynchronous Python library and command-line tool for interacting with Google Lens. It allows you to perform advanced Optical Character Recognition (OCR), get segmented text blocks (e.g., for comics), translate text, and get precise word coordinates.

🚀 Quick Start for Windows Users

If you don't want to install Python, you can download the standalone lens_scan-windows-amd64.exe from the Releases section.

📸 Automated ShareX Setup

If you use ShareX, you can fully automate the setup with one command:

# Using the installed package:
lens_scan --setup-sharex

# Or using the standalone .exe:
lens_scan-windows-amd64.exe --setup-sharex

This will automatically configure a hotkey (Ctrl + O) and the necessary actions to use Google Lens OCR.

✨ Key Features

• Modern Backend: Utilizes Google's official Protobuf endpoint (v1/crupload) for robust and accurate results.
• Asynchronous & Safe: Built with asyncio and httpx. Includes a built-in semaphore to prevent API abuse and IP bans from excessive concurrent requests.
• Powerful OCR & Segmentation:
  • Extract text from images as a single string.
  • Get text segmented into logical blocks (paragraphs, dialog bubbles) with their own coordinates.
  • Get individual text lines with their own precise geometry.
• Built-in Translation: Instantly translate recognized text into any supported language.
• Versatile Image Sources: Process images from a file path, URL, bytes, PIL Image object, or NumPy array.
• Chromium-Accurate Text Overlay: Repaints translated text in place the way Chrome's own Lens overlay does — erasing the source with the server's inpainted background patch, then drawing the translation in the original colours, angle and line box. Handles right-to-left scripts, vertical CJK, and per-character font fallback.
• Region Queries: Re-read one region of an image at native resolution (process_region), which reads small print far better than the full-image pass.
• Local Daemon: lens_scan --serve keeps a warm process so callers skip the ~1.2 s interpreter start on every invocation. Doubles as a backend for browser userscripts.
• Feature-Rich CLI: A simple yet powerful command-line interface (lens_scan) for quick use.
• Proxy Support: Full support for HTTP, HTTPS, and SOCKS proxies, plus --no-env-proxy when you want to bypass the proxy your environment sets.
• Clipboard Integration: Instantly copy OCR or translation results to your clipboard with the --sharex flag.
• Flexible Configuration: Manage settings via a config.json file, CLI arguments, or environment variables.

📦 Which install to pick

Measured on the same machine, --help only, median of six runs.

The standalone build starts fastest of the three because Nuitka has compiled the imports away; the one-file variant lost to both by unpacking itself first.

There are three ways in, and they differ in what they need from you:

uv tool install "chrome-lens-py[clipboard]"   # or: pipx install chrome-lens-py
pip install chrome-lens-py                    # if you manage Python yourself

One-line install with uv

Installs uv if you do not have it, then installs lens_scan as a uv tool and puts it on PATH. Nothing is frozen into an executable, so there is nothing for antivirus heuristics to react to, and no Python is needed beforehand — uv brings its own.

irm https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install-uv.ps1 | iex

curl -fsSL https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install-uv.sh | sh

Installs the [clipboard] extra too, so --sharex works out of the box.

uv tool upgrade chrome-lens-py      # later
uv tool uninstall chrome-lens-py

If another lens_scan is already on your PATH — a pip install behind a pyenv shim, say, or an older standalone build — that one may come first and keep winning after this install. The installer warns you when it spots this. lens_scan --version prints the path it actually ran from, which settles it.

One-line install of the standalone build

Downloads the latest release, verifies its SHA-256, unpacks it, puts lens_scan on your PATH, and checks it starts before declaring victory.

# Windows
irm https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install.ps1 | iex

# Linux / macOS
curl -fsSL https://raw.githubusercontent.com/bropines/chrome-lens-py/main/scripts/install.sh | sh

Installs to %LOCALAPPDATA%\Programs\lens-scan or ~/.local/share/lens-scan; re-running replaces the existing install. Both scripts refuse to install a pre-standalone release rather than quietly handing you the self-extracting build that caused the antivirus reports.

Every release publishes a .sha256 next to each archive, and the installers check it before unpacking. Verify one by hand with:

sha256sum -c lens_scan-linux-amd64.zip.sha256

Be clear about what that buys: it catches a corrupted or truncated download and a CDN serving something other than what was uploaded. It is not a signature — anyone able to rewrite the release could rewrite the checksum with it.

Not sure which copy you are running? lens_scan --version prints the version along with the path it was loaded from, which settles the usual confusion between a pip install, a uv tool install, an editable checkout and a standalone build all owning the same name.

🚀 Installation

You can install the package using pip:

pip install chrome-lens-py

To enable clipboard functionality (the --sharex flag), install the library with the [clipboard] extra:

pip install "chrome-lens-py[clipboard]"

Or, install the latest version directly from GitHub:

pip install git+https://github.com/bropines/chrome-lens-py.git

🚀 Usage

lens_scan <image_source> [ocr_lang] [options]

lens_scan "path/to/your/image.png" -t en

lens_scan "path/to/manga.jpg" ja -b

lens_scan "path/to/document.png" --output-lines

lens_scan "path/to/diagram.png" --get-coords

lens_scan "path/to/manga.jpg" ja -t ru -to "translated_manga.png" -sx

lens_scan "https://i.imgur.com/VPd1y6b.png" en --ocr-single-line

lens_scan "image.png" --proxy "socks5://127.0.0.1:9050"

import asyncio
from chrome_lens_py import LensAPI

async def main():
    # Initialize the API. You can pass a proxy, region, etc. here.
    # By default, an API key is not required.
    api = LensAPI()

    image_source = "path/to/your/image.png" # Or a URL, PIL Image, NumPy array

    try:
        # Process the image and get a single string of text
        result = await api.process_image(
            image_path=image_source,
            ocr_language="ja",
            target_translation_language="en"
        )

        print("--- OCR Text ---")
        print(result.get("ocr_text"))

        print("\n--- Translated Text ---")
        print(result.get("translated_text"))
        
    except Exception as e:
        print(f"An error occurred: {e}")

if __name__ == "__main__":
    asyncio.run(main())

from PIL import Image
import numpy as np

# ... inside an async function ...

# From a URL
result_url = await api.process_image("https://i.imgur.com/VPd1y6b.png")

# From a PIL Image object
with Image.open("path/to/image.png") as img:
    result_pil = await api.process_image(img)

# From a NumPy array (e.g., loaded via OpenCV)
with Image.open("path/to/image.png") as img:
    numpy_array = np.array(img)
    result_numpy = await api.process_image(numpy_array)

import asyncio
from chrome_lens_py import LensAPI

async def process_comics():
    api = LensAPI()
    image_source = "path/to/manga.jpg"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='blocks' # Get segmented blocks instead of a single string
    )

    # The result now contains a 'text_blocks' key
    text_blocks = result.get("text_blocks", [])
    print(f"Found {len(text_blocks)} text blocks.")

    for i, block in enumerate(text_blocks):
        print(f"\n--- Block #{i+1} ---")
        print(block['text'])
        # block also contains 'lines' and 'geometry' keys

asyncio.run(process_comics())

import asyncio
from chrome_lens_py import LensAPI

async def process_document_lines():
    api = LensAPI()
    image_source = "path/to/document.png"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='lines' # Get individual lines with their geometry
    )

    # The result now contains a 'line_blocks' key
    line_blocks = result.get("line_blocks", [])
    print(f"Found {len(line_blocks)} lines.")

    for i, line in enumerate(line_blocks):
        print(f"\n--- Line #{i+1} ---")
        print(f"Text: {line['text']}")
        print(f"Geometry: {line['geometry']}")

asyncio.run(process_document_lines())

import asyncio
from chrome_lens_py import LensAPI

async def process_with_details():
    api = LensAPI()
    image_source = "path/to/document.png"
    
    result = await api.process_image(
        image_path=image_source,
        output_format='detailed' # Get the fully nested structure
    )

    # The result now contains a 'detailed_blocks' key
    detailed_blocks = result.get("detailed_blocks", [])
    print(f"Found {len(detailed_blocks)} detailed blocks.")

    for i, block in enumerate(detailed_blocks):
        print(f"\n--- Block #{i+1} ---")
        print(f"  Geometry: {block['geometry']}")
        for j, line in enumerate(block['lines']):
            print(f"    --- Line #{j+1}: '{line['text']}' ---")
            for k, word in enumerate(line['words']):
                 print(f"      - Word: '{word['text']}', Geometry: {word['geometry']}")

asyncio.run(process_with_details())

api = LensAPI(
    api_key: str = "YOUR_API_KEY_OR_DEFAULT",
    client_region: Optional[str] = None,
    client_time_zone: Optional[str] = None,
    proxy: Optional[str] = None,
    timeout: int = 60,
    font_path: Optional[str] = None,
    font_size: Optional[int] = None,
    max_concurrent: int = 5,
    trust_env: bool = True,
)

async with LensAPI() as api:
    ...
# or: api = LensAPI(); ...; await api.aclose()

result: dict = await api.process_image(
    image_path: Any,
    ocr_language: Optional[str] = None,
    target_translation_language: Optional[str] = None,
    source_translation_language: Optional[str] = None,
    output_overlay_path: Optional[str] = None,
    new_session: bool = True,
    ocr_preserve_line_breaks: bool = True,
    output_format: Literal['full_text', 'blocks', 'lines', 'detailed'] = 'full_text',
    # overlay rendering; see the rendering section for what each one is for
    overlay_mode: Literal['chromium', 'legacy'] = 'chromium',
    vertical_text: Literal['keep', 'auto', 'horizontal'] = 'auto',
    erase_mode: Literal['patch', 'hull'] = 'patch',
    hull_padding: float = 0.45,
    outline_scale: float = 1.0,
    min_readable_px: float = 0.0,
    text_align: Literal['auto', 'left', 'center', 'right'] = 'auto',
    manga_mode: bool = False,
    manga_box_growth: float = 1.45,
    include_raw_response: bool = False,
)

result: dict = await api.process_region(
    image_path: Any,
    region: Tuple[float, float, float, float],   # center_x, center_y, w, h
    ocr_language: Optional[str] = None,
    text_query: Optional[str] = None,
    ocr_preserve_line_breaks: bool = True,
)

{
  "api_key": "OPTIONAL! If you don't know what this is, I don't recommend setting it here.",
  "proxy": "socks5://127.0.0.1:9050",
  "client_region": "DE",
  "client_time_zone": "Europe/Berlin",
  "timeout": 90,
  "font_path": "/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf",
  "ocr_preserve_line_breaks": true
}

🖥️ Daemon mode

Importing Pillow, protobuf and httpx costs roughly a second. If something calls lens_scan repeatedly — ShareX on every screenshot, say — run it as a daemon and pay that once:

lens_scan --serve                      # http://127.0.0.1:8765
lens_scan --serve --port 9000 --token secret

Every rendering option the CLI has is accepted here under the same name, so --erase-mode hull --outline 3 on the command line is "erase_mode": "hull", "outline_scale": 3 in the body. Requests must be Content-Type: application/json.

What can reach it

This daemon holds an API key and will call Google for anyone who reaches it, so the defaults are deliberately closed:

• No web page can use it. A request carrying an Origin header is refused unless you named that origin with --allow-origin. The check is on the request, not the response, because a page can send a POST it is forbidden to read and still get the side effect it wanted — a Google call billed to your key, or a file opened off your disk.
• --host anywhere but loopback requires --token, and then the daemon accepts only image_b64. A path or a URL is something it would go and open itself, which on a network listener means reading your disk, or fetching a host of the caller's choosing.
• Tokens are compared in constant time.

# let one page you control use it
lens_scan --serve --allow-origin https://yoursite.example

# on the network: token required, pixels only
lens_scan --serve --host 0.0.0.0 --token "$(openssl rand -hex 24)"

The Tampermonkey userscript does not need any of this — it talks to Google directly through GM_xmlhttpRequest, which is not subject to CORS. Opening the daemon to a browser origin is only for pages you write yourself.

Measured here, same work each time:

The binary being the slowest is not a typo. Onefile extracts its payload to a temporary directory on every run, which costs more than importing the modules did: startup alone is 1.67 s against 0.52 s from source. Freezing is for distribution, not for speed - if startup is what you care about, the daemon is the answer, and --standalone rather than --onefile would at least stop the binary making it worse.

🔍 Region queries

The full-image pass downscales anything over 1600px, which is exactly when small print suffers. A region query sends only the cropped pixels, at native scale:

async with LensAPI() as api:
    # (center_x, center_y, width, height), normalized against the full image
    result = await api.process_region("page.png", (0.8, 0.82, 0.35, 0.06))
    print(result["ocr_text"])

lens_scan page.png --region 0.8,0.82,0.35,0.06
lens_scan page.png --region 0.8,0.82,0.35,0.06 --text-query "what does this say"

Geometry comes back in full-image coordinates, so it lines up with process_image output directly.

🎨 Overlay rendering

lens_scan manga.png -t ru -to out.png                      # Chromium-style, default
lens_scan manga.png -t ru -to out.png --vertical-text horizontal
lens_scan page.png  -t ru -to out.png --overlay-mode legacy

--vertical-text controls what happens to top-to-bottom CJK source text:

Making the result readable

Chromium repaints each line into the box the original occupied. That is right for a street sign and wrong for a manga bubble, where the translation is far longer than the Japanese it replaces. These knobs exist for that gap:

lens_scan page.png -t en -to out.png --manga
lens_scan page.png -t en -to out.png --erase-mode hull --outline 3 --min-text-size 18

Right-to-left targets (Arabic, Hebrew, Persian…) need shaping that Pillow's published wheels cannot do, since they are built without libraqm:

pip install "chrome-lens-py[rtl]"      # python-bidi + arabic-reshaper
pip install "chrome-lens-py[fonts]"    # fontTools, for per-character font fallback

Without the fonts extra the renderer still works, but a font lacking glyphs for the target script will draw tofu boxes.

Sharex Integration

Check sharex.md for more information on how to use this library with ShareX.

❤️ Support & Acknowledgments

• OWOCR: Greatly inspired by and based on OWOCR. Thank you to them for their research into Protobuf and OCR implementation.
• Chrome Lens OCR: For the original implementation and ideas that formed the basis of this library. The update with SHAREX support was originally tested and added by me to chrome-lens-ocr, thanks for the initial implementation and ideas.
• AI Collaboration: A significant portion of the v3.0 code, including the architectural refactor, asynchronous implementation, and Protobuf integration, was developed in collaboration with an advanced AI assistant.
• GOOGLE: For the convenient and high-quality Lens technology.
• Support the Author: If you find this library useful, you can support the author - Boosty

Star History

Disclaimer

This project is intended for educational and experimental purposes only. Use of Google's services must comply with their Terms of Service. The author is not responsible for any misuse of this software.