Jina-Serve
Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets. Scale your services from local development to production while focusing on your core logic.
Key Features
- Native support for all major ML frameworks and data types
- High-performance service design with scaling, streaming, and dynamic batching
- LLM serving with streaming output
- Built-in Docker integration and Executor Hub
- One-click deployment to Jina AI Cloud
- Enterprise-ready with Kubernetes and Docker Compose support
Comparison with FastAPI
Key advantages over FastAPI:
- DocArray-based data handling with native gRPC support
- Built-in containerization and service orchestration
- Seamless scaling of microservices
- One-command cloud deployment
Install
pip install jina
See guides for Apple Silicon and Windows.
Core Concepts
Three main layers:
- Data: BaseDoc and DocList for input/output
- Serving: Executors process Documents, Gateway connects services
- Orchestration: Deployments serve Executors, Flows create pipelines
Build AI Services
Let's create a gRPC-based AI service using StableLM:
from jina import Executor, requests
from docarray import DocList, BaseDoc
from transformers import pipeline
class Prompt(BaseDoc):
text: str
class Generation(BaseDoc):
prompt: str
text: str
class StableLM(Executor):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.generator = pipeline(
'text-generation', model='stabilityai/stablelm-base-alpha-3b'
)
@requests
def generate(self, docs: DocList[Prompt], **kwargs) -> DocList[Generation]:
generations = DocList[Generation]()
prompts = docs.text
llm_outputs = self.generator(prompts)
for prompt, output in zip(prompts, llm_outputs):
generations.append(Generation(prompt=prompt, text=output))
return generations
Deploy with Python or YAML:
from jina import Deployment
from executor import StableLM
dep = Deployment(uses=StableLM, timeout_ready=-1, port=12345)
with dep:
dep.block()
jtype: Deployment
with:
uses: StableLM
py_modules:
- executor.py
timeout_ready: -1
port: 12345
Use the client:
from jina import Client
from docarray import DocList
from executor import Prompt, Generation
prompt = Prompt(text='suggest an interesting image generation prompt')
client = Client(port=12345)
response = client.post('/', inputs=[prompt], return_type=DocList[Generation])
Build Pipelines
Chain services into a Flow:
from jina import Flow
flow = Flow(port=12345).add(uses=StableLM).add(uses=TextToImage)
with flow:
flow.block()
Scaling and Deployment
Local Scaling
Boost throughput with built-in features:
- Replicas for parallel processing
- Shards for data partitioning
- Dynamic batching for efficient model inference
Example scaling a Stable Diffusion deployment:
jtype: Deployment
with:
uses: TextToImage
timeout_ready: -1
py_modules:
- text_to_image.py
env:
CUDA_VISIBLE_DEVICES: RR
replicas: 2
uses_dynamic_batching:
/default:
preferred_batch_size: 10
timeout: 200
Cloud Deployment
Containerize Services
- Structure your Executor:
TextToImage/
├── executor.py
├── config.yml
├── requirements.txt
- Configure:
# config.yml
jtype: TextToImage
py_modules:
- executor.py
metas:
name: TextToImage
description: Text to Image generation Executor
- Push to Hub:
jina hub push TextToImage
Deploy to Kubernetes
jina export kubernetes flow.yml ./my-k8s
kubectl apply -R -f my-k8s
Use Docker Compose
jina export docker-compose flow.yml docker-compose.yml
docker-compose up
JCloud Deployment
Deploy with a single command:
jina cloud deploy jcloud-flow.yml
LLM Streaming
Enable token-by-token streaming for responsive LLM applications:
- Define schemas:
from docarray import BaseDoc
class PromptDocument(BaseDoc):
prompt: str
max_tokens: int
class ModelOutputDocument(BaseDoc):
token_id: int
generated_text: str
- Initialize service:
from transformers import GPT2Tokenizer, GPT2LMHeadModel
class TokenStreamingExecutor(Executor):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.model = GPT2LMHeadModel.from_pretrained('gpt2')
- Implement streaming:
@requests(on='/stream')
async def task(self, doc: PromptDocument, **kwargs) -> ModelOutputDocument:
input = tokenizer(doc.prompt, return_tensors='pt')
input_len = input['input_ids'].shape[1]
for _ in range(doc.max_tokens):
output = self.model.generate(**input, max_new_tokens=1)
if output[0][-1] == tokenizer.eos_token_id:
break
yield ModelOutputDocument(
token_id=output[0][-1],
generated_text=tokenizer.decode(
output[0][input_len:], skip_special_tokens=True
),
)
input = {
'input_ids': output,
'attention_mask': torch.ones(1, len(output[0])),
}
- Serve and use:
# Server
with Deployment(uses=TokenStreamingExecutor, port=12345, protocol='grpc') as dep:
dep.block()
# Client
async def main():
client = Client(port=12345, protocol='grpc', asyncio=True)
async for doc in client.stream_doc(
on='/stream',
inputs=PromptDocument(prompt='what is the capital of France ?', max_tokens=10),
return_type=ModelOutputDocument,
):
print(doc.generated_text)
Support
Jina-serve is backed by Jina AI and licensed under Apache-2.0.
1 maintainer
Metadata
1 maintainer
Release files for jina 3.34.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| jina-3.34.0.tar.gz | 378.9 kB | Without provenance | Details for jina-3.34.0.tar.gz |
Built distributions (wheels)
Total release size: 76.1 MB
Release files / jina-3.34.0.tar.gz
| Download URL | jina-3.34.0.tar.gz |
|---|---|
| Size | 378.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2f7a485677ebe82c9592eb3d59e9777ffcbf3049b6b6d1a6f8cfdad1ee96e0a5
|
|
BLAKE2b-256 checksum How to use checksums |
ac40833ec94df33aa7f4725d0769ca076437d4f8992f0341808032ff10626fb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | jina-3.34.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 7.9 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
c5fe00a591afcda544c3b2eea1e34c4a460d67aa9c4a5a5b7784c17c7f6512ca
|
|
BLAKE2b-256 checksum How to use checksums |
c5c6e6292db78ceb5fe79ceb42a4e2ce065014d7584e3c6f2d302318ef89dd93
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | jina-3.34.0-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 8.4 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
2a0e693f058a2d7341fb088f26471bed4cb0b4511971a6c650d5bf98ea6b4dea
|
|
BLAKE2b-256 checksum How to use checksums |
17fdbd1cb9f17603360f826a1530339dc384e2c0d193bf45151ec1c3ba2f82d3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp311-cp311-macosx_10_9_x86_64.whl
| Download URL | jina-3.34.0-cp311-cp311-macosx_10_9_x86_64.whl |
|---|---|
| Size | 9.0 MB |
| Tags | CPython 3.11 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
aa38560538af23110d04d56e99c4348a0ca283ca3f6326c8af920c45751d585d
|
|
BLAKE2b-256 checksum How to use checksums |
7795649f68f55e372aa7e75e5e3c874789f5f9fc238d0ca1f698d762ae61e06d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | jina-3.34.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 7.9 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
4e0cc23d00e3872574fd700b9904b91ea850ee796035fd9a8465e9de2978fa2f
|
|
BLAKE2b-256 checksum How to use checksums |
df93e4bdaf545bf35bbd460d9c40f489673acc799d175d5c035dbd51fcbacfa5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | jina-3.34.0-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 8.4 MB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
9346acb3efa2a58819feb427562418f436eaa350571d9f9f2fad04d8d167c47c
|
|
BLAKE2b-256 checksum How to use checksums |
0dd560ff27eddeb30ad039271c053a001364bc7a51ea4ab6aeb59e20fc5cc2cb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp310-cp310-macosx_10_9_x86_64.whl
| Download URL | jina-3.34.0-cp310-cp310-macosx_10_9_x86_64.whl |
|---|---|
| Size | 9.0 MB |
| Tags | CPython 3.10 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
b2ebd1405f528c668fc9f1547bd0be08d3cc9e7131be05e9d956ec1406a8a44a
|
|
BLAKE2b-256 checksum How to use checksums |
c540c625bab7c541ff8efedcaa02088250b9ee233958d6e8d26c02e4f79e062b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | jina-3.34.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 7.9 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 Linux glibc 2.5+ x86-64 |
|
SHA-256 checksum How to use checksums |
4621cc1ce0873969d7691af446d065401546c6dd0a6752169d96019b5ecd7c09
|
|
BLAKE2b-256 checksum How to use checksums |
465a681ecf250cd66a7b0d57526c94817604d792cd7409a73e9fa28d5c6630c9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp39-cp39-macosx_11_0_arm64.whl
| Download URL | jina-3.34.0-cp39-cp39-macosx_11_0_arm64.whl |
|---|---|
| Size | 8.4 MB |
| Tags | CPython 3.9 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
7dcf83dd32f2ea6d9be7a68ac0b3caacddead1179329c8dfbcb0c5e2f3d6e5de
|
|
BLAKE2b-256 checksum How to use checksums |
40d22f9bcabf1c7cbce115748b6287c5c9db98fd68eeb242c081c18787356b9c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / jina-3.34.0-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | jina-3.34.0-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 9.0 MB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
4170a1c6e409cae6d95105caa55f770caeebaa698eb25a573701cfcf1a1061ed
|
|
BLAKE2b-256 checksum How to use checksums |
d32f64fcdf0de7fc860fde22214052973d3abae7cf994f8e0ec0d61b6dda1956
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Security for jina 3.34.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
