vis3 1.0.0a37

Visualize s3 data

English | 简体中文

OSS browser based on s3

A visualization tool designed for large language models and machine learning data. It supports cloud storage platforms with S3 protocol (AWS, Alibaba Cloud) and handles various data formats (json, jsonl.gz, warc.gz, md, etc.). Interactive visualization through JSON, Html, Markdown, and image views for efficient data analysis.

Features

• Supports multiple formats such as JSON, JSONL, WARC, intelligently recognizes data structures and visually presents key information, making data clear at a glance.

• One-click preview of any field, supports free switching between multiple view modes such as web pages, Markdown, images, simple and intuitive operation.

• Seamlessly connects with mainstream cloud storage platforms (Alibaba Cloud, AWS, Tencent Cloud, and other cloud storage platforms that support the S3 protocol), supports local file parsing, making data access easy.

https://github.com/user-attachments/assets/6bf58c28-9d18-4b9e-80bf-2449dde43e8e

Getting Started

# python >= 3.9.2
pip install vis3

Or create a Python environment using conda:

Install miniconda

# 1. Create Python 3.11 environment using conda
conda create -n vis3 python=3.11

# 2. Activate environment
conda activate vis3

# 3. Install vis3
pip install vis3

# 4. Launch
vis3

# ----------------

# Default launch doesn't require authentication, if you want to enable authentication to distinguish users
vis3 --auth

# Specify database (sqlite) directory
BASE_DATA_DIR=your/database/path vis3

# Enable authentication through ENABLE_AUTH
ENABLE_AUTH=true vis3

Local Development

conda create -n vis3-dev python=3.11

# Activate virtual environment
conda activate vis3-dev

# Install poetry
# https://python-poetry.org/docs/#installing-with-the-official-installer

# Install Python dependencies
poetry install

# Install frontend dependencies (install pnpm: https://pnpm.io/installation)
cd web && pnpm install

# Build frontend assets (in web directory)
pnpm build

# Start vis3
uvicorn vis3.main:app --reload

React Component

We provide a standalone React component via npm for customizing your data preview ui.

Community

Welcome to join the Opendatalab official WeChat group!

Related Projects

• LabelU-kit Web frontend annotation kit (LabelU is developed based on this kit)
• LabelLLM Open-source LLM dialogue annotation platform
• Miner U One-stop high-quality data extraction tool

License

This project is licensed under the Apache 2.0 license.