Skip to main content

Fury is a blazing fast multi-language serialization framework powered by jit, vectorization and zero-copy

Project description


Build Status Twitter

Fury is a blazing-fast multi-language serialization framework powered by jit(just-in-time compilation) and zero-copy, providing up to 170x performance and ultimate ease of use.

https://furyio.org

Features

  • Multiple languages: Java/Python/C++/Golang/JavaScript/Rust/Scala/TypeScript.
  • Zero-copy: Cross-language out-of-band serialization inspired by pickle5 and off-heap read/write.
  • High performance: A highly-extensible JIT framework to generate serializer code at runtime in an async multi-thread way to speed serialization, providing 20-170x speed up by:
    • reduce memory access by inlining variables in generated code.
    • reduce virtual method invocation by inline call in generated code.
    • reduce conditional branching.
    • reduce hash lookup.
  • Multiple binary protocols: Object graph, row format, and so on.

In addition to cross-language serialization, Fury also features at:

  • Drop-in replace Java serialization frameworks such as JDK/Kryo/Hessian without modifying any code, but 100x faster. It can greatly improve the efficiency of high-performance RPC calls, data transfer, and object persistence.
  • 100% compatible with JDK serialization API with much faster implementation: supporting JDK writeObject/readObject/writeReplace/readResolve/readObjectNoData/Externalizable API.
  • Supports Java 8~21, Java 17+ record is supported too.
  • Support AOT compilation serialization for GraalVM native image, and no reflection/serialization json config are needed.
  • Supports shared and circular reference object serialization for golang.
  • Supports scala serialization
  • Supports automatic object serialization for golang.

Protocols

Different scenarios have different serialization requirements. Fury designed and implemented multiple binary protocols for those requirements:

  • Cross-language object graph protocol:
    • Cross-language serialize any object automatically, no need for IDL definition, schema compilation and object to/from protocol conversion.
    • Support shared reference and circular reference, no duplicate data or recursion error.
    • Support object polymorphism.
  • Native java/python object graph protocol: Highly-optimized based on type system of the language.
  • Row format protocol: A cache-friendly binary random access format, supports skipping serialization and partial serialization, and can convert to column-format automatically.

New protocols can be easily added based on fury existing buffer, encoding, meta, codegen and other capabilities. All of those share the same codebase, and the optimization for one protocol can be reused by another protocol.

Benchmarks

Different serialization frameworks are suitable for different scenarios, and benchmark results here are for reference only.

If you need to benchmark for your specific scenario, make sure all serialization frameworks are appropriately configured for that scenario.

Dynamic serialization frameworks support polymorphism and references, but they often come with a higher cost compared to static serialization frameworks, unless they utilize JIT techniques like Fury does. Because Fury generates code at runtime, it is recommended to warm up the system before collecting benchmark statistics.

Java Serialization

Title containing "compatible" represent schema compatible mode: support type forward/backward compatibility.

Title without "compatible" represent schema consistent mode: class schema must be the same between serialization and deserialization.

Struct is a class with 100 primitive fields, MediaContent is a class from jvm-serializers, Sample is a class from kryo benchmark.

See benchmarks for more benchmarks about type forward/backward compatibility, off-heap support, zero-copy serialization.

Installation

Java

Nightly snapshot:

<repositories>
  <repository>
    <id>sonatype</id>
    <url>https://s01.oss.sonatype.org/content/repositories/snapshots</url>
    <releases>
      <enabled>false</enabled>
    </releases>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
</repositories>
<dependency>
  <groupId>org.furyio</groupId>
  <artifactId>fury-core</artifactId>
  <version>0.5.0-SNAPSHOT</version>
</dependency>
<!-- row/arrow format support -->
<!-- <dependency>
  <groupId>org.furyio</groupId>
  <artifactId>fury-format</artifactId>
  <version>0.5.0-SNAPSHOT</version>
</dependency> -->

Release version:

<dependency>
  <groupId>org.furyio</groupId>
  <artifactId>fury-core</artifactId>
  <version>0.4.1</version>
</dependency>
<!-- row/arrow format support -->
<!-- <dependency>
  <groupId>org.furyio</groupId>
  <artifactId>fury-format</artifactId>
  <version>0.4.1</version>
</dependency> -->

Scala

libraryDependencies += "org.furyio" % "fury-core" % "0.4.1"

Python

# Release version will be provided in the future.
pip install pyfury --pre

JavaScript

npm install @furyjs/fury

Golang

go get github.com/alipay/fury/go/fury

Quickstart

Here we give a quick start about how to use fury, see user guide for more details about java, cross language, and row format.

Fury java object graph serialization

If you don't have cross-language requirements, using this mode will have better performance.

import io.fury.*;
import io.fury.config.*;
import java.util.*;

public class Example {
  public static void main(String[] args) {
    SomeClass object = new SomeClass();
    // Note that Fury instances should be reused between 
    // multiple serializations of different objects.
    {
      Fury fury = Fury.builder().withLanguage(Language.JAVA)
        // Allow to deserialize objects unknown types, more flexible 
        // but may be insecure if the classes contains malicious code.
        .requireClassRegistration(false)
        .build();
      // Registering types can reduce class name serialization overhead, but not mandatory.
      // If class registration enabled, all custom types must be registered.
      fury.register(SomeClass.class);
      byte[] bytes = fury.serialize(object);
      System.out.println(fury.deserialize(bytes));
    }
    {
      ThreadSafeFury fury = Fury.builder().withLanguage(Language.JAVA)
        // Allow to deserialize objects unknown types, more flexible 
        // but may be insecure if the classes contains malicious code.
        .requireClassRegistration(false)
        .buildThreadSafeFury();
      byte[] bytes = fury.serialize(object);
      System.out.println(fury.deserialize(bytes));
    }
    {
      ThreadSafeFury fury = new ThreadLocalFury(classLoader -> {
        Fury f = Fury.builder().withLanguage(Language.JAVA)
          .withClassLoader(classLoader).build();
        f.register(SomeClass.class);
        return f;
      });
      byte[] bytes = fury.serialize(object);
      System.out.println(fury.deserialize(bytes));
    }
  }
}

Cross-language object graph serialization

Java

import io.fury.*;
import io.fury.config.*;
import java.util.*;

public class ReferenceExample {
  public static class SomeClass {
    SomeClass f1;
    Map<String, String> f2;
    Map<String, String> f3;
  }

  public static Object createObject() {
    SomeClass obj = new SomeClass();
    obj.f1 = obj;
    obj.f2 = ofHashMap("k1", "v1", "k2", "v2");
    obj.f3 = obj.f2;
    return obj;
  }

  // mvn exec:java -Dexec.mainClass="io.fury.examples.ReferenceExample"
  public static void main(String[] args) {
    Fury fury = Fury.builder().withLanguage(Language.XLANG)
      .withRefTracking(true).build();
    fury.register(SomeClass.class, "example.SomeClass");
    byte[] bytes = fury.serialize(createObject());
    // bytes can be data serialized by other languages.
    System.out.println(fury.deserialize(bytes));
  }
}

Python

from typing import Dict
import pyfury

class SomeClass:
    f1: "SomeClass"
    f2: Dict[str, str]
    f3: Dict[str, str]

fury = pyfury.Fury(ref_tracking=True)
fury.register_class(SomeClass, type_tag="example.SomeClass")
obj = SomeClass()
obj.f2 = {"k1": "v1", "k2": "v2"}
obj.f1, obj.f3 = obj, obj.f2
data = fury.serialize(obj)
# bytes can be data serialized by other languages.
print(fury.deserialize(data))

Golang

package main

import furygo "github.com/alipay/fury/go/fury"
import "fmt"

func main() {
	type SomeClass struct {
		F1 *SomeClass
		F2 map[string]string
		F3 map[string]string
	}
	fury := furygo.NewFury(true)
	if err := fury.RegisterTagType("example.SomeClass", SomeClass{}); err != nil {
		panic(err)
	}
	value := &SomeClass{F2: map[string]string{"k1": "v1", "k2": "v2"}}
	value.F3 = value.F2
	value.F1 = value
	bytes, err := fury.Marshal(value)
	if err != nil {
	}
	var newValue interface{}
	// bytes can be data serialized by other languages.
	if err := fury.Unmarshal(bytes, &newValue); err != nil {
		panic(err)
	}
	fmt.Println(newValue)
}

Row format

Java

public class Bar {
  String f1;
  List<Long> f2;
}

public class Foo {
  int f1;
  List<Integer> f2;
  Map<String, Integer> f3;
  List<Bar> f4;
}

RowEncoder<Foo> encoder = Encoders.bean(Foo.class);
Foo foo = new Foo();
foo.f1 = 10;
foo.f2 = IntStream.range(0, 1000000).boxed().collect(Collectors.toList());
foo.f3 = IntStream.range(0, 1000000).boxed().collect(Collectors.toMap(i -> "k"+i, i->i));
List<Bar> bars = new ArrayList<>(1000000);
for (int i = 0; i < 1000000; i++) {
  Bar bar = new Bar();
  bar.f1 = "s"+i;
  bar.f2 = LongStream.range(0, 10).boxed().collect(Collectors.toList());
  bars.add(bar);
}
foo.f4 = bars;
// Can be zero-copy read by python
BinaryRow binaryRow = encoder.toRow(foo);
// can be data from python
Foo newFoo = encoder.fromRow(binaryRow);
// zero-copy read List<Integer> f2
BinaryArray binaryArray2 = binaryRow.getArray(1);
// zero-copy read List<Bar> f4
BinaryArray binaryArray4 = binaryRow.getArray(3);
// zero-copy read 11th element of `readList<Bar> f4`
BinaryRow barStruct = binaryArray4.getStruct(10);

// zero-copy read 6th of f2 of 11th element of `readList<Bar> f4`
barStruct.getArray(1).getLong(5);
RowEncoder<Bar> barEncoder = Encoders.bean(Bar.class);
// deserialize part of data.
Bar newBar = barEncoder.fromRow(barStruct);
Bar newBar2 = barEncoder.fromRow(binaryArray4.getStruct(20));

Python

@dataclass
class Bar:
    f1: str
    f2: List[pa.int64]
@dataclass
class Foo:
    f1: pa.int32
    f2: List[pa.int32]
    f3: Dict[str, pa.int32]
    f4: List[Bar]

encoder = pyfury.encoder(Foo)
foo = Foo(f1=10, f2=list(range(1000_000)),
         f3={f"k{i}": i for i in range(1000_000)},
         f4=[Bar(f1=f"s{i}", f2=list(range(10))) for i in range(1000_000)])
binary: bytes = encoder.to_row(foo).to_bytes()
foo_row = pyfury.RowData(encoder.schema, binary)
print(foo_row.f2[100000], foo_row.f4[100000].f1, foo_row.f4[200000].f2[5])

Compatibility

Schema Compatibility

Fury java object graph serialization support class schema forward/backward compatibility. The serialization peer and deserialization peer can add/delete fields independently.

We plan to add support cross-language serialization after meta compression is finished.

Binary Compatibility

We are still improving our protocols, binary compatibility is not ensured between fury major releases for now. it's ensured between minor versions only. Please versioning your data by fury major version if you will upgrade fury in the future, see how to upgrade fury for further details.

Binary compatibility will be ensured when fury 1.0 is released.

Security

Static serialization is secure. But dynamic serialization such as fury java/python native serialization supports deserializing unregistered types, which provides more dynamics and flexibility, but also introduce security risks.

For example, the deserialization may invoke init constructor or equals/hashCode method, if the method body contains malicious code, the system will be at risk.

Fury provides a class registration option that is enabled by default for such protocols, allowing only deserialization of trusted registered types or built-in types. Do not disable class registration unless you can ensure your environment is secure.

If this option is disabled, you are responsible for serialization security. You can configure io.fury.resolver.ClassChecker by ClassResolver#setClassChecker to control which classes are allowed for serialization.

How to Build

Please read the BUILD guide for instructions on how to build.

How to Contribute

Please read the CONTRIBUTING guide for instructions on how to contribute.

For ecosystem projects, please see https://github.com/fury-project

License

Licensed under the Apache License, Version 2.0

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

pyfury-0.4.1-cp312-cp312-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.12

pyfury-0.4.1-cp312-cp312-macosx_12_0_x86_64.whl (959.2 kB view details)

Uploaded CPython 3.12 macOS 12.0+ x86-64

pyfury-0.4.1-cp311-cp311-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.11

pyfury-0.4.1-cp311-cp311-macosx_12_0_x86_64.whl (973.7 kB view details)

Uploaded CPython 3.11 macOS 12.0+ x86-64

pyfury-0.4.1-cp310-cp310-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.10

pyfury-0.4.1-cp310-cp310-macosx_12_0_x86_64.whl (972.0 kB view details)

Uploaded CPython 3.10 macOS 12.0+ x86-64

pyfury-0.4.1-cp39-cp39-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.9

pyfury-0.4.1-cp39-cp39-macosx_12_0_x86_64.whl (975.2 kB view details)

Uploaded CPython 3.9 macOS 12.0+ x86-64

pyfury-0.4.1-cp38-cp38-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.8

pyfury-0.4.1-cp38-cp38-macosx_12_0_x86_64.whl (970.6 kB view details)

Uploaded CPython 3.8 macOS 12.0+ x86-64

pyfury-0.4.1-cp37-cp37m-manylinux1_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.7m

pyfury-0.4.1-cp37-cp37m-macosx_12_0_x86_64.whl (952.2 kB view details)

Uploaded CPython 3.7m macOS 12.0+ x86-64

File details

Details for the file pyfury-0.4.1-cp312-cp312-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp312-cp312-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 54c4bf0dc2c5beef01a1eeb8dc07d2e1244c12e2fee803d64f07eea98c268927
MD5 7c388d1163d8cdd8994ad98efa380bff
BLAKE2b-256 7dfab7abbdcdf5c494baca476235ed2bf8363db1d70da120aed2538356c4a0e9

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp312-cp312-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp312-cp312-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 9d8783a032c7590b73f80c573a395e61a0117bee779159316d05a4d4a6ec9bbe
MD5 aaa1a74f13b0ed34397e0a6c071d59f1
BLAKE2b-256 610b71055c932023b04349171fc4aead9f547e47ed2192ecb690e8390409dae4

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp311-cp311-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp311-cp311-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 c605b214e003fb2cd9f7b4916d3c395cf4ef021760d9b30c4bd36e68cb43382e
MD5 7674f5da60edd520bdf9ba561217796b
BLAKE2b-256 249769ee3cb055477e9e08ed162c13cc9362a0c8b6a0a311d70b274c16566641

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp311-cp311-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp311-cp311-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 1d3b1a1f34c89c17c6081f50dce8a22f42e6c26cd12f15d12629799eadedeb60
MD5 4d2f7bd6d6d1853e10b69c05a2516c14
BLAKE2b-256 c691374a9dcd83b2ca3b4e1da2399ef321218bfc1b15b77da379a682d0ee10d5

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp310-cp310-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp310-cp310-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 c8133ac9fb73f356de3995cb14f0647d7070fbf1d8c114b42b0d7a3699eff793
MD5 e3a04a008cbf03ec074166be6b3c2df0
BLAKE2b-256 1c64759d6cbabe3dea9d30034ce3519b18794535320a7fa4646d032ee2a7d35b

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp310-cp310-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp310-cp310-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 9288de26152d5841f06cd0fa2a7e273b71957c18e7dabf61a55b29fd10c8aa8f
MD5 464d8677794cb9579469068ad568813e
BLAKE2b-256 008c5bd90651aba3e8d858c59b1a438e2c933958904696a04d12ce94a3276ea8

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp39-cp39-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp39-cp39-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 ce95d97ac1411312152f09daaf1f60b378acb33afcca61dfdab1f5375c8d2e73
MD5 118a48e38cd0ed19c648de0d3b8f9290
BLAKE2b-256 94fe156a50427c814bdc041b098b011ade55d4bd5c76b74a41aad0ffa8127852

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp39-cp39-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp39-cp39-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 c2cb3e5bc5b06b2543dcc70c68d6ff8182c5d7b1c12ebda53ce39f780d58e826
MD5 652dcb400cc927c2dbe965bb2ba38fd5
BLAKE2b-256 84e669a31e652a3ba0c66b094281996287e4beb484dc6bd1886cf894688d953a

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp38-cp38-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp38-cp38-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 d928cb07114559e9d0136c3db9438d0902a2d61db7be8738b219c889b87a5b69
MD5 9f20f6ba20e7c5c07d2cad7b7f0cbb7a
BLAKE2b-256 131728a4b56ba4f2f5937ee7be3561a03f9bac470c830075057dfa8a71d6a562

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp38-cp38-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp38-cp38-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 0531eb8cc02d5b67d8622f47eb4039861cb4697a87ee6db2a98a2e28a7697d51
MD5 019f4fe21ad941ac6ca6bbf93c404d92
BLAKE2b-256 7a24e6d5188d1b1692cd1cff9bf786f7b63353e5dcb6f7690e78bb362327aed4

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp37-cp37m-manylinux1_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp37-cp37m-manylinux1_x86_64.whl
Algorithm Hash digest
SHA256 4d31de96ff99307ff9d3d7b245d1078cd25689cd483bb2e1a442c73999556740
MD5 a0d08037d38aef6ea469430137593a6a
BLAKE2b-256 74b67581104cd0e36b2b07e82c2681f8d27bf97af64a8603ebcc51ecedf22cf8

See more details on using hashes here.

File details

Details for the file pyfury-0.4.1-cp37-cp37m-macosx_12_0_x86_64.whl.

File metadata

File hashes

Hashes for pyfury-0.4.1-cp37-cp37m-macosx_12_0_x86_64.whl
Algorithm Hash digest
SHA256 13c5489763aaa9559c5d76264d09d09892355aa7eb4a4a36c3dffd34abdbff0c
MD5 e46c912cd3c2a56aace000a16a2f1ee2
BLAKE2b-256 657719eec0177a72928ea0495b76714d111795fb4ee1e29f969f7ba56d5cbb5b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page