PyDec is a linear decomposition toolkit for neural network based on PyTorch, which can decompose the tensor in the forward process into given components with a small amount of code. The result of decomposition can be applied to tasks such as attribution analysis.
Features:
- Fast. Compute decomposition in foward process and benefit from GPU acceleration.
- Run once, decompose anywhere. Obtain the decomposition of all hidden states (if you saved them) in forward propagation.
- Applicable to networks such as Transformer, CNN and RNN.
Examples
Data flow visualization
Requirements and Installation
- PyTorch version >= 1.11.0
- Python version >= 3.7
- To install PyDec and develop locally:
git clone https://github.com/DoubleVII/pydec
cd pydec
pip install --editable ./
- To install the latest stable release:
pip install pydec
Getting Started
Example: deompose a tiny network
As a simple example, here's a very simple model with two linear layers and an activation function. We'll create an instance of it and get the decomposition of the output:
import torch
class TinyModel(torch.nn.Module):
def __init__(self):
super(TinyModel, self).__init__()
self.linear1 = torch.nn.Linear(4, 10)
self.activation = torch.nn.ReLU()
self.linear2 = torch.nn.Linear(10, 2)
def forward(self, x):
x = self.linear1(x)
x = self.activation(x)
x = self.linear2(x)
return x
tinymodel = TinyModel()
Given an input x, the output of the model is:
x = torch.rand(4)
print("Input tensor:")
print(x)
print("\n\nOutput tensor:")
print(tinymodel(x))
Out:
Input tensor:
tensor([0.7023, 0.3492, 0.7771, 0.0157])
Output tensor:
tensor([0.2751, 0.3626], grad_fn=<AddBackward0>)
To decompose the output, just input the Composition initialized from x:
c = pydec.zeros(x.size(), c_num=x.size(0))
c = pydec.diagonal_init(c, src=x, dim=0)
print("Input composition:")
print(c)
c_out = tinymodel(c)
print("\n\nOutput composition:")
print(c_out)
Out:
Input composition:
composition{
components:
tensor([0.7023, 0.0000, 0.0000, 0.0000]),
tensor([0.0000, 0.3492, 0.0000, 0.0000]),
tensor([0.0000, 0.0000, 0.7771, 0.0000]),
tensor([0.0000, 0.0000, 0.0000, 0.0157]),
residual:
tensor([0., 0., 0., 0.])}
Output composition:
composition{
components:
tensor([-0.0418, -0.0296]),
tensor([0.0566, 0.0332]),
tensor([0.1093, 0.1147]),
tensor([ 0.0015, -0.0018]),
residual:
tensor([0.1497, 0.2461]),
grad_fn=<AddBackward0>}
Each component of the output composition represents the contribution of each feature in x to the output.
Summing each component yields the tensor of original output:
print("Sum of each component:")
print(c_out.c_sum())
Out:
Sum of each component:
tensor([0.2751, 0.3626], grad_fn=<AddBackward0>)
Documentation
The full documentation contains examples of implementations on real-world models, tutorials, notes and Python API descriptions.
Linear Decomposition Theory
To understand the principles and theories behind PyDec, see our paper Local Interpretation of Transformer Based on Linear Decomposition.
Metadata
Release files for pydec 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pydec-0.3.0.tar.gz | 49.5 kB | Without provenance | Details for pydec-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pydec-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for pydec-0.3.0-py3-none-any.whl |
Total release size: 103.2 kB
Release files / pydec-0.3.0.tar.gz
| Download URL | pydec-0.3.0.tar.gz |
|---|---|
| Size | 49.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3f8961bdc6203fd98f50564d89ad976b51f4a6bf604689ad1478d24d7d1882ad
|
|
BLAKE2b-256 checksum How to use checksums |
d6b1281f0f1c2bc508652d0b44db6d6d5f98d953caa2d501af73057c4594e1b6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Release files / pydec-0.3.0-py3-none-any.whl
| Download URL | pydec-0.3.0-py3-none-any.whl |
|---|---|
| Size | 53.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e615e8fb798f2e1f6cd86ce0a7be69f906d2880da472dc04fea85c2d0f39395a
|
|
BLAKE2b-256 checksum How to use checksums |
1528f2a150bc1fcd8767cb66c3fe0c1452ba89a1106f140b11899d217530e01c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.17
|
Security for pydec 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.