Learning3D: A Modern Library for Deep Learning on 3D Point Clouds Data
Project description
Learning3D: A Modern Library for Deep Learning on 3D Point Clouds Data.
Documentation | Blog | Demo
Learning3D is an open-source library that supports the development of deep learning algorithms that deal with 3D data. The Learning3D exposes a set of state of art deep neural networks in python. A modular code has been provided for further development. We welcome contributions from the open-source community.
Latest News:
- [24 Oct, 2023]: MaskNet++ is now a part of learning3d library.
- [12 May, 2022]: ChamferDistance loss function is incorporated in learning3d. This is a purely pytorch based loss function.
- [24 Dec. 2020]: MaskNet is now ready to enhance the performance of registration algorithms in learning3d for occluded point clouds.
- [24 Dec. 2020]: Loss based on the predicted and ground truth correspondences is added in learning3d after consideration of Correspondence Matrices are Underrated paper.
- [24 Dec. 2020]: PointConv, latent feature estimation using convolutions on point clouds is now available in learning3d.
- [16 Oct. 2020]: DeepGMR, registration using gaussian mixture models is now available in learning3d
- [14 Oct. 2020]: Now, use your own data in learning3d. (Check out UserData functionality!)
Available Computer Vision Algorithms in Learning3D
Sr. No. | Tasks | Algorithms |
---|---|---|
1 | Classification | PointNet, DGCNN, PPFNet, PointConv |
2 | Segmentation | PointNet, DGCNN |
3 | Reconstruction | Point Completion Network (PCN) |
4 | Registration | PointNetLK, PCRNet, DCP, PRNet, RPM-Net, DeepGMR |
5 | Flow Estimation | FlowNet3D |
6 | Inlier Estimation | MaskNet, MaskNet++ |
Available Pretrained Models
- PointNet
- PCN
- PointNetLK
- PCRNet
- DCP
- PRNet
- FlowNet3D
- RPM-Net (clean-trained.pth, noisy-trained.pth, partial-pretrained.pth)
- DeepGMR
- PointConv (Download from this link)
- MaskNet
- MaskNet++ / MaskNet2
Available Datasets
- ModelNet40
Available Loss Functions
- Classification Loss (Cross Entropy)
- Registration Losses (FrobeniusNormLoss, RMSEFeaturesLoss)
- Distance Losses (Chamfer Distance, Earth Mover's Distance)
- Correspondence Loss (based on this paper)
Technical Details
Supported OS
- Ubuntu 16.04
- Ubuntu 18.04
- Ubuntu 20.04.6
- Linux Mint
Requirements
- CUDA 10.0 or higher
- Pytorch 1.3 or higher
- Python 3.8
How to use this library?
Important Note: Clone this repository in your project. Please don't add your codes in "learning3d" folder.
- All networks are defined in the module "models".
- All loss functions are defined in the module "losses".
- Data loaders are pre-defined in data_utils/dataloaders.py file.
- All pretrained models are provided in learning3d/pretrained folder.
Documentation
B: Batch Size, N: No. of points and C: Channels.
Use of Point Embedding Networks:
from learning3d.models import PointNet, DGCNN, PPFNet
pn = PointNet(emb_dims=1024, input_shape='bnc', use_bn=False)
dgcnn = DGCNN(emb_dims=1024, input_shape='bnc')
ppf = PPFNet(features=['ppf', 'dxyz', 'xyz'], emb_dims=96, radius='0.3', num_neighbours=64)
Sr. No. | Variable | Data type | Shape | Choices | Use |
---|---|---|---|---|---|
1. | emb_dims | Integer | Scalar | 1024, 512 | Size of feature vector for the each point |
2. | input_shape | String | - | 'bnc', 'bcn' | Shape of input point cloud |
3. | output | tensor | BxCxN | - | High dimensional embeddings for each point |
4. | features | List of Strings | - | ['ppf', 'dxyz', 'xyz'] | Use of various features |
5. | radius | Float | Scalar | 0.3 | Radius of cluster for computing local features |
6. | num_neighbours | Integer | Scalar | 64 | Maximum number of points to consider per cluster |
Use of Classification / Segmentation Network:
from learning3d.models import Classifier, PointNet, Segmentation
classifier = Classifier(feature_model=PointNet(), num_classes=40)
seg = Segmentation(feature_model=PointNet(), num_classes=40)
Sr. No. | Variable | Data type | Shape | Choices | Use |
---|---|---|---|---|---|
1. | feature_model | Object | - | PointNet / DGCNN | Point cloud embedding network |
2. | num_classes | Integer | Scalar | 10, 40 | Number of object categories to be classified |
3. | output | tensor | Classification: Bx40, Segmentation: BxNx40 | 10, 40 | Probabilities of each category or each point |
Use of Registration Networks:
from learning3d.models import PointNet, PointNetLK, DCP, iPCRNet, PRNet, PPFNet, RPMNet
pnlk = PointNetLK(feature_model=PointNet(), delta=1e-02, xtol=1e-07, p0_zero_mean=True, p1_zero_mean=True, pooling='max')
dcp = DCP(feature_model=PointNet(), pointer_='transformer', head='svd')
pcrnet = iPCRNet(feature_moodel=PointNet(), pooling='max')
rpmnet = RPMNet(feature_model=PPFNet())
deepgmr = DeepGMR(use_rri=True, feature_model=PointNet(), nearest_neighbors=20)
Sr. No. | Variable | Data type | Choices | Use | Algorithm |
---|---|---|---|---|---|
1. | feature_model | Object | PointNet / DGCNN | Point cloud embedding network | PointNetLK |
2. | delta | Float | Scalar | Parameter to calculate approximate jacobian | PointNetLK |
3. | xtol | Float | Scalar | Check tolerance to stop iterations | PointNetLK |
4. | p0_zero_mean | Boolean | True/False | Subtract mean from template point cloud | PointNetLK |
5. | p1_zero_mean | Boolean | True/False | Subtract mean from source point cloud | PointNetLK |
6. | pooling | String | 'max' / 'avg' | Type of pooling used to get global feature vectror | PointNetLK |
7. | pointer_ | String | 'transformer' / 'identity' | Choice for Transformer/Attention network | DCP |
8. | head | String | 'svd' / 'mlp' | Choice of module to estimate registration params | DCP |
9. | use_rri | Boolean | True/False | Use nearest neighbors to estimate point cloud features. | DeepGMR |
10. | nearest_neighbores | Integer | 20/any integer | Give number of nearest neighbors used to estimate features | DeepGMR |
Use of Inlier Estimation Network (MaskNet):
from learning3d.models import MaskNet, PointNet, MaskNet2
masknet = MaskNet(feature_model=PointNet(), is_training=True) masknet2 = MaskNet2(feature_model=PointNet(), is_training=True)
Sr. No. | Variable | Data type | Choices | Use |
---|---|---|---|---|
1. | feature_model | Object | PointNet / DGCNN | Point cloud embedding network |
2. | is_training | Boolean | True / False | Specify if the network will undergo training or testing |
Use of Point Completion Network:
from learning3d.models import PCN
pcn = PCN(emb_dims=1024, input_shape='bnc', num_coarse=1024, grid_size=4, detailed_output=True)
Sr. No. | Variable | Data type | Choices | Use |
---|---|---|---|---|
1. | emb_dims | Integer | 1024, 512 | Size of feature vector for each point |
2. | input_shape | String | 'bnc' / 'bcn' | Shape of input point cloud |
3. | num_coarse | Integer | 1024 | Shape of output point cloud |
4. | grid_size | Integer | 4, 8, 16 | Size of grid used to produce detailed output |
5. | detailed_output | Boolean | True / False | Choice for additional module to create detailed output point cloud |
Use of PointConv:
Use the following to create pretrained model provided by authors.
from learning3d.models import create_pointconv
PointConv = create_pointconv(classifier=True, pretrained='path of checkpoint')
ptconv = PointConv(emb_dims=1024, input_shape='bnc', input_channel_dim=6, classifier=True)
OR
Use the following to create your own PointConv model.
PointConv = create_pointconv(classifier=False, pretrained=None)
ptconv = PointConv(emb_dims=1024, input_shape='bnc', input_channel_dim=3, classifier=True)
PointConv variable is a class. Users can use it to create a sub-class to override create_classifier and create_structure methods in order to change PointConv's network architecture.
Sr. No. | Variable | Data type | Choices | Use |
---|---|---|---|---|
1. | emb_dims | Integer | 1024, 512 | Size of feature vector for each point |
2. | input_shape | String | 'bnc' / 'bcn' | Shape of input point cloud |
3. | input_channel_dim | Integer | 3/6 | Define if point cloud contains only xyz co-ordinates or normals and colors as well |
4. | classifier | Boolean | True / False | Choose if you want to use a classifier with PointConv |
5. | pretrained | Boolean | String | Give path of the pretrained classifier model (only use it for weights given by authors) |
Use of Flow Estimation Network:
from learning3d.models import FlowNet3D
flownet = FlowNet3D()
Use of Data Loaders:
from learning3d.data_utils import ModelNet40Data, ClassificationData, RegistrationData, FlowData
modelnet40 = ModelNet40Data(train=True, num_points=1024, download=True)
classification_data = ClassificationData(data_class=ModelNet40Data())
registration_data = RegistrationData(algorithm='PointNetLK', data_class=ModelNet40Data(), partial_source=False, partial_template=False, noise=False)
flow_data = FlowData()
Sr. No. | Variable | Data type | Choices | Use |
---|---|---|---|---|
1. | train | Boolean | True / False | Split data as train/test set |
2. | num_points | Integer | 1024 | Number of points in each point cloud |
3. | download | Boolean | True / False | If data not available then download it |
4. | data_class | Object | - | Specify which dataset to use |
5. | algorithm | String | 'PointNetLK', 'PCRNet', 'DCP', 'iPCRNet' | Algorithm used for registration |
6. | partial_source | Boolean | True / False | Create partial source point cloud |
7. | partial_template | Boolean | True / False | Create partial template point cloud |
8. | noise | Boolean | True / False | Add noise in source point cloud |
Use Your Own Data:
from learning3d.data_utils import UserData
dataset = UserData(application, data_dict)
Sr. No. | Application | Required Key | Respective Value |
---|---|---|---|
1. | 'classification' | 'pcs' | Point Clouds (BxNx3) |
'labels' | Ground Truth Class Labels (BxN) | ||
2. | 'registration' | 'template' | Template Point Clouds (BxNx3) |
'source' | Source Point Clouds (BxNx3) | ||
'transformation' | Ground Truth Transformation (Bx4x4) | ||
3. | 'flow_estimation' | 'frame1' | Point Clouds (BxNx3) |
'frame2' | Point Clouds (BxNx3) | ||
'flow' | Ground Truth Flow Vector (BxNx3) |
Use of Loss Functions:
from learning3d.losses import RMSEFeaturesLoss, FrobeniusNormLoss, ClassificationLoss, EMDLoss, ChamferDistanceLoss, CorrespondenceLoss
rmse = RMSEFeaturesLoss()
fn_loss = FrobeniusNormLoss()
classification_loss = ClassificationLoss()
emd = EMDLoss()
cd = ChamferDistanceLoss()
corr = CorrespondenceLoss()
Sr. No. | Loss Type | Use |
---|---|---|
1. | RMSEFeaturesLoss | Used to find root mean square value between two global feature vectors of point clouds |
2. | FrobeniusNormLoss | Used to find frobenius norm between two transfromation matrices |
3. | ClassificationLoss | Used to calculate cross-entropy loss |
4. | EMDLoss | Earth Mover's distance between two given point clouds |
5. | ChamferDistanceLoss | Chamfer's distance between two given point clouds |
6. | CorrespondenceLoss | Computes cross entropy loss using the predicted correspondence and ground truth correspondence for each source point |
To run codes from examples:
- Copy the file from "examples" folder outside of the directory "learning3d"
- Now, run the file. (ex. python test_pointnet.py)
- Your Directory/Location
- learning3d
- test_pointnet.py
References:
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
- Dynamic Graph CNN for Learning on Point Clouds
- PPFNet: Global Context Aware Local Features for Robust 3D Point Matching
- PointConv: Deep Convolutional Networks on 3D Point Clouds
- PointNetLK: Robust & Efficient Point Cloud Registration using PointNet
- PCRNet: Point Cloud Registration Network using PointNet Encoding
- Deep Closest Point: Learning Representations for Point Cloud Registration
- PRNet: Self-Supervised Learning for Partial-to-Partial Registration
- FlowNet3D: Learning Scene Flow in 3D Point Clouds
- PCN: Point Completion Network
- RPM-Net: Robust Point Matching using Learned Features
- 3D ShapeNets: A Deep Representation for Volumetric Shapes
- DeepGMR: Learning Latent Gaussian Mixture Models for Registration
- CMU: Correspondence Matrices are Underrated
- MaskNet: A Fully-Convolutional Network to Estimate Inlier Points
- MaskNet++: Inlier/outlier identification for two point clouds
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for learning3d-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f35ff7e3ffdc37f7d8e03f887c84060d0f7ff2a1d3ffd63f6841e79931579978 |
|
MD5 | 86c7fc765c4b69bbd06864e347e9516c |
|
BLAKE2b-256 | ea0e3950e63b7fcbff04ea9a025c3d10e1516c2d269ca2906bd5ed05c4f50300 |