9DTact
9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation (RAL & ICRA'24)
Install / Use
/learn @linchangyi1/9DTactREADME
9DTact
Feel free to use this open-source project for commercial purposes!
Table of contents
Overview <a name="overview"></a>
This repository contains the code and the hardware source files for the paper:
<b>9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation</b> <br> Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu <br> RAL, 2023 <br> Website / Arxiv Paper / Video Tutorial / Bom (CN) / Production
Installation <a name="installation"></a>
Create a conda environment:
conda create -n 9dtact python=3.8
Install pytorch (choose the version that is compatible with your computer):
conda activate 9dtact
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
In this repository, install the other requirements:
pip install -e .
3D Shape Reconstruction <a name="reconstruction"></a>
For all the terminals used in this section, they are located in the shape_reconstruction directory and based on the 9dtact conda environment:
cd shape_reconstruction
conda activate 9dtact
If you are using the production version of 9DTact, you do not need to calibrate the camera and sensor. Please proceed directly to Step 3 (Shape Reconstruction).
1. Camera Calibration <a name="camera_calibration"></a>
Before getting started, adjust the camera focus by rotating the lens until objects approximately 15 mm away appear clear.
Then, 3d print the calibration board.<br> Run:
python _1_Camera_Calibration.py
Just follow the printed instructions.
2. Sensor Calibration <a name="sensor_calibration"></a>
Firstly, prepare a ball with a radius of 4.0 mm. (The radius of the ball depends on the thickness of the sensor surface. 4.0 mm is just a recommendation.)<br> Then, run:
python _2_Sensor_Calibration.py
Just follow the printed instructions.
3. Shape Reconstruction <a name="shape_reconstruction"></a>
python _3_Shape_Reconstruction.py
Press 'y' when the tactile image is stably captured, which will served as the reference image.
6D Force Estimation <a name="estimation"></a>
1. BOTA Driver <a name="bota"></a>
If you want to collect force data with a BOTA MiniONE Pro sensor, you need to:<br> Create a directory named 'bota_ws' as the ROS workspace, and install the bota driver package.
2. Data Collection <a name="collection"></a>
At the first terminal, open the BOTA sensor:
cd ~/xxx/bota_ws # Modify 'xxx' to enter the workspace directory
source devel/setup.bash
roslaunch rokubimini_serial rokubimini_serial.launch
At the second terminal, run:
source ~/xxx/bota_ws/devel/setup.bash
cd data_collection
conda activate 9dtact
python collect_data.py
At the third terminal, open the 9DTact sensor:
cd shape-force-ros
conda activate 9dtact
python _1_Sensor_ros.py
3. Data Processing <a name="processing"></a>
Open a terminal, normalize the wrench:
cd data_collection
conda activate 9dtact
python wrench_normalization.py # remember to modify the object_num
At the same terminal, split the data by running:
python split_train_test.py
and also:
python split_train_test(objects).py
4. Model Training <a name="training"></a>
To train the model on the stadard training dataset, run:
cd force_estimation
python train.py --model_name="Densenet" --model_layer=169 --optimizer="ADAM" --lrs=False --image_type="RGB" --cuda_index=6 --resize_img=False --train_mode=True --test_object=False --mixed_image=True --pretrained=False --batch_size=64 --num_epoch=200 --learning_rate=5.0e-4 --weight_decay=0.0
You may also choose to use Weights and Bias (wandb) by setting use_wandb as True, which helps to track the training performance.
5. Force Estimation <a name="inference"></a>
You need to specify a model saved in the 'saved_models' directory as an estimator, by modifying the 'weights' parameters in the force_config.yaml.<br> After that, run:
cd force_estimation
python _1_Force_Estimation.py
Run in ROS <a name="ros"></a>
1. Shape Reconstruction in ROS <a name="shape_ros"></a>
At the first terminal, open the 9DTact sensor:
cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _2_Shape_Reconstruction_ros.py
2. Force Estimation in ROS <a name="force_ros"></a>
At the first terminal, open the 9DTact sensor:
cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py
(Optional for visualization) At the third terminal, open the visualization window:
cd force_estimation
conda activate 9dtact
python force_visualizer.py
3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS <a name="shape_force"></a>
At the first terminal, open the force estimator:
cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _4_Shape_Force_ros.py
DTact Series Papers <a name="papers"></a>
- DTact: A Vision-Based Tactile Sensor that Measures High-Resolution 3D Geometry Directly from Darkness, Lin et al., ICRA 2023
- 9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation, Lin et al., RAL 2023
- Design and Evaluation of a Rapid Monolithic Manufacturing Technique for a Novel Vision-Based Tactile Sensor: C-Sight, Fan et al., MDPI Sensors 2024
- DTactive: A Vision-Based Tactile Sensor with Active Surface, Xu et al., IROS 2025
- VET: A Visual-Electronic Tactile System for Immersive Human-Machine Interaction, Zhang et al., arxiv 2025
- PP-Tac: Paper Picking Using Tactile Feedback in Dexterous Robotic Hands, Lin et al., RSS 2025
- AllTact Fin Ray: A Compliant Robot Gripper with Omni-Directional Tactile Sensing, Liang et al., arxiv 2025
- SuperMag: Vision-based Tactile Data Guided High-resolution Tactile Shape Reconstruction for Magnetic Tactile Sensors, Hou et al., IROS 2025
- UTact: Underwater Vision-Based Tactile Sensor with Geometry Reconstruction and Contact Force Estimation, Zhang et al., Advanced Robotics Research 2025
- exUMI: Extensible Robot Teaching System with Action-aware Task-agnostic Tactile Representation, Xu et al., CoRL 2025
- TacScope: A Miniaturized Vision-Based Tactile Sensor for Surgical Applications, Prince et al., Advanced Robotics Research 2025
- Design and application of multimodal visual-tactile sensor for object information perception, Wang et al., Sensors and Actuators 2026
- A Low-Cost Vision-Based Tactile Gripper with Pretraining Learning for Contact-Rich Manipulation, Liu et al., arxiv 2026
- SpikingTac: A Miniaturized Neuromorphic Visuotactile Sensor for High-Precision Dynamic Tactile Imprint Tracking, Jiang et al., arxiv 2026
Reference
@inproceedings{lin2023dtact,
title={Dtact: A vision-based tactile sensor that measures high-resolution 3d geometry directly from darkness},
author={Lin, Changyi and Lin, Ziqi and Wang, Shaoxiong and Xu, Huazhe},
booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
pages={10359--10366},
year={2023},
organization={IEEE}
}
@article{lin20239dtact,
title={9dtact: A compact vision-based tactile sensor for accurate 3d shape reconstruction and generalizable 6d force estimation},
author={Lin, Changyi and Zhang, Han and Xu, Jikai and Wu, Lei and Xu, Huazhe},
journal={IEEE Robotics and Automation Letters},
volume={9},
number={2},
pages={923--930},
year={2023},
publisher={IEEE}
}
