v2e

Python torch + opencv code to go from conventional stroboscopic video frames with low frame rate into realistic synthetic DVS event streams with much higher effective timing precision. v2e includes finite intensity-depenedent photoreceptor bandwidth, Gaussian pixel to pixel event threshold variation, and noise 'leak' events.

See the v2e home page for videos and further information.

Our paper (below) about v2e debunks incorrect assertions about event cameras that pervade the current computer vision literature.

See v2e-sample-input-data for sample input files.

v2e-tennis-example

Vote for new v2e features.

v2e development was supported by the Swiss National Competence Center for Robotics (NCCR Robotics).

News

See changelog for latest news

Contact

Yuhuang Hu (yuhuang.hu@ini.uzh.ch) Tobi Delbruck (tobi@ini.uzh.ch)

Citation

If you use v2e, we appreciate a citation to the paper below. See the v2e home page for futher background papers.

Y. Hu, S-C. Liu, and T. Delbruck. v2e: From Video Frames to Realistic DVS Events. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), URL: https://arxiv.org/abs/2006.07722, 2021

To reproduce the experiments of the paper, please find this repository.

Advice about conversion time

We recommend running v2e on a CUDA GPU or it will be very slow, particularly when using SuperSloMo upsampling. Even with a low-end GTX-1050, v2e runs about 50-200X slower than real time using 10X slowdown factor and 346x260 video.

Conversion speed depends linearly on the reciprocal of the desired DVS timestamp resolution. If you demand fine resolution of e.g. 100us, then expect many minutes of computing per second of source video. Running on Google colab with GPU, it took 500s per second of 12FPS source video, because of the very high upsampling ratio of over 800X and the 220k frames that needed to be produced for DVS modeling.

We advise using the --stop option for trial run before starting a long conversion.

Using v2e in Google CoLab

If you don't want to install, try opening v2e in google colab.

Local Installation

There are 3 general steps

Make a conda environment
Install pytoch and other conda distributed packages to this environment
Install the rest of the packages and v2e to the conda enviroment with pip.

pip is needed because some packages are not availble from the conda repositories. It is important to go in this order because conda in general is not aware of pip installs.

Make conda environment

Install v2e to a separate Python environment such as conda environment:

conda create -n v2e python=3.10  # create a new environment
conda activate v2e  # activate the environment

Ensure you have CUDA enabled GPU-accelerated pytorch

See https://pytorch.org/get-started/locally/ to generate the correct conda install command to enable GPU-accelerated CUDA. On windows 10, for example, this tool generated the following command. v2e does not use torchaudio so you can leave it out.

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

You may want to check this stackoverflow question: https://stackoverflow.com/questions/57238344/i-have-a-gpu-and-cuda-installed-in-windows-10-but-pytorchs-torch-cuda-is-availa

Use pip to install rest of packages and v2e

After installing pytorch to your CUDA environment, to install v2e in developer mode (so your edits to source take effect immediately), run the following command in your terminal inside the activated conda environment. The python -m pip install -e . command runs setup.py which installs more packages and adds a script to run v2e from the command line to the conda enviroment python path; see https://stackoverflow.com/questions/39023758/what-does-pip-install-dot-mean:

conda activate v2e # activate your env with pytoch already installed by conda
git clone https://github.com/SensorsINI/v2e
cd v2e
python -m pip install -e . # use pip to install requirements from setup.py in user mode (so your edits to source files in v2e take effect immediately

If you want an additional Windows GUI interface, you will need to install Gooey package. This package works the best on Windows:
```
pip install Gooey
```
On Linux, Gooey can be hard to install.

For a sample of conversion using the gooey GUI, see https://youtu.be/THJqRC_q2kY

Download SuperSloMo model

We use the excellent Super SloMo framework to interpolate the APS frames. However, since APS frames only record light intensity, we retrained it on grayscale images.

Download our pre-trained model checkpoint from Google Drive SuperSloMo39.ckpt (151 MB) and save it to the input folder. The default value of --slomo_model argument is set to this location.

Special thanks to Zhe He for recovering the checkpoint file.

Download sample input data

The sample input videos to try v2e with are in v2e-sample-input-data on google drive.

Download the tennis.mov video and put in the input folder to run the example below.

Usage

v2e serves multiple purposes. Please read to code if you would like to adapt it for your own application. Here, we only introduce the usage for generating DVS events from conventional video and from specific datasets.

Render emulated DVS events from conventional video.

v2e.py reads a standard video (e.g. in .avi, .mp4, .mov, or .wmv), or a folder of images, and generates emulated DVS events at upsampled timestamp resolution.

Don't be intimidated by the huge number of options. Running v2e.py with no arguments sets reasonable values and opens a file browser to let you select an input video. Inspect the logging output for hints.

Hint: Note the options [--dvs128 | --dvs240 | --dvs346 | --dvs640 | --dvs1024]; they set output size and width to popular DVS cameras.

On headless platforms, with no graphics output, use --no_preview option to suppress the OpenCV windows.

usage: v2e.py [-h] [-o OUTPUT_FOLDER] [--avi_frame_rate AVI_FRAME_RATE]
              [--output_in_place [OUTPUT_IN_PLACE]] [--overwrite]
              [--unique_output_folder [UNIQUE_OUTPUT_FOLDER]]
              [--skip_video_output]
              [--auto_timestamp_resolution [AUTO_TIMESTAMP_RESOLUTION]]
              [--timestamp_resolution TIMESTAMP_RESOLUTION]
              [--dvs_params DVS_PARAMS] [--pos_thres POS_THRES]
              [--neg_thres NEG_THRES] [--sigma_thres SIGMA_THRES]
              [--cutoff_hz CUTOFF_HZ] [--leak_rate_hz LEAK_RATE_HZ]
              [--shot_noise_rate_hz SHOT_NOISE_RATE_HZ]
              [--photoreceptor_noise]
              [--leak_jitter_fraction LEAK_JITTER_FRACTION]
              [--noise_rate_cov_decades NOISE_RATE_COV_DECADES]
              [--refractory_period REFRACTORY_PERIOD]
              [--dvs_emulator_seed DVS_EMULATOR_SEED]
              [--show_dvs_model_state SHOW_DVS_MODEL_STATE [SHOW_DVS_MODEL_STATE ...]]
              [--save_dvs_model_state]
              [--record_single_pixel_states RECORD_SINGLE_PIXEL_STATES]
              [--output_height OUTPUT_HEIGHT] [--output_width OUTPUT_WIDTH]
              [--dvs128 | --dvs240 | --dvs346 | --dvs640 | --dvs1024]
              [--disable_slomo] [--slomo_model SLOMO_MODEL]
              [--batch_size BATCH_SIZE] [--vid_orig VID_ORIG]
              [--vid_slomo VID_SLOMO] [--slomo_stats_plot] [-i INPUT]
              [--input_frame_rate INPUT_FRAME_RATE]
              [--input_slowmotion_factor INPUT_SLOWMOTION_FACTOR]
              [--start_time START_TIME] [--stop_time STOP_TIME] [--crop CROP]
              [--hdr] [--synthetic_input SYNTHETIC_INPUT]
              [--dvs_exposure DVS_EXPOSURE [DVS_EXPOSURE ...]]
              [--dvs_vid DVS_VID] [--dvs_vid_full_scale DVS_VID_FULL_SCALE]
              [--no_preview] [--ddd_output] [--dvs_h5 DVS_H5]
              [--dvs_aedat2 DVS_AEDAT2] [--dvs_text DVS_TEXT]
              [--label_signal_noise] [--cs_lambda_pixels CS_LAMBDA_PIXELS]
              [--cs_tau_p_ms CS_TAU_P_MS] [--scidvs]
v2e: generate simulated DVS events from video.
optional arguments:
  -h, --help            show this help message and exit
Output: General:
  -o OUTPUT_FOLDER, --output_folder OUTPUT_FOLDER
                        folder to store outputs.
  --avi_frame_rate AVI_FRAME_RATE
                        frame rate of output AVI video files; only affects
                        playback rate.
  --output_in_place [OUTPUT_IN_PLACE]
                        store output files in same folder as source video (in
                        same folder as frames if using folder of frames).
  --overwrite           overwrites files in existing folder (checks existence
                        of

V2e

Install / Use

README

v2e