Videogan
Generating Videos with Scene Dynamics. NIPS 2016.
Install / Use
/learn @cvondrick/VideoganREADME
Generating Videos with Scene Dynamics
This repository contains an implementation of Generating Videos with Scene Dynamics by Carl Vondrick, Hamed Pirsiavash, Antonio Torralba, to appear at NIPS 2016. The model learns to generate tiny videos using adversarial networks.
Example Generations
Below are some selected videos that are generated by our model. These videos are not real; they are hallucinated by a generative video model. While they are not photo-realistic, the motions are fairly reasonable for the scene category they are trained on.
<table><tr><td> <strong>Beach</strong><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/1.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/2.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/3.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/4.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/5.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/6.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/7.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/8.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/9.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/10.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/11.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/12.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/13.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/14.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/beach/15.gif'> </td><td> <strong>Golf</strong><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/1.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/2.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/3.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/4.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/5.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/6.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/7.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/8.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/9.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/10.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/11.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/12.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/13.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/14.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/golf/15.gif'> </td></tr><tr><td> <strong>Train Station</strong><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/1.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/2.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/3.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/4.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/5.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/6.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/7.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/8.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/9.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/10.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/11.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/12.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/13.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/14.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/train_station/15.gif'> </td><td> <strong>Baby</strong><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/1.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/2.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/3.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/4.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/5.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/6.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/7.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/8.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/9.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/10.gif'><br> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/11.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/12.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/13.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/14.gif'> <img src='http://carlvondrick.com/tinyvideo/supp/supp/hospital/15.gif'> </td></tr></table>Training
The code requires a Torch7 installation.
To train a generator for video, see main.lua. This file will construct the networks, start many threads to load data, and train the networks.
For the conditional version, see main_conditional.lua. This is similar to main.lua, except the input to the model is a static image.
To generate videos, see generate.lua. This file will also output intermediate layers, such as the mask and background image, which you can inspect manually.
Data
The data loading is designed assuming videos have been stabilized and flattened into JPEG images. We do this for efficiency. Stabilization is computationally slow and must be done offline, and reading one file per video is more efficient on NFS.
For our stabilization code, see the 'extra' directory. Essentially, this will convert each video into an image of vertically concatenated frames. After doing this, you create a text file listing all the frames, which you pass into the data loader.
Models
You can download our pre-trained models here (1 GB ZIP file).
Notes
The code is based on DCGAN and our starter code in Torch7.
If you find this useful for your research, please consider citing our NIPS paper.
License
MIT
Related Skills
qqbot-channel
345.9kQQ 频道管理技能。查询频道列表、子频道、成员、发帖、公告、日程等操作。使用 qqbot_channel_api 工具代理 QQ 开放平台 HTTP 接口，自动处理 Token 鉴权。当用户需要查看频道、管理子频道、查询成员、发布帖子/公告/日程时使用。
docs-writer
100.0k`docs-writer` skill instructions As an expert technical writer and editor for the Gemini CLI project, you produce accurate, clear, and consistent documentation. When asked to write, edit, or revie
model-usage
345.9kUse CodexBar CLI local cost usage to summarize per-model usage for Codex or Claude, including the current (most recent) model or a full model breakdown. Trigger when asked for model-level usage/cost data from codexbar, or when you need a scriptable per-model summary from codexbar cost JSON.
arscontexta
2.9kClaude Code plugin that generates individualized knowledge systems from conversation. You describe how you think and work, have a conversation and get a complete second brain as markdown files you own.
