Vid2Player: Controllable Video Sprites that Behave and Appear like Professional Tennis Players

Abstract

We present a system that converts annotated broadcast video of tennis matches into interactively controllable video sprites that behave and appear like professional tennis players. Our approach is based on controllable video textures, and utilizes domain knowledge of the cyclic structure of tennis rallies to place clip transitions and accept control inputs at key decision-making moments of point play. Most importantly, we use points from the video collection to model a player’s court positioning and shot selection decisions during points. We use these behavioral models to select video clips that reflect actions the real-life player is likely to take in a given match play situation, yielding sprites that behave realistically at the macro level of full points, not just individual tennis motions. Our system can generate novel points between professional tennis players that resemble Wimbledon broadcasts, enabling new experiences such as the creation of matchups between players that have not competed in real life, or interactive control of players in the Wimbledon final. According to expert tennis players, the rallies generated using our approach are significantly more realistic in terms of player behavior than video sprite methods that only consider the quality of motion transitions during video synthesis.

Siggraph Presentation Video (5 min)

Siggraph Presentation Video (20 min)

Materials

Citation

@article{zhang2021vid2player,
        title={Vid2player: Controllable video sprites that behave and appear like professional tennis players},
        author={Zhang, Haotian and Sciutto, Cristobal and Agrawala, Maneesh and Fatahalian, Kayvon},
        journal={ACM Transactions on Graphics (TOG)},
        volume={40},
        number={3},
        pages={1--16},
        year={2021},
        publisher={ACM New York, NY}
      }
      

Related Work

In our latest project Vid2Player3D, we present a system that learns physically simulated tennis skills from the broadcast videos.

Contact

For any question regarding this research paper, please contact Haotian Zhang via haotianz@cs.stanford.edu.

Acknowledgement

The project is supported by financial and computing gifts from the Brown Institute for Media Innovation and Nvidia. The video dataset was downloaded from Youtube Wimbledon channel and US Open channel.