During Georgia Tech's Fall 2016 semester I participated in an independent study with the program director, Dr. Michael Nitsche, that focused on the exploration of puppetry techniques into virtual reality. Michael came into the project with an interest in puppetry as a medium and he wanted help exploring methods for simulating puppetry in VR. Specifically, he liked how the Vive used motion controls and room-scale simulations. He felt that the Vive could simulate actual puppets and expand the materials or form used in puppetry to ones only available digitally.
For the project we collaborated with the Center for Puppetry Arts, a local Atlanta non-profit dedicated to the art of puppet theater. We used their expertise in puppets to learn where the design space is for our project and what sorts of puppets would most quickly translate to VR. After that, I met Michael weekly as I slowly built up a system using the Vive and Unity to control a marionette puppet.
They weren't created without difficulty though. I started by simulating the strings, which proved too computationally expensive to proceed with. Knowing that, I iterated on the system and found a simpler method for drawing the string. I ended up finding a tutorial that talked about how to draw a spline in Unity and drop an indeterminate number of objects along that spline. That let me drop and connect joints at runtime while also faking the visible string.
As for the the puppet itself, it's controlled by the user with the two Vive hand controllers, one held sideways. We decided to have one held sideways to simulate the act of holding a marionette puppet, many of which have crossed wooden bars for handles. I used a 3rd party plugin called VRTK to read input from the controller itself and connect to my code and the puppet.
This implementation proved to be successful enough to prove that the initial inquiry could be fruitful. At the time of writing Michael was able to secure grant funding to further develop the project. During this Spring semester I'll be working as a research assistant on the project, and hopefully I'll have a more complete demo to showcase too.