Research & Creative Work

Collaboration with “Plunderphonics” composer John Oswald (2020-present) to upgrade the tech for his 2001 composition, Concerto for Conductor (now renamed a sum of distractions). Mentored four students in User-Centered Musical Design course (IMM470/MUS370) in Fall 2020 to construct a wireless trigger system, integrate into fitted gloves, connect to custom Max8 code, and control a sample bank in Ableton Live. Project video documentation by TCNJ students Braden Kirk, Danny Loos, Meaghanne McBride, and Benjamin Spizuco.

Collaboration with the Robert Wood Johnson University Hospital (2019-present) in Hamilton, NJ, on a research project entitled “Online Music Therapy Collaborative Study for Cancer Patients.” Approved by RWJUH and TCNJ Institutional Rewview Boards. Worked directly with RWJUH Foundation board and staff in the Cancer Center. TCNJ students in User-Centered Musical Design course created music therapy interventions, designing a musical app for cancer patients to explore during chemotherapy infusions. Students worked with hospital staff to distribute surveys and communicate with patients in the user testing and evaluation of the app. Student internship in Fall 2021. Run the app on a web browser. See TCNJ's School of the Arts & Communication's news story about this project.

Trenton Makes Music (2014-present) Project Co-Director with TCNJ Professor Kim Pearson. Goal: to coalesce the community of Trenton, NJ around its rich cultural legacy and encourage reinvestment, local pride, and socioeconomic development. We worked with TCNJ students to develop online resources and public-facing media including a website, podcasts, audio recordings, theme songs, oral histories, interviews, interactive maps, timelines, social media posts, and ongoing conversations with regional constituencies. We produced seven public events, featuring artists including Sarah Dash and Michael Ray. See the Trenton Makes Music website and archive for artist biographies, interactive timelines, podcasts, and other project content. Listen to Professor Pearson's Trenton Music History playlist on Youtube. NJTV News segment:


The Flame in the Flood interactive animation, presented with the Boston Pops Orchestra in a sold-out concert, May 2017. Real-time audio signals drove the visual display, controlled by a Max patch, a MIDI controller (APC40), and a game controller. Collaboration with indy game designer and AI expert Damian Isla. Nominated by Harvard class reunion committee, approved by Boston Pops artistic staff. Live concert video:


Jersey Fresh (2017) composition for string orchestra, harp, and electronics. Commissioned by the TCNJ Orchestra (Harold Levin, conductor), premiered in Mayo Concert Hall with student soloists. Dress rehearsal excerpt. Premiere performance:


Conduct the Orchestra interactive museum exhibits (2010-2018) for ABC Extreme Makeover Home Edition (episode 806), the Clay Center for the Arts & Sciences in Charleston WV, Children’s Discovery Museum in Normal IL, Curious Kids Museum in St. Joseph MI, Kohl Children’s Museum in Chicago, and the Ravinia Music Festival. Professional contracts totaling $42,890.60 produced by Immersion Music. Exhibit at the Kohl Children's Museum in 2013:

Software demo of the "Conduct It!" exhibit for the Clay Center for the Arts and Sciences (2017):


Collaboration with Marvin Minsky (2010-2015), mentor and AI pioneer (1927-2016). Initially, documented Minsky’s engagement with music through the lens of his “Music, Mind, and Meaning” paper (Computer Music Journal 5.3, 1981). The project later evolved into a set of recorded conversations and an NSF-funded workshop in 2012 (Music, Mind, and Invention). The purpose of the workshop was to explore the rich interconnections between music, cognition, computation, and creativity. It featured talks, panel discussions, demonstrations, and concert performances by respected scholars including Harvard mathematician Noam Elkies, MIT composer Tod Machover, NYU psychologist and cognitive scientist Gary Marcus, Princeton music theorist Dmitri Tymoczko, and Eran Egozy, co-founder of Harmonix Music Systems, creators of the Guitar Hero and Rock Band video game series. Hundreds attended, including Nobel Prizewinner John Nash. Minsky’s Keynote address is available on the TCNJ Center for the Arts Youtube channel, and TCNJ issued a remembrance at Minsky’s passing. Two TCNJ students and I participated in a follow-up workshop ("Reconstructing Beethoven's Improvisations") at the MIT Media Lab in 2014, for which I served on the Advisory Board. For more information: The Musical Mind of Marvin Minsky.


Conducting Robots interdisciplinary research collaboration (2008-2013) with TCNJ colleagues Andrea Salgian, Chris Ault, and Jennifer Wang. Funded by an RUI (Research in Undergraduate Institutions) CreativeIT grant from the National Science Foundation, this project provided a research-based platform for teaching teamwork and creative problem solving to undergraduate students from diverse backgrounds. Students worked in groups to design and build human-scale robots and graphic visualizers that conducted our college orchestra at the end of each semester. Students developed expertise in building real-time systems that could listen to music, estimate pitch and tempo, track beats, generate expressive gestures, and follow scores. The robots interacted directly with human musicians and received feedback that was then applied toward iterative revision of the system designs. We offered an advanced course for four consecutive semesters, followed by a third year during which we organized an international workshop and published our research findings. The course was cross-listed Computer Science, Mechanical Engineering, Interactive Multimedia, and Music. We used pre- and post-assessments to study whether students increased their creative problem-solving capacities at the end of the semester. For more information: Giving the Maestro a Human Heart: Fostering Creativity in a Multi-Disciplinary Undergraduate Environment. Fall 2009 prototypes. TCNJ Magazine story on the course.

Prototype orchestra conducting sessions, 2009-2011:


Platonic Harmonic (2013) composition for violin and electronics in Max/MSP. Plato wrote some of the earliest treatises on music, proposing a way of thinking about music based upon mathematical proportions and geometric ratios. I would argue that real music lies somewhere between the pure rationality of patterns and the complexities of physical and emotional experiences. This piece was motivated by a question that a student asked me once about whether overtones could be played in the same way that notes are. At the time, I didn't have a satisfying answer, but the question captured my imagination, and I have attempted to answer it with this piece. May 2014 invited performance in the Automatic Accompaniment Demonstration Concert at the Acoustical Society of America annual conference:

April 2013 performance at the TCNJ “Mostly Millennial” concert:


Virtual Maestro interactive conducting kiosk (2008-2011), developed for UBS. Presented in performing arts venues including Avery Fisher Hall, Boston Symphony Hall, Kimmel Center (Philadelphia), Walt Disney Hall (LA), Severance Hall (Cleveland), Montreux Jazz Festival, Ravinia Festival. Invited exhibit, 2011 NJ Festival of Electronic Arts. Featured a Nintendo Wiimote as a conducting baton. Reviewed in Wired Magazine, The Guardian (UK), and many other US press outlets. 2008 user testing footage at TCNJ. Conference paper published in NIME 2009 proceedings. "Conductor Hero" video produced by TCNJ in 2008.


Research projects in Conducting Gesture Analysis using Computer Vision (2007-2010). During Summer 2010 I worked with Timothy Song, a double-major in Biology and Computer Science, on a Mentored Undergraduate Summer Experience project. Our goal was to analyze physiological data gathered from musicians during a 2006 Boston Symphony Orchestra project. (See below for more information about the collaboration with the BSO.) Tim produced a video of a conductor's heart rate that compellingly demonstrated the roller-coaster effects that shape musical experience. (Tim went on to Carnegie Mellon University's graduate program in Computational Biology.) Here is that video:


In addition, 2007-2010 I collaborated with Dr. Andrea Salgian and her mentored research students in the TCNJ Computer Science Department. We developed new methods to extract meaningful information from video recordings of professional conductors. We produced three conference papers and a few more video analyses, listed below.

Conducting analysis of Philip Tate with the TCNJ Orchestra (2010):

Nakra, T.M., Tilden, D. and A. Salgian (2010),"Improving Upon Musical Analyses of Conducting Gestures Using Computer Vision." Proceedings of the International Computer Music Conference, SUNY Stony Brook, 2010. ISBN: 0-9713192-8-6.)


Musical Analysis of Conducting Gestures Using Methods from Computer Vision (2009):

Nakra, T.M., A. Salgian, and M. Pfirrmann (2009), "Musical Analysis of Conducting Gestures Using Methods from Computer Vision." International Computer Music Conference, McGillUniversity, Montreal.

Salgian, A., Pfirrmann, M. & T.M. Nakra (2007). Follow the Beat? Understanding Conducting Gestures from Video. 3rd International Symposium on Visual Computing, Lecture Notes in Computer Science Series, Springer.


Guru Shishya Parampara (2010) composition for violin and electronics in Max/MSP. The title of this piece refers to the ancient system of teacher-student transmission in the North Indian classical music tradition. When I was 22 years old, I travelled to India to experience this concept for myself. For ten months I took daily music lessons with Pandit Sreeram G. Devasthali, whom I called "Guruji." He taught not only the tradition and mechanics of the Raga system but also a graceful approach to life and to music. This piece features an interactive patch in the Max/MSP software environment that I created to re-animate a recording of one of our lessons in Raag Malkauns; there are also occasional audio snippets from my Guru's Guru, Gajananrao Joshi, on violin. Sept. 2011 invited performance at “Musical Creativity East and West,” a concert blending Indian Classical traditions with American Jazz, hosted by the MIT Club of Boston. March 2010 premiere solo performance at the TCNJ Faculty Composers Concert:


You’re the Conductor exhibit (2003-2009) at the Boston Children’s Museum. Visitors conducted the Boston Pops Orchestra in their signature piece, "The Stars and Stripes Forever," using an infrared sensor baton. Centerpiece of the $2 million Making America’s Music project. Custom gesture-tracking electronics, software, and playback control. Collaborated with a research team at Stanford University, secured corporate sponsors, presented on Discovery Channel & WIRED NextFest. 1 million visitors for 2004-2007 North American tour of children's museums. 2005 video of Boston exhibit. 2005 video from WIRED NextFest. Early project website.


Yezie Five Loops (2008) audio composition. This piece features my son at age 5, exaggerating his ability to quickly switch between high-energy and relaxed reflective moments. I also experimented with two well-known electronic techniques, both pioneered by Steve Reich: phasing of loops, and emphasizing the melody in recorded speech. I picked short segments that were appropriate for looping, and then created layered textures where the loop points were purposefully out-of-phase. There are five primary loops, because "I'm five!" There is a string quartet that plays along with the looped speech, sometimes taking over from the spoken word to emphasize the melodic content or further developing the melodies. Premiered at TCNJ Faculty Composers Concert in February 2008:


collaboration with the Boston Symphony Orchestra and McGill University (2005-2006). Worked with Dan Levitin, Stephen McAdams, and Myran Parker-Brass on experiment to quantify the emotional effects of live music and how emotion is conveyed from conductor to musicians to audience. Physiology sensors worn by conductor, five musicians, and 100 audience members. Second experiment with HD video recording undertaken at McGill. Covered in NY Times, USA Today, AP, Boston Globe, Mix, Stereophile, La Presse, Toronto Star. CBS4 TV News segment. Featured on the Discovery Channel:


Digital Conducting Laboratory at Arizona State University (2000-2005). Designed student etudes, wrote software to process sensor data, interpret conducting patterns, tempo, dynamics, and output MIDI sound. Developed sensor sleeve. 150 students used the lab for coursework, including two Masters’ theses and two journal articles (including Hollinger, Diana, and Jill M. Sullivan. "The Effects of Technology-Based Conducting Practice on Skill Achievement in Novice Conductors." Research and Issues in Music Education 5.1 (2007): 1-6.). Featured on BBC World Service and Discovery Channel. 2001 ABCNews article.


Interactive animation for Arvo Pärt's Fratres (2002). Collaboration with violinist Joanna Kurkowicz and Jan Kubasiewicz for presentation at Massachusetts College of Art Dynamic Media Institute conference. I contributed the sensor/interaction design component. Rehearsal video footage.


Interactive Lighting Design for Bach g minor Violin Sonata (2002). Created for a Clifton Visiting Artist Residency and presented in performance at the Harvard University Adams House Pool Theater, February 2002 -- I contributed the sensor/interaction design, Sarah Sidman did the lighting design, and violinist MIki-Sophia Cloud performed Bach Violin Sonata No. 1 in g minor, BWV 1001.


Conductor’s Jacket sensor interface (1997-2000), presented with Boston Pops Orchestra 1998, MIT Sens*bles Symposium 1999, CNN Headline News 1999, featured in the 2011 MIT150 Exhibition (nominated and voted by MIT alumni), and held since 2011 in the permanent collection of the MIT Museum. MIT Media Lab project page. Local TV news coverage of concert. 1999 MIT Sens*bles Symposium presentation in Kresge Auditorium.


Digital Baton sensor interface (1996-2000), designed and built with a team at MIT; US Patent issued 1999, referenced by 170 other US patents. Featured in Tod Machover’s Brain Opera; presented 150 performances on four continents over two years, 1996-1998. Featured on PBS Scientific American Frontiers with Alan Alda, at the Ars Electronica Festival, and many other international and national media outlets. Video of Lincoln Center premiere. Compilation research video.


SP/RING (2001) composition for interactive Conductor’s Jacket interface with digital video control. Collaboration with video artist Walter Wright. Premiered at Boston Cyberarts Festival 2001, repeated at Harvard Clifton Artists concert 2002, featured in a segment on Discovery Channel:

SP/RING audio recording:


musicBottles (2000)
Violinist for the musicBottles, a Human-Computer Interface project developed in Professor Hiroshi Ishii's Tangible Media Group at the MIT Media Lab:

(Lalo's Piano Trio in c minor, op. 7 begins at 1'50", with cellist Bernd Schoner and pianist Jason Wong)


Illusion of Safety (2000) audio collaboration with Dan Burke:


Pine (2000) audio collaboration with Maria Moran:


Affective Carpet (1999) for cloth pressure-sensitive controller (performed here by Noah Feehan and by another student here in a composition by Teresa Nakra at Harvard Clifton concert, 2002).


SWAMPED! (1998) audio composition for interactive installation (MIT Media Lab collaboration with Bruce Blumberg et al, presented at ACM SIGGRAPH 1998: "Swamped! Using Plush Toys to Direct Autonomous Animated Characters").


Neruda Canon (1997) audio composition for E-Mu Morpheus synthesizer:


Fish Food (1995) for interactive animated short film by Rolf Rando and David Allport:


Dogmatic Overture (1994) audio composition for interactive storytelling environment by Tinsley Galyean (see video demo here):


Raga Bhairav (1994) MIDI composition: