Between Barriers and Prospects: Merging Art Performance and Engineering in Mobile Music Education and Research
White Paper

Between Barriers and Prospects: Merging Art Performance and Engineering in Mobile Music Education and Research

Georg Essl, University of Michigan

1. Introduction

There is little doubt that mobile smart devices are a socio-cultural game changer. The reach of sophisticated, networked, interactive computational technology will soon be universal. This means that technology with tremendous capabilities for artistic expression open up a space of exploration for new forms of culture and creativity.

The mobile phone has become the most widely distributed, accessible communication and computing device. In 2010, the mobile subscription base has reached an estimated 5 billion and the penetration of these devices in North America, Europe and Japan is considered to have reached 100% but sales are still growing due to the drastic improvements in mobile smart phone technology (Wingfield 2009). The mobile phone is no longer just a portable version of the classical telephone, but has become a device serving a large set of diverse needs. It is a digital camera, a media consumption device ranging from e-book reader, to music and video player, to portable TV and gaming platform. It serves as personal data organizer, Internet client, navigator through GPS integration, and increasingly as a general-purpose computing device. Given the high density of sensor technologies, such as integrating accelerometers, gyroscopes, magnetic field sensors, microphones and cameras, mobile devices offer a new sensor environment that additionally is highly mobile, making the platform significantly different to desktop computing. The exploding ubiquity of these devices will make new forms of artistic  and collaborative activities possible that are much harder to envision with desktop systems that need to be used in static settings.

Mark Weiser's prediction of a "ubiquitous computing" world (Weiser 1991) is becoming true through mobile technology. The size of the device means that it is often carried like a wallet. This in turn means that the computational capacity of the device moves with its owner and computational capacity scales with number number of members in the group who brought their personal device. The mobile phone is the new personal computer, re-spawned in a new environment where connectivity is a constant, participation is growing at a rapid pace, and the support of content creation becomes ever more important.
Over the last five years we have seen an explosion of user generated content (Dawson 2008) on sites like Flickr, YouTube, Facebook and blogs. Especially teenagers are now confident and frequent users of the Internet. In 2007 about 93% of teenagers in the US used the Internet, but more importantly already in 59% participated in some form of digital content creation and sharing (Lenhart et al 2007). Internet access through mobile devices is growing drastically and data traffic has already surpassed voice traffic in 2010 and is expected to dominate traffic volumes in the future (Thomas 2007).

2. Prospects and Barriers of Mobile Technology for Creativity and Participation

This means that an unprecedented opportunity for participation in creative endeavors mediated through commodity computing devices is emerging and a tremendous impact on communities who have as yet find participating in the digital culture. Due to the mobile nature of the device, the ability to participate will drastically increase, as well as change where participation can and will happen. Hence we should expect to not only see drastic changes in the demographics of participation, but also the general character of the participation itself.

Currently we face a barrier of access for most users of mobile technology due to the complexity and sophistication of the devices and the depth of domain knowledge required to build creative applications on them. Typical users of smart devices are not trained engineers, nor necessarily trained performance artists.

Mobile devices also have already shown to be vehicles for diminishing, perhaps overcoming, the various domains of separation of access and literacy in information and computational technologies (ICTs) called the Digital Divide. Difference in access and participation can be seen along many dimensions, including geography, age, gender, socio-economic status and ethnicity (Hargittai 2003). Mobile devices are significantly cheaper and by serving multiple needs such as communication and computation at the same time, often more viable. Mobile devices tend to see diminished use segregation by gender and by socio-economic group that we observe with classical computational platform, and underrepresented minority show a disproportionally large segment in mobile internet use (Brown et al 2011).

Figure 1 The progression of computational literacy for mobile programming today (top curve) versus the kind of literacy progression that is desirable for broad participation (bottom)
So far industry focuses on media consumption, rather than broader participation on digital content creation and computational literacy. For example Apple has removed academic computational literacy projects such as Scratch from their mobile AppStore to ensure a policy that all programming happens through Apple-controlled pathways (Chen 2010). Current mobile platforms are not optimized for easy programming. Level of entry for participation on the process is high and requires low level programming knowledge, heterogeneous hardware setups such as a laptop, USB cable connectivity and large screen IDEs such as Xcode for iPhones or Eclipse for Android. This requires high levels of training and knowledge. It also imposes an additional financial burden, by requiring additionally substantially more expensive hardware (such as a laptop) for the programming activity.
This leads to a kind of Computational Literacy Barrier, as illustrated in Figure 1, separating those who had substantial preparation and resources to acquire it away from mobile technology, from those who have access to mobile technology but limited alternative means, opportunities or interest to acquire literacy elsewhere.

However, it has been widely argued that computational literacy is important in a world immersed in ICTs (Nelson 1974, diSessa 2001) and we see it as critical to create pathways mobile technology users to learn, create, and participate.

3. Research Challenges

Mobile smart devices are also shifting various technological paradigms. Creative content creation on laptop and desktop computers assumed a given interaction model centered around keyboard, large monitor, and mouse. A multi-touch centric device with a small display and additional rich input sensors such as cameras and motion sensors replaces this.  A further significantly changed factor is the size of the device which can be substantially smaller than other general purpose computing devices.

Hence existing models for supporting computational creativity have to be rethought and we need sustained research to develop fitting models of Human-Computer Interaction that solves key problems in creativity support and content creation, and allows accessibility for as wide a target audience as possible.

Central questions that are persistent research topics include:
1.    The evolution of hardware for expressiveness. Commodity devices in creative expression form a delineating canvas of the possible. This means that a concern for expressivity will mean a continued evolution of hardware. For example current mobile multi-touch technology tends to not support pressure sensing or tactile feedback, yet these are important factors in fine motor control.
2.    The development of programming and content creation paradigm that fit the input and output modalities of the form factor of commodity mobile devices. This may well require a re-envisioning of the very basis of programming as no longer a necessarily textual paradigm, but one that is constructivist or symbolic.
3.    Design for universal accessibility in order to reach diverse target audiences.
4.    Design for on-line interactive and performative use.

To this end we develop an environment called urMus, which seeks to provide a mobile-centric design of open, and accessible creativity support. However we do see our individual effort as but one proposal to offer technological solutions to key questions of participating and we suggest a broad range of engagement with the field.

4. Mobile Technology in Interdisciplinary SEAD Teaching

We designed a senior level undergraduate course titled "Mobile Phones as Musical Instruments." It is cross-listed between the College of Engineering and the School of Music, Theater and Dance at the University of Michigan. The placement of such interdisciplinary course has numerous challenges but also clear benefits.  The course is designed to blend students from diverse preparatory backgrounds. All students engage in the full range of activities in the course without distinguishing if they pursue education in engineering or the arts. We discuss describe advantages and pitfalls of this course design. UrMus is the central programming platform in our course. Its design allows rapid access, early rewards, and a sense of mastery. As students become more proficient the design allows deep engagement and open expression. An exit surveys show that students largely see the approach as successful, independent of prior background.
Mobile smart devices have already had a drastic impact on how we use computation and expanded who is able to participate. The prospect of enabling broad participation on technological creativity is tremendous with potential impact on who can enter STEM and creative fields.

4.1 Evaluation of Cross-Disciplinary Teaching of Art and Mobile Technology

In Fall 2010 the course "Designing Mobile Phone Musical Instruments for Ensemble Performance" co-listed in Electrical Engineering & Computer Science in the College of Engineering as well as Performance Art Technology in the School of Music, Theater & Dance was funded by the College of Engineering Curriculum Innovation program.

The course is inherently multi-disciplinary engaging students in both music and engineering practices. The primary goal of the course is open-ended problem solving and creative engagement with a clear final outcome. The course asks students to design their own mobile phone musical instruments, conceptualize and write music for it and ultimately perform the results in a live concert, which is open to the public. Here we report the outcome of such a SEAD course.

The course has two main of challenges:

1) Integration of a very diverse set of student preparations and experience in an upper level course bridging two colleges.
2) Innovating technological teaching with emerging new platforms such as commodity mobile smart devices.

There are also a number of further challenges such as the reality of proprietary mobile phone programming. Hence it was important to find ways to allow the students to go beyond a limited set of devices or specific form factors.

For the first time students were allowed to loan devices for the duration of the course hence were able to work on assignments on the device on their preferred schedule. The learning curve of programming was adjusted to be platform independent and use a high-level programming language. This addressed two concerns of the first iteration of the course. One was a learning curve that was too challenging for many Performance Arts Technology majors, as it required rapid learning of Objective-C from the start. Second was the ability to postpone hardware specific details to the end of the course and focus the teaching efforts on principles that are not specific to one platform or form factor.

4.1.1 Assessment

In order to assess the innovations introduced to the course we conducted an informal survey asking questions specific to the curricular changes and their benefits as well as broader questions about the reception of the course using a 5-point Likert scale. The questions were fully anonymous and participation was voluntary. Six students chose to answer the survey. One of the six students omitted the questions of the second page (Q9-11). The survey questions are attached at the end of the report.

The questions were designed to address the following questions: Did introducing access to hardware help the course in the student's mind (Question 2, Question 11)? Did the learning curve adjustment work (Questions 3-5)? Does the cross-disciplinary integration work in the eyes of the students and stimulate curiosity (Question 1, Question 7-10)?

Figure 2 Mean and standard deviation of answers to course survey questions.

The mean answers to the questions can be seen in Figure 1. The most direct measure of the devices on loan being helpful is Q2 which was answered very positively by all students (avg=4.97, stdv=0.08) however students were less clear if this was essential (Q11, avg=3.56, stdv=0.8). Students felt that the cross-disciplinary integration was positive (combined score of Q1, Q7-Q10, avg=4.08, stdv=0.48) and that they learned many aspects of the subject matter (Q6, avg=4.2, stdv=0.69). Question about the learning curve (Q3-5) have a combined score of avg=3.01 with a relatively large deviation (stdv=1.04). This is likely an indication of the bi-modal distribution of prior experience of students in the class. The fact that the mean is at neutral is a good sign, indicating that a balanced between the two populations was nevertheless possible.

All these results have to be taken with care because of the small sample size (N=6) and possible other uncontrolled biases.

4.1.2 Other outcomes

The course has a public concert of student pieces as final. Thanks to the new diversity of devices the repertoire and style of pieces was greatly expanded and specifically the engagement with different form factors (iPad vs iPod Touch) clearly is visible. Overall the student projects had more depth and detail thanks to the improved learning curve and the added time to deal with more detailed issues such as graphics rendering, projection and networking.

The full pieces can be found on the YouTube channel of the department of Electrical Engineering and Computer Science of the University of Michigan.

Figure 3 Impressions of the final concert outcomes of the cross-disciplinary course "Designing Mobile Phone Musical Instruments for Ensemble Performance" at the University of Michigan

4.1.3 Sustainability

The original course of Fall 09 has seen drastic changes in its design and use of technology. The primary change is in the software environment that is used. We have developed a platform called urMus, which is meant to support mobile development with an abstraction of the hardware layer. This means that changes in future hardware and differences in hardware available can be mitigated. The control over learning difficulty in the environment is a major factor of allowing a proper retuning of the learning curve of the course. Overall urMus allows a range of mobile programming and technology related courses to be taught in a platform independent matter and should translate into other settings as well.

A broader lesson learned is that on-device learning with new technology is possible, while retaining scalability when proper support software is developed to achieve hardware abstraction. Students benefit from being able to interact with emerging technologies for their assignments by loaning the devices for assignments and projects and content can be presented in a more compact yet still accessible fashion. The main obstacle of using the devices the student own is platform heterogeneity and the lack of flexible cross-platform support.

5. Socio-Cultural Impact and Challenges

Mobile technology already shows tremendous socio-cultural impact. The permanent availability of computation, networking and connectivity restructures access and participation. However there are also emerging roadblocks in the way important stakeholders in industry to shape the marketplace and commodification of mobile content. In particular mobile platforms can be rather closed and are designed around a consumer-centric model of content delivery. Even computational content in form of apps are delivered like other consumer media, such as music, video, or books through online marketplaces such as the Apple AppStore. However industry is still establishing the standards and different companies take different approaches. Also access and delivery models such as web-based interactive content may change access barriers that current delivery models to have.

6. Suggested Actions

Obstacle 1: Heterogeneity and closedness of commodity platforms that are suitable for open creative expression in the marketplace.

Suggested Action 1: Advocacy with mobile platform industry to offer openness and free content creation on their devices along with efforts to standardize or support cross-platform content exchange.

Obstacle 2: Lacking unified forum for open exchange and archival access of SEAD art and products.

Suggested Action 2: Efforts for creating open access archival platforms for SEAD mobile art products that may or may not be commodified. In particular library function should be extended to allow for the archiving and delivery of interactive and performative content, which could be in the form of apps or dynamic online content.

Obstacle 3: Academic participation in shaping the mobile platform space to allow open innovation for SEAD research and artistic engagement.

Suggested Action 3: Develop funding initiatives with NSF that target the mobile platform and foster research that create acceleration of SEAD in broad public use.

7. References

Brown, Katie; Campbell, Scott W.; Ling, Rich. (2011). "Mobile Phones Bridging the Digital Divide for Teens in the US?." Future Internet 3, no. 2: 144-158.Butts, L.,Cockburn, A. (2002). An evaluation of mobile phone text input methods. Aust. Comput. Sci. Commun., 24(4):55”"59.
Cennamo, K.. & Vernon, M. (2008). Fostering Creativity in the Classroom:  A Case Study of a Multidisciplinary Design Project. Invited paper presented at "Success factors in fostering creativity in IT research and education", Arizona State University, January 18-20, 2008, workshop sponsored by the National Science Foundation.
Chen, B. X. (2010). Apple Rejects Kid-Friendly Programming App, Wired, April 20, http://www.wired.com/gadgetlab/2010/04/apple-scratch-app/

Dawson, R. (2008) Future of Media Report , July, http://futureexploration.net/images/Future_of_Media_Report2008.pdf

de Paula, R., Fischer, G. and Ostwald, J. (2001). Courses as Seeds: Expectations and Realities. In P. Dillenbourg, A. Eurelings and K. Hakkarainen (Ed.), Proceedings of Euro-CSCL' 2001, Maastricht, Netherlands, pp. 494-501.

diSessa, A. (2001). Changing Minds: Computers, Learning, and Literacy. Cambridge: The MIT Press.

Essl, G. (2003b). "On Gender in New Music Interface Technology," Organised Sound, 8:1, 19-30.
Essl, G. (2010). UrSound - Live Patching of Audio and Multimedia using a Multi-Rate Normed Single-Stream Data-flow Engine. In Proceedings of the International Computer Music Conference (ICMC), Stony Brooks/New York, June 1-5.
Essl, G., Rohs, M. (2009). Interactivity for Mobile Music Making. Organised Sound 14(2) 197-207.
Essl, G., Wang, G., Rohs, M. (2008). Developments and Challenges turning Mobile Phones into Generic Music Performance Platforms. In Proceedings of the Mobile Music Workshop, Vienna.
Essl, G. (2010a). Mobile phones as programming platforms. In Proceedings of the Pervasive Workshop on Programming Methods for Mobile and Pervasive Systems, Helsinki.

Essl, G. (2010b). UrMus ”" An Environment for Mobile Instrument Design and Performance, In Proceedings of the International Computer Music Conference (ICMC), Stony Brooks/New York, June 1-5.

Essl, G. and Müller, A. (2010). Designing Mobile Musical Instruments and Environments with urMus, In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Sydney, Australia, June 15-18.

Hargittai, E. (2003). The Digital Divide and What to Do About It. New Economy Handbook, D. C. Jones, ed., San Diego, CA: Academic Press. 822-841.
Hazas, M., Scott, J. and Krumm, J. (2004). "Location-Aware Computing Comes of Age", IEEE Computer Magazine, vol. 37, no. 2, February, pp. 95-97.
Hewett, T.,(2005). Informing the Design of Computer-Based Environments to Support Creativity, International Journal of Human-Computer Studies 63, 4-5, Special Issue on Computer Support for Creativity, E. Edmonds, L. Candy (Eds.), pp. 383-409.
Johnston, W. M. , Hanna, J. R. P. , Millar, R. J. (2004). Advances in dataflow programming languages. ACM Comput. Surv., 36(1):1”"34.
Lenhart, A., Madden, M., Rankin Macgill, A., Smith, A. (2007). Teens and Social Media, Pew/Internet, http://pewinternet.org.
Maisonneuve, N., Stevens, M., Niessen, M. E., and Steels, L. (2009). Noisetube: Measuring and mapping noise pollution with mobile phones. In ITEE 2009: Proceedings of the 4th International Symposium on Information Technologies in Environmental Engineering.

Maloney, J, Silver, J.  Brennan, K., Monroy-Hernandez, A., and Resnick, M. (2011) Mobile Scratch, Retrieved online at http://llk.media.mit.edu/projects.php?id=1974, on July 16, 2011.
Mistry, P. Maes, P. and Chang, L. (2009). WUW - Wear Ur World - A Wearable Gestural Interface. In CHI'09 extended abstracts on Human factors in computing systems, Boston, USA.

Myers, B. A., and Ko, A. J. (2009). The Past, Present and Future of Programming in HCI. In Human-Computer Interaction Consortium (HCIC'09), Winter Park, CO. February 4th - 8th.
Nelson, T. Computer Lib: You can and must understand computers now / Dream Machines: New freedoms through computer screens—a minority report (1974), Microsoft Press, revised edition 1987.
Thomas, M. (2007). Trends and Evolution in the Wireless Communications Industry - a perspective from Ericsson. Wireless Connections Conference, November. http://www.vitel.ca/wc2007/documents/WC07_Thomas.pdf

Wang, G., Essl. G., Penttinen, H. (2008). Do Mobile Phones Dream of Electric Orchestras? In Proceedings of the International Computer Music Conference (ICMC), Belfast.
Weinberg, G. M. (1971/1985). The Psychology of Computer Programming. John Wiley & Sons, Inc., New York, NY, USA.

Weiser, M. (1991). "The Computer for the Twenty-First Century," Scientific American, pp. 94-104, September.

Wingfield, N. (2009). Time to Leave the Laptop Behind. Wall Street Journal, February 23, http://online.wsj.com/article/SB122477763884262815.html.