Music Teaching Aid for Children
I developed the Piano Hero Junior concept, based on my own experience as a child, when I was introduced to the piano using numbers. I combined this with the popular rhythm gaming concept in order to produce this beta version of the app.
App Design
I have included the original academic paper that I submitted when the app was first designed.
Multimedia is changing and expanding exponentially. Ever increasing internet speeds mean that optical and printed media are rapidly become obsolete as people chose to download their films, music and even magazines and newspapers. This revelation has also provoked a change in the way we view and interact with multimedia. 3D viewing is entering the consumer market and handheld interfaces and touch screen interaction are becoming a part of our daily life. The video game industry has already changed the way people interact with games with products such as the Nintendo Wii, and upcoming Microsoft technology promises to negate the need for physical interaction with human interface devices all together. Could tomorrow spell the end for the tried and tested mouse and keyboard?
“Learning by Doing Is More Effective than Learning By Watching or Listening”
(Bartle: 2010)
The above ethos was the backbone for this project.
Having been made aware of the module content and major assessed brief, I knew I would be able to draw upon my existing experience of Adobe Flash (Y02).
Before the project began I created a production schedule (X00) which I followed (albeit loosely) to ensure that all elements of the production ran smoothly.
For the first step of the project I drafted a plethora of potential ideas targeted at various age ranges established in the brief. (X02-07)
The web cam interfaced interactive farm game was of a great interest to me however due crudity of the system used to detect notable change in live imagery it wasn’t a viable choice for young children. For the same reason the system didn’t lend itself to operating anything other than large well-spaced buttons and was thus thrown on the novelty pile.
The touch screen GCSE book was also tempting; however I had already done several similar things like that already.
I therefore decided to go all out and opted for the piano learning tool for children.
My first port of call was research. Although I had a brief knowledge of it, Guitar Hero (Y03) was the inspiration for my program, it’s a game where:
“Players use a guitar-shaped peripheral to simulate the playing of lead, bass guitar and rhythm guitar across numerous rock music songs. Players match notes that scroll on-screen to colored fret buttons on the controller, strumming the controller in time to the music in order to score points.”
(Wikipedia, 2010a)
There have been a number of spin offs (Y04) and other instruments such as the base guitar and drums kits are available. What is interesting about this product is that the enthusiasm and commitment children invest in it is dramatic. Despite this a seasoned player could be handed a real guitar and be none the wiser. If the guitar was replaced by a keyboard of course it would be a different story.
Keyboardmania (Y05) is an arcade game that works on this premise
“In this game up to two players use 24-key keyboards to play the piano or keyboard part of a selected song. Notes are represented on-screen by small bars that scroll downward above an image of the keyboard itself.”
(Wikipedia, 2010b)
Dr Kawashima’s More Brain Training – Masterpiece Recital (Y06) utilises the Nintendo DS’s adjacent screens. The one on the left displays a musical score and the screen on the right shows a piano where the participant uses a stylus to select the corresponding notes. It should be noted that this game has an option to display the notes as letters. In this respect it is probably the most similar example to my final product.
Synthesia (Y07) is an incredibly elaborate (relatively speaking) program that in fact used to be call Piano Hero until its author wrote:
“I’ve received a cease and desist letter from Activision for the use of the name Piano Hero.”
(Piegdon, 2007)
Although the programs features are too numerous to list, it should be said that the program will support any midi keyboard and can facilitate up to 49 keys. A virtual keyboard is displayed and coloured bars of various lengths (the length defining the duration of the note) drop down onto corresponding keys that the user is required to press at the correct time and for the correct duration. The program can also display side scrolling musical notes. At its most basic the program can be played using the ‘QWERTYUIOP’ keys on a computer keyboard.
My plan was to make a program similar to this only suitable for children as after conducting extensive research (Y08) I discovered an important link between music, learning and development.
Although highly functional, the script behind the program is relatively simple. It was designed with expansion in mind and as such the keys instances mimic their midi counterparts 60-76. Each virtual key is linked with an mp3 file and is also associated with the relevant key on computer keyboard.
For this project I used ActionScript 2.0. This was because the more I investigated potential methods for adding function to my application, the more it became apparent that AS2 is a much simpler script, to the extent where I questioned why there even was an AS3. The following reference answered my question.
“ActionScript 3.0 code executes much faster than earlier versions of the language – usually 2 – 10 times, but sometimes up to 100 times faster.”
(Florio, 2009: p.11)
Creating the Piano Hero Junior App
The biggest technical issue was orchestrating the correct timing and thus spacing of numbers that make up the song in Piano Hero Junior. The first method I used to solve this issue was recording the song and then playing it back while pressing the interval button on a stop watch each time a note was played. Subsequently to this I just played the song in my head while pressing the interval button instead of a piano key, this was an arduous process considering my stopwatch only allowed for 5 intervals. In the end it boiled down to mathematics. For example if my scene was set at 24 frames per seconds I could create a straight horizontal line, if the song lasted 20 second I would animate the end of the line to finish where the start of the line began over 480 frames. By putting incremental markers on line the, effectively turning it into a ruler, I was able to place each individual number that made up the song accurately. I soon discovered that because the songs are rhythmic I could accurately place the numbers by adjusting the kerning, thus negating the need to move each individual number.
The program was designed to be stepping stone to an actual piano, therefore it was iperitive that the interface resembled a piano as much as possible therefore I wanted to replace the piano keys (currently vector drawn) with photographic ones. I took a digital SLR camera to the small room near the drama department that contains a piano. Although I took 73 photographs (X08), the best ones were done when I turned off the lights in the room and took the photos on a long exposure.
There were a number of small imperfections with the keys which I was able to remove digitally with Photoshop.
“Digital imaging and the possibilities offered by the digital darkroom have opened up great new creative opportunities beyond anything possible using traditional means.”
(Cope, 2003: p.6)
I then set about cutting out each of the 16 necessary keys for Piano Hero. Initially I tried copy and pasting them into Flash, however they did not paste as cut outs. I soon discovered however that Flash could import Photoshop’s native format as objects. Because of the 3 dimensional perspectives that the photo keys had, my initial idea of animating the keys to show realistic depression as they were activated proved nearly impossible as the image area of adjacent keys would be affected with each press. To rectify this I used ActionScript to change the dimensions of the images on press to give the illusion of depression. To make the keys clickable I had to break the images apart and then use a brightly coloured rectangle (the colour was simply to make it easier to see) to place over the non-hit areas of the images and them remove it, thus removing that part of the image.
The sounds in the Piano Hero Junior app were tricky. At first I returned to the piano room with some digital recording equipment to capture the appropriate samples, however that room also doubles as some sort of server room and a notable buzzing sound can be heard from the cooling fan. Next I tried to sample the sounds from my girlfriends’ parents piano, however getting a consistent velocity and sustain on each of the 16 relevant (and slightly out of tune) keys was extremely difficult. So I resorted to using a midi instrument from a program called Reason. Because the grand piano profiles on the program are midi, there are no existing wav or mp3 files that I could have simply copied. Instead I had to play all of the relevant notes and then export the song as a wav before importing it into an open source audio editing program called Audacity, where I was able to highlight and save the relevant audio data and import it into Flash.
My designs started with hand rendering and a Photoshop mock-up (Y09), after which I considered the following statement:
“The power of the golden ratio to create harmony arises from its unique capacity to unite different parts of the a whole so that each preserves its own identity andy yet blends into the greater pattern of a single whole”
(Doczi, 1994: p.8)
As a result the programs window is proportioned to the golden ration of 5:8 (500 x 800 px) and its contents are harmonically proportioned within (X09).
Having heard that monochrome colours stimulate children’s development, I initially designed the program in black and white. However after being able to find no substantial reference for this I added a single colour (blue) to my design. The spinning piano was created by taking each key image and rotating it 2° more than its predecessor until a full circle was created, this image is also used for the CD print (X11). A number of the buttons work on some standard givens of human interface. The arrow pointing backwards takes you back, the forward pointing plays audio, the square shape stops the audio, etc.
Initially I designed several buttons within illustrator and Photoshop (X10) attempting to utilize a wood effect similar to the veneer of a piano. Having struggled to find just the right look I ended up creating and animating the buttons using a program called Swift 3D.
I considered the following statement:
“Usability is always relative to two questions: Who are the users, and what are they trying to do? The same design may be great for one type of user and horrible for another”.
(Nielson & Pernice, 2010: p.14)
Above all my product is aimed at children so my modus operandi was to make it simple and eye catching. That quote was taken from a book called Eyetracking Web Usability which was crucial in the development of my program. Although the program isn’t web based (yet) the book also states:
“There’s not much difference between eyetracking for Web usability and eyetracking for application software usability”.
(Nielson & Pernice, 2010: p.17)
Probably the most useful revelation I obtained from the book was the deconstruction of the human eye. The human eye has two fields of vision, the peripheral section which makes up the vast majority and the foveal vision which is “a small central area with a very high resolution”. As we flick our eyes (saccades) between fixations. A small amount of time elapses. As a result of this I moved the number scrolling bar from the top of the screen to the bottom to reduce the amount of time it took for the user to flick between that and the human interface device.
Peer assessment was not particularly helpful, however I was able to get my required feedback through a process of user testing.
My initial plan was to take the app into a nursery for testing. This was all arranged and approved however as the program developed it became clear that nursery age range wouldn’t be a suitable demographic. I began talks with a school in Dudley who were keen on the idea but understandably wanted to trial the program first. Frustratingly, developing the program to test ready status took a tremendous amount of time. Following this the school were pleased with the program however they insisted (again understandably) that I have a CRB check. At this point the time frame collapsed and I was no longer feasibly able to conduct the test before the dead line. However I do plan to trial a later release.
But I was able to trial the product on several people across a wide age range (Y10). I was able to get a number of my peers to test and review the program and I also made it available online with same questionnaire, which allowed me to correlate the data into graphs and charts (Y11). The results also tipped me off to several changes I will be making. I plan to develop the program to a much greater extent. This will probably involve a complete over haul.
From a design point of view the two background images are not my own. I may take my own photo of a concert hall, probably from the point of view of the pianist looking out at the auditorium. Something similar to the musical notes picture used for the background of the help and song selection pages could be created from scratch in Photoshop. The 3D buttons need to be created with higher resolution materials and they also need to been made to the exact size that they will appear on the program, as resizing in Flash causes edginess. I could take better photographs of the keys from a better piano and cut them out with more care and accuracy, again these would need to be exported at the correct size to avoid resizing in Flash. The samples could be taken from a real (tuned) piano using high end sampling equipment.
From a technical point of view the biggest change will be the ordering of text. At the moment the text is simply a timed animation. The text needs to be controlled entirely by script, by doing this; further improvements can be made more easily. For example on one song (Twinkle Twinkle) I was forced to cut out a verse so the song would fit on the maximum size of pithy stage. A scoring system could be implemented by adjusting a variable if the correct key was pressed when a number was in a certain coordinate range. More songs could be added with greater ease. A training mode could exist whereby the game would pause until the correct key was pressed. Another big issue that would need to be overcome is the ASIO (Audio Stream Input/Output) drivers, since most Windows computers have a slight delay between pressing a key and hearing the sound.
In conclusion I feel the app, piano hero juniorwas a success, In testing I’ve seen children struggle to play the song and an hour later they are able to play the same song practically fluently without the numbers. It took a copious amount of time to make and will probably take much more until I’m able to make a public free release of it.
References
Bruce, T. & Meggit, C. (2006) Childcare and Education. Fourth Edition. London, Hodder Arnold.
Cope, P. (2003) Secrets of the Digital Darkroom. New York, Watson-Guptill Publications.
Elam, K. (2001) Geometry of Design. New York, Princeton Architectural Press.
Florio, C. (2009) Classroom in a Book: The Official Training Workbook from Adobe Systems. California, Adobe Press.
Nielsen, J. & Pernice, K. (2010) Eyetracking Web Usability. California, New Riders
Pica, R. (2010) Experiences in Music and Movement: Birth to Age 8. USA, Cengage Learning.
Smidt, S. (2006) The Developing Child in the 21st Century. Oxon, Routledge.
Synthesia (2007) Piano Hero Name Cease and Desist from Activision. [Online] Available from: http://www.synthesiagame.com/cease.aspx [Accessed 14th April 2010]
Community Empowerment (2010) Learning by Doing Is More Effective than Learning By Watching or Listening. [Online] Available from: http://www.scn.org/cmp/modules/lit-l06.htm [Accessed 06th May 2010]
Wikipedia. (2010a) Guitar Hero. [Online] Available from: http://en.wikipedia.org/wiki/Guitar_Hero [Accessed 18th March 2010].
Wikipedia. (2010b) Keyboardmania. [Online] Available from: http://en.wikipedia.org/wiki/Keyboardmania [Accessed 18th March 2010].
