in development :: palette class

in development :: palette class

The series of ‘in development’ posts will cover individual classes at various stages of the build, and serve as a reference to the changes made during the development process.

An early sketch planning the palette class

Planning some changes to the original specification

 

As the user’s means of interaction with the application is through drawing, colour plays a vital role in their control over the state of the program. A two-player session will assign a unique palette of colours to each player, which they can mix on the canvas to create new colours, or use to fill the background colour of the canvas itself to change the overall ‘feel’ of the composition.

In the initial planning for the palette, it was to be responsible for drawing the colour selection controls to the screen itself. Furthermore, it was planned that the parent of the palette object register as its delegate in order to receive instruction from it. In the second image above, the palette has undergone some revisions which change its design considerably (for example, it is no longer required to draw to the screen, and has been switched from type UIView to an NSObject subclass).





For the program to work as intended, each player must have a unique palette of colours available to them. The client which chooses to launch a session creates two Palette objects, one for each user. The first of these draws five colours at random indexes from the ‘availableRangeOfColors’ array, using the ‘create’ method. The indexes of these colours is stored in the ‘colorsTaken’ int array. Once this process has been completed, the second palette is created from the remaining colours in the available range (the second palette is the return value of the ‘createOpposite’ method).






This is the ’1.0′ implementation – it works as intended, but contains log statements, and has not yet undergone any optimisation. Using this type of Palette object, the instigator of a session can transmit the palettes of both clients to the server at session start – the joiner/second party can then simply download their palette from the server, and it will be guaranteed unique.This also means that colour data need not be transmitted multiple times – a shape can simply make reference to the user, and index of the colour in their palette (for example U = 0 C = 4 for user one, colour four).



 		

		
		

			January 27, 2012			 0 comments  						
			Read More		

	

	
		
	

		sketches :: data flow and classes
		
sketches :: data flow and classes
		
		

 			
As the task of formalising the structure of the program progresses, a clearer definition of classes, data types and flow emerges. Below is a sketch of how data flow from both users could be implemented, from touch input to audio output. The ‘path data’ refers to the raw coordinates drawn from the path input, however, this is likely to be developed into a ‘shape specification’ data which will require much less bandwidth.



 		

		
		

			January 27, 2012			 0 comments  						
			Read More		

	

	
		
	

		sketches :: network controller
		
sketches :: network controller
		
		

 			

 		

		
		

			January 27, 2012			 0 comments  						
			Read More		

	

	
		
	

		notes :: networking use cases
		
notes :: networking use cases
		
		

 			
Part of the process of determining the class structure and determining the responsibilities held by each object is writing use case scenarios for the application. As the networking component requires action from three parties (client, server and remote client) these use cases are useful for stepping through an operation. This process dictates the required behaviours for networking classes, both client- and server-side.


 


Interruption & Disconnection


A client is in a session, when they lose connection to the server (this might be through a loss in cell signal, wifi outage, or they perhaps receive a telephone call). When the user returns to the program, it must begin running in ‘offline mode’ – the session resumes, but with the networking component disabled.


User starts a multiplayer session [internet disconnected]


The user selects ‘Enter Online Multiplayer’ from the main menu screen. Before attempting to start a new session, the program must establish that a network connection to the server is available. When this process fails, the user is informed (via a UIAlertView popup) that a connection could not be established, and offers to start a session in offline mode.


User starts a multiplayer session [internet connected]…


The user selects ‘Enter Online Multiplayer’ from the main menu screen. Before attempting to start a new session, the program must establish that a network connection to the server is available. When this process succeeds, the application can start establishing a session with the server.


…and there is no immediately available partner


All currently available sessions already have two players, so a new session ID is created and returned to the application. The user is informed that the session is awaiting a second player, but is allowed to start creating the score on the screen.


…and a session is available


The server returns to the client an ID for an existing session as well as the existing state of the session. The client screen is populated with this data and the user can immediately begin interacting with the canvas.


 



 		

		
		

			January 27, 2012			 0 comments  						
			Read More		

	

	
		
	

		sketches :: planning core functionality
		
sketches :: planning core functionality
		
		

 			
Posts to the ‘sketches’ category are diagrams which have been drawn during the planning process. 




Preliminary plans for some core app functionality.




Setting some plans for the Audio Controller. This class needs to be implemented implemented in multiple languages (C / Objective-C / C++), which need to receive data in formats appropriate for each language, likely to be either an NSArray variant or an unsigned char pointer array.




Gathering some ideas for user input. The touch area has now been fixed into the specification for the program, however the user input is now going to be handled as a single entity (rather than analysed through the ‘playback window’). The extension functionality – using MFCC and FFT analysis – is still possible, and can be used to gauge the overall spectral quality of the canvas.




A few multiplayer and networking features. Connecting players as individuals, using something like Apple’s ‘Game Centre’ (through the GameKit framework), makes disconnection handling more difficult, and also precludes a ‘performance’ mode, in which a client can connect to a session being played by two other people and listen to their composition evolve.

 		

		
		

			January 26, 2012			 0 comments  						
			Read More		

	

	
		
	

		notes :: multiplayer and networking
		
notes :: multiplayer and networking
		
		

 			
This series of posts are of notes made in the process of writing my requirements documentation.




Core Functionality


The core of the networking component of this application facilitates the connection of two users in a session. Each user is assigned a different color palette, giving each player unique control over the composition. They must mix colors on the screen to affect the audio output, with some textures only accessible by mixing with the other player’s palette. The user is to be totally blind to the connection process (they may be presented with a ‘connecting’ screen while the connection is negotiated, but they are not required to ‘choose a server’ or enter a ‘lobby’ of any kind; the experience is totally ‘hands off’).


When the player requests to enter a session, they will request a ‘session ID’ from the server. If there is a session in progress with only one player, that ID will be returned, otherwise a new ID will be generated. Each path will be submitted to the server when it is updated with the following information:


    

  • Session ID
    • 

  • Player (1 / 2)
    • 

  • Path number (1 – 6)
    • 

  • Color ID
    • 

  • Path coordinates
    • 



The path coordinates are stored in an NSMutableArray object which is populated either with the user’s input (from the local device) or from the other player’s input (from the remote device). NSMutableArray is a serialisable object which can be filled with the contents of an XML Property List (henceforth referred to as plists). Plists can be dynamically generated by a PHP program, and this is how the remote state will be delivered to each client.


Each user submits their local state to a PHP script on the server. The data highlighted above is passed as a series of arguments to this script, which stores the session data in a MySQL database. The user creates an NSURL instance which is dynamically populated with their session data. This is then passed as an initialisation argument to an NSMutableArray, which will be used to store the response data. By using this technique, the user submits their state data and receives the state data from the other player in one action.


Secondary Functionality


Not all target devices have the same feature set. While the screen resolution differs between devices, the software handles this by scaling the touch input to a common resolution, normalising the data to be processed. However, it cannot be guaranteed that the device will have an internet connection: while the iPhone has a persistent cellular connection, the iPod Touch requires wifi to connect. The program will therefore require an ‘offline mode’. Essentially, this will be identical to the regular running of the application, however instead of dividing the palette between two players, a single player will have control over all shades.


Even with the cellular data connection, there is every chance that a user might be disconnected from the server in an area of poor signal. In this event, they must be presented with a notification that the connection has failed, and be offered the opportunity to enter offline mode. At any point during this ‘disconnected’ mode the user can attempt to reconnect to the server.


Extension


A further online ‘listening mode’ would request an existing session ID from the server and forward the contents of that session to the client for both players. The client will have no control over the session, and will simply be monitoring a session being played by two other players. This mode could be used for a ‘live performance’ of the software.



 		

		
		

			January 26, 2012			 0 comments  						
			Read More		

	

	
		
	

		notes :: user input
		
notes :: user input
		
		

 			
This series of posts are of notes made in the process of writing my requirements documentation.




Core Functionality


User control of the program will occur through the device touch screen. On an abstract level, the user will paint to the screen by selecting colors from an on screen palette. The application will parameterise and map this data to the audio output, creating events, effects, pitch & rhythm changes, and other audible changes. The user uses this ‘graphic score’ to compose their session. On a more technical level, the colors in the users’ palette will have an alpha channel, allowing multiple different colors to blend and create new colors (which will be mapped to additional audible controls).


Secondary Functionality


The images painted to the screen will be subjected to a feature extraction process. Each shape drawn to the screen will yield a number of parameters, for example how angular the shape is, its size, and the thickness of its border. Even if the line isn’t ‘joined up’ (and so doesn’t appear as a shape on the screen), the program will still be able to extract features from it, for example how long the line is. The ‘angle’ analysis will still apply – a zig zagged line will have a great number of angles, for example.


Extension


In order to extend this base of functionality, some more advanced parameterisation techniques could be used. For example, FFT or MFCC analysis of contents of the window is possible, yielding a spectral decomposition of the contents of the window. This could be used to route the signal through different effects units the parameters of which are set by alternate analysis of the window itself.





 		

		
		

			January 26, 2012			 0 comments  						
			Read More		

	

	
		
	

		notes :: audio engine
		
notes :: audio engine
		
		

 			
This series of posts are of notes made in the process of writing my requirements documentation.


Core Functionality


The audio engine object will be responsible for interfacing with the audio hardware of the device. It will run at a 44.1 kHz sample rate, which will result in high quality, and shouldn’t have a detrimental effect on application performance. Although the device only has a mono speaker, the user will likely use headphones when interacting, therefore the audio interface will need to run in stereo.


A method of the audio interface (which will be called by the audio rendering callback itself) will be used as a runtime for sound synthesis. Because it includes synthesis and sample playback, the Maximilian library will be used.


The object will be visible externally as an Objective-C NSObject subclass, however internally it will employ Objective-C, C++, and C. CoreAudio requires C implementation, while the Maximilian library is implemented in C++. These language variations will be employed where appropriate.


This object is fundamental to the operation of the application; any extension of the responsibilities and specifications outlined above will likely be made in pursuit of optimisation.

 		

		
		

			January 26, 2012			 0 comments  						
			Read More		

	

	
		
	

		graphicscore :: github
		
graphicscore :: github
		
		

 			
The graphicscore project is now linked to a repository on github. By doing this, it is possible to sync the state of the project across multiple computers – my desktop and laptop – and keep track of the changes made to the code through the development process.


Any significant commits to this repository might also be accompanied by an update to this page.

 		

		
		

			January 26, 2012			 0 comments  						
			Read More		

	

	
		
	

		graphicscore :: hello world
		
graphicscore :: hello world
		
		

 			


This is the development blog of my final major project for Goldsmiths’ Music Computing course. My project, ‘graphicscore’, is an online, two-player music creation system. Players share a canvas upon which they create the graphical score for a constantly developing piece of music. Colors and shapes interact with one another to create new textures using filters, sound synthesis, samples, and effects units.


The application is being developed for the Apple iOS platform, and will be available for iPhone and iPod Touch.

 		

		
		

			January 26, 2012			 0 comments  						
			Read More