mapping :: pad, color, mixing, triggers, fx, meta

mapping :: pad, color, mixing, triggers, fx, meta

Mapping, in the case of graphicscore, is a matter of creating links between combinations of shape and colour, and sounds output by the audio engine. The first mapping is simple – points on the screen trigger sounds at a certain pitch, depending on their position on the Y axis. An array of these triggers is iterated across using a metronome, and sounds are played.

This very simple mapping between points and sound events is an obvious one to the user, and as such is a good basis for crafting the ‘sound’ of the application and creating a feeling of control for the user. Alone, however, this process of triggering samples creates a stark, staccato soundscape. Adding effects alone is not enough to enrich the sound image – for the effects to have a distinct impact, they must be set to such extremes that they become much less effective, and much more difficult to control.

In order to enhance the soundscape, a new voice is added – a pad. The pad is created using delays, taking the combined output signal from the triggers and creating a melange of the various timbres which are in play. The pad becomes representative of the changing overall timbre of the piece, slowing any stark changes, but still allowing the user to have obvious control over the piece.

One thing proved from several failed approaches to mapping in this project is that in order for a clear link between FX, mixing etc… and the shapes/colors drawn to the screen to be made, the FX should be applied globally, thus creating a large and audible impact, and building a connection between cause and effect in the mind of the user. The pad goes some way to blurring any major changes, and in concert global FX and the pad create a full and interactive sonic environment.

The mapping patterns which seem to be most effective at creating this connection between user and sound adhere to the following rules:

- Points -> Triggers

- Shapes -> Distinct sounds

- Colors -> FX parameters

- ‘Meta’ data (alpha, shape proximity etc…) -> Pan, mixing.

For example, the following mapping model seems to provide a flexible and rewarding user experience, and adheres to these rules (for the most part):

- Shape points are counted as triggers

- Each pointed shape triggers a different sample and therefore has a distinct timbre

- Red content degrades the overall sound using bitcrushing and  distortion. Green mixes a flanger into the signal, while blue affects two tremolo stages, adding texture

- Ellipses enhance the tremolo effect.

- Shape proximity (busy-ness) mixes the dry signal with the pad (a screen with many overlapping shapes will mix 100% of the pad signal).

- The ‘brightness’ of the pad is controlled by the literal brightness (derived through colour and alpha readings) of the shapes.

Using these rules, there seems to be scope for a great many variations using the same sample set. (This is a rule set in development.)

March 15, 2012 0 comments Read More
graphicscore :: listen mode in action

graphicscore :: listen mode in action

The video embedded above shows two devices in the audience of a session, using the graphicscore ‘listen mode’. In ‘listen mode’ the devices have no control over the audio output, but can listen to each loop, and are updated with any changes when they occur.

Note the expanded sample vocabulary which is being added in the process of developing the aesthetic rehearsal/training mode.

March 4, 2012 0 comments Read More
mapping :: ai, training, unique experiences

mapping :: ai, training, unique experiences

As mentioned previously, graphicscore.app is in a working state (there are a very small number of bugs remaining in the current state of the application), and at this stage, development moves towards harnessing the data flow which now exists through the application. In abstract, the users are able to draw to a shared canvas (synchronised using an internet connection), which is processed/parameterised by the application; the output of this process is manifested aurally by the audio controller. This last stage in the chain, where the parameters become audio, is the new focus of the application development process.

The first stage in this process of mapping is to isolate the data which is extracted from each frame.

A principal at the core of the way the application delivers sound is the division of the screen into four frequency bands, labeled W, X, Y, and Z in descending pitch order. These pitch bands are used to subdivide the incoming information, and create a more varied sonic texture. The scope of these variables has been divided, therefore, into ‘global’ and ‘local’, where ‘global’ variables are totals and averages, and ‘local’ are totals from individual frequency bands.

The global variables are:

  • Alpha total / average
  • Number of shapes on screen
  • Number of curves
  • Red
  • Green
  • Blue
  • Shape sizes
  • Proximity of shapes to one another

And local:

  • Points
  • Curves

This data is used to control the range of effects held by the audio controller:

  • Flanger
  • Bitcrusher
  • Distortion
  • Delay
  • Tremolo
  • Filter

Additionally, the audio controller chooses its source material according to the parameterisation process (this is frequency band localised).

For this mapping exercise to be considered effective, the source parameters must have a clear connection to their sonic manifestation. With a wide range of effects available, and a number of source sample choices, there is an opportunity for a broad range of sounds to be output by the application. Listed below are a series of mapping possibilities:

Creating a static mapping for these parameters would produce a distinctive sound for the application, and in fact might be considered a compositional process on the part of the software developer. However, from a user perspective, one might be put off  using the application if exception is taken to the aesthetic choices made in the mapping stage. In order to counter this notion, an active, mutable mapping system must be placed between the ‘parameteriser’ and the audio controller.

In order for this intermediary object to exert control over the output in a manner customised by the user, it will require a ‘training session’ to create a profile, or ‘fingerprint’  for the user (heretofore referred to as ‘Rehearsal’, in keeping with preexisting nomenclature). The purpose of this rehearsal session is to define their preferences in the handling of the raw numeric parameter stream through a series of aesthetic choices – ‘sharp’, ‘spacey’, ‘dirty’, ‘clean’, ‘smooth’, ‘metallic’ etc…

The rehearsal process will consist of a dozen or so predefined scenarios, featuring a variety of shapes, colours, proximites etc…, which the user will be asked to rate on a sliding scale. The results of the rehearsal session will be used to create a profile for the user so that they have a unique sonic experience when using the application. The profile will be saved to the device, and overwritten if another rehearsal is undertaken. Additionally, the rehearsal process might be used to match users to specific partners who have a similar profile. In this case, the two connected users would be hearing the overall composition played back through their respective profiles which, while similar, will produce different audio for each party.

March 3, 2012 0 comments Read More
graphicscore :: multiplayer, listening

graphicscore :: multiplayer, listening

The core mechanics of graphicscore are now in place, with a defined path for data from client to server (and vice versa), and from the images drawn to the screen to the audio engine via a parameterisation process. Now, the project shifts from implementation details to the organisation of the parameter data, and mapping to audio features. The application currently features a multiplayer mode (which is in test across multiple iOS devices) and the newly added ‘Listen’ mode, which connects a guest user to a current session as an ‘audience member’. Below are some screenshots of the application in action between two devices:

graphicscore.app on iPhone homescreens

The graphicscore menu screen, with options to begin the multiplayer experience, or listen in on another performance

A multiplayer graphicscore session. Players one and two are drawing to the same canvas, sharing the objects they uniquely have access to. Player one has drawn a red ellipse and magenta star, while player two has defined a blue triangle and a green star.

In another instance, a live player is involved in a session, while a guest is connected in ‘listen mode’. As the players draw more shapes to the screen, the guest screen is updated with the new content.

March 3, 2012 0 comments Read More
networking :: requesting the session state

networking :: requesting the session state

The final link in the network transaction chain is the request for and reconstitution of GSShape objects from the server. As shown in the data submission section of the network model, the client can, having requested a session_id and member_id, transmit their data to the server, where it is committed to a MySQL table which tracks the state of both clients, and represents the shared graphicscore canvas.

An excerpt from a typical session table is shown above From this, it is determined that the fourth shape drawn by player two (starting from row 81) is triangle, with a frame of origin (340, 61) and size 132*143. The triangle has a colour index of 3, and an alpha channel set at 0.817907. As it is a GSTriangle object, it has three additional variables which other shapes do not – a left point at 40, peak at 91, and right point at 27. This data, accessible to both clients, can be used to reconstitute these GSShapes, allowing both clients to draw to the same canvas.

The touchesEnded method of TouchArea is called whenever a shape has been drawn to the screen by the user, and is already used to trigger the submission process. Because data structures are now being shared between objects which employ network communication, timing becomes important to avoid bad access errors through mutating an array which is being accessed by another method. The Parameteriser is one such object which requires access to a data structure whilst it processes its contents, therefore the call to this processing method is made within a loop which prevents the application from proceeding before it is completed.

At the final line in the listing above, a call is made to processIncomingDataFromNetwork:, which is passed the return of the GSNetworkController requestData method as its argument. This is the first stage of a request for data.

requestData makes a request to the file request_data.php on the server. This response from this request is used to populate the responseFromServer NSMutableArray, which is then returned (in the working example, it is therefore returned as the argument to the processIncomingDataFromNetwork: selector in TouchArea).

request_data.php, the server-side script which responds to requestData with the current contents of the session, is listed above. When the request is made to the server, the client’s session_id is appended as an argument to the URL. This value is used to request relevant table from the database, and is assigned to the $id variable by requesting it from $_GET (the URL has been constructed in such a way that $_GET can be queried by key, which is known to be ‘id’).

$result  is assigned as the return of a query to select all entries from the session_ table specified by $id. The entire contents of this table is to be returned to the client in a form which can be used to initialise an NSArray, which is prepared using the createPlistForResult() method. As with the ID request procedure, this method synthesises a Plist file which mimics one which would be created by calling one of the various NSArray writeToFile: methods.

This method employs a loop to extract the value in each MySQL table row and add it as a string to the property list. Even though only half of these values will be needed, the size of the response file is just 2KB and requires a single MySQL query.

The result of this request is passed as an argument to processIncomingDataFromNetwork, which populates the screen with the shapes downloaded over the network. An equivalent to the shapesOnScreen array, shapesFromNetwork stores the incoming GSShape objects which are created using the incoming data from the server. The contents of both is drawn to the screen, and they are sent together to the Parameteriser, which combines their contents into the parameter stream for the audio component. Because the full session state is downloaded, but only the remote client’s shapes need to be added, the loop which renders the shapes is bounded by the start variable, which is set to run the loop from shapes 5 – 9 if the local player is member 0 (player one), and from 0 – 4 if the local player is player two (member 1).

processIncomingDataFromNetwork: is responsible for the view/subview handling which renders the shapes to the screen, but the method which actually recreates those subviews is the createShapeUsingParameters: withIndex: method, which is called for each of the incoming shapes. The index variable is used to calculate the offset, which is a mirror of the submission process, and is used to determine the generic GSShape common characteristics from the data stream, which are assigned to recreate a CGRect frame. The shape_id variable is used to determine what the actual type of the shape is – this type then replaces the GSShape generic, and is initialised with the frame. Again mirroring the submission procedure, any extra variables required by the GSQuadrilateral or GSTriangle types are assigned. The shape is assigned the appropriate index value, colour palette, and alpha value, and is returned. The return type for this method, id, is an Objective-C generic type, so any object can be returned – in this case it is employed so that any of the shapes from the GSShape library can be returned from the same method.

February 15, 2012 0 comments Read More
networking :: submit data to the server

networking :: submit data to the server

Each user has control over five shapes in the session, which are stored in the table session_*. Each session table has 100 entries – (up to) 10 variables per shape times five shapes times two users – where the first player (with a member_id of 1) has control over the entries 1 – 50, and the second player has control over entries 51 – 100. Therefore, each submission requires 50 pieces of data to be transmitted to the server. This data is derived from the shapesOnScreen NSMutableArray and submitted using the PHP _GET method. _GET is a somewhat unattractive and insecure method of data transmission, as it uses the URL string to transmit data, however as this process is occurring out of the user’s sight, and the data is not sensitive these points are rendered moot.

The first stage of submission is made by the touchesEnded method in TouchArea. This is called when a touch event has ended, in this case when the user has finished drawing a shape to the screen, and after processing the drawing event locally – adding the new shape to shapesOnScreen and sending the updated array to the CAController instance via the Parameteriser – shapesOnScreen is sent as an argument to the [GSNetworkController submitData:] method.

submitData prepares the NSString URLSuffix, which is appended to the remote address of submit.php. URLSuffix is a string containing the 50 pieces of data identifying the shapes contained in shapesOnScreen, and is prepared by the URLGetStringForShape: withIndex: method, which will be detailed a shortly. Once this string has been appended to the URL, instances of NSURLRequest and NSURLConnection are created, using serverAddress as their target. When the connection has been configured, it is instructed to start, completing the transmission.

This is a different process to the method used to request the member_id & session_id variables from the server, where a local NSArray is initialised with the contents of the response from the server, because the server does not need to respond with any information. In an extension of this method, the server might respond with an arbitrary value to confirm that the data has been received, which could be used as a connectivity test for the client. This ’1.0′ implementation, however, uses the NSURLRequest technique listed above.

URLGetStringForShape: withIndex: takes a GSShape from the incoming shapesOnScreen NSArray, and prepares an appropriately formatted URL string containing the data derived from the shape, and its index in the remote SQL table.

As detailed in the plan above, each shape is described by 7 pieces of common information (id, alpha etc…), with 3 extra rows for unique information (the points of a triangle, angle of rotation for a quadrilateral). URLGetStringForShape: withIndex: prepares a URL string for a shape by creating a series of key-value pairs for each of these pieces of information. The values are derived from the shape passed as an argument, with a couple of additional assignments for the unique identifiers when a GSQuadrilateral or GSTriangle is passed, while the keys relate to the ID of the row in the remote table. This value is calculated as: ((((member_id-1)*5)+number_of_shape)*10).

member_id-1 equals zero for member 1, and 1 for member two. Using this formula, member_id 1 will address the table entries with an offset of zero, while member_id 2 will have an offset of 50. When the URL string is prepared, the keys for each value are calculated as the index of the shape plus the offset plus the index of the value. While the local variables are zero-based, the MySQL table is one-based, so there is some occasional mathematical oddness in this formula (such the 5th shape being referred to as shape 4, with an offset of 40).  For example:

Player one, shape 5

(((0 * 5) + 4) * 10) = 40 (rows 41 – 50)

Player two, shape 2

(((1 * 5) + 1) * 10) = 60 (rows 61 – 70)

Player two, shape 4

(((1 * 5) + 3) * 10) = 80 (rows 81 – 90)

All of the URL string formatting work done on the client-side means that the server component of the submission is remarkably simple. The iOS client has formatted the incoming data so that the $_GET array can be interacted with directly, with no need for any server-side typecasting or formatting. $_GET is iterated across by a 1-based for loop, and updates the keys in the database table with the values they have been paired with by the client. In fact, the way in which the client prepares the submission string means that the two members will never be able to overwrite each other’s data: when client one submits to the server, it will use $_GET keys 1 – 50, while keys 51 – 100 will have no argument in the URL string (the reverse is true of player two) – when the server iterates across $_GET, no entry will be found for the missing half, and the SQL query will not be executed.

February 15, 2012 0 comments Read More
networking :: requesting an id

networking :: requesting an id

The network controller is created by the UserScreen instance in the init process for that object (which sets up the whole user runtime). As part of its init method, GSNetworkController requests a session ID from the server by calling its own request_id method (below):

request_id populates the two local ID tokens ‘session_id’ and ‘member_id’ which are owned by the network controller and used to identify the instance for all subsequent requests to the server during this session.

The NSMutableArray is ’responseFromServer’ initialised with the response from the file ‘request_id.php’ on the server, listed below.

The goal of request-id.php is to assign a session ID and member ID token to the client. The outcomes are as follows:

  • An ID is free and the user is assigned as member 2
  • All current IDs are full (have 2 members). A new ID is generated, and the user is assigned as member 1.

This is accomplished through the following process:

  • First, the maximum ID is determined and assigned to the $max_id variable.
  • Next, the database is queried for the number of members in this row (the result value is assigned to $members)
  • $members is used as the argument to a switch.

If there is only one member, $max_id is the session_id to be returned to the client. The database is updated, with the members for $max_id updated to 2.

The other scenario is that the session $max_id is already full. In this case, a new session needs to be created, through a call to ‘createNewSession’. This method inserts a new member value of 1 into the session_tracker table. Because id is set to automatically increment, MySQL takes care of creating the new row with an incremental id value. $max_id is then reassigned (again, using getMax), which is used to create a new table with the incremented id value. This table is then initialised with 100 rows (used to store the session data) by a for loop, with zeroes assigned to the ‘data’ column.

When the final values of $max_id and $members are confirmed, they can be returned to the client. NSArray type objects can be initialised using ‘…a file containing a string representation of an array produced by the writeToFile:atomically: method.’ Files created using this NSArray method are XML documents in the Plist format, and this is what needs to be spoofed in the return from the request-id.php script.

The createPlistForID method takes id and member variables as its arguments, and returns a string ($xml) formatted to mimic a saved NSArray containing two strings, the first containing the id, the second the member value. This string is then echoed as the return value for the script, the content used to populate the NSArray in the GSNetworkController.

If the request to the server fails, either through an error on the server side (an inability to connect to the database, for example), or a problem connecting to the internet on the part of the client, the responseFromServer will remain unpopulated, with a count value of 0. If this is the case, the initial values assigned to session_id and member_id of 0 remain.

The rest of the initialisation process for the user space is dependant on the value returned by this process: an integer value above 0 can only be the result of a successful connection, and the init process continues. However, if the value remains at 0 then an error has occurred. If this is the case, the program presents the user with a notification, listed below first in code, and then the notification itself.

The UserArea assigns itself as the delegate for this UIAlertView, and uses the UIAlertViewDelegate protocol method UIAlertView: didDismissWithButtonIndex: to handle the dismissal and destruction of the UserArea once the OK button has been pushed, returning the user to the main menu.

February 13, 2012 0 comments Read More
classes :: gsnetworkcontroller

classes :: gsnetworkcontroller

Below are listings for the GSNetworkController object. Because this class is designed to operate in conjunction with a server side component, actual commentary will be made alongside the listings of the server software, in a breakdown of the transactional process.

February 13, 2012 0 comments Read More
sketches :: network architecture

sketches :: network architecture

One of the fundamental components of the application is its connectivity between clients. Each session is a two-player event, and both have a unique ability to affect the sonic output (with different colours, for example). This networking has to be implemented both on the client and in a separate piece of server-side software. The network solution, in various degrees of abstraction, is listed below.

Each time a client updates their screen by drawing a new shape on the canvas, the current ‘state data’ is submitted to the server. Once the server has received the data and responded with a confirmation, the client then requests an update of the current state, synchronising with the contents of the database.

In order that the clients can address the relevant database, they both need to be assigned the same ‘session ID’ value when the session is initialised. This token is then used as an argument to the submission and data request calls. The first role of the network controller is to request this ID value from the server, so that it can be used during runtime in these requests.

A table in the database, ‘session_tracker’, has two columns, one for the ID value, and one with a track of the number of members in each session. The logic is listed below:

For example, a client will request an ID. The maximum ID session has only one player, and therefore is returned. Now, both players can submit to the same database with which their canvases are synchronised.

The session tables must be of adequate size to hold all of the required information to draw all of the shapes on each screen exactly. Examining the GSShape base class, and the unique identifying features of its subclasses, a maximum of 10 pieces of information are required for each shape:

Each shape will have a maximum of 10 identifying features:

  • Shape ID
  • Origin X
  • Origin Y
  • Width
  • Height
  • Color index
  • Alpha value
  • Up to 3 unique identifiers

The unique identifiers are, for example, the point positions of a triangle, or the angle of rotation for the quadrilateral.

With 10 pieces of identification, and 10 shapes on screen at any one time (each player has five colours), the session table must hold 100 pieces of information.

With a table allocated, both clients can now begin to submit their shapes to the same database table. As they are committing to a shared table, when a request is made to pull its contents, the client states are combined. Member ’1′ commits their state to the first 50 values, and the second 50 pertain to the second member. When the client makes a submission to the server, it addresses specifically its assigned range, preventing one client from ‘undoing’ the submission of another by overwriting their ‘older’ version of the state before pulling the update.

February 13, 2012 0 comments Read More
classes :: audio fx library

classes :: audio fx library

The maximilian library contains the building blocks for creating digital audio effects, using filters, delay lines, and oscillators. By combining these objects, complex effects can be created, however this is a relatively ‘code-heavy’ approach. As an extension, a library of more complex effects units, which combine the building blocks from maximilian into standalone audio processing objects, has been created. The aim of the FXLibrary is to place an objective wrapper around these blocks of code, allowing for them to be assigned different parameters and connected in different orders (akin to a guitarist’s pedalboard).

The initial commit to the FX Library consists of:

Bitcrusher (variable downsampler)

Delay

Distortion

Filter

Flanger

Tremolo (amplitude modulation)

Each of these objects takes a signal input and control parameters as input, and returns the affected signal. These effects units can be reordered and controlled by the parameter stream derived from the shapes drawn to the screen.

 

February 13, 2012 0 comments Read More