Musical Fireflies

(Top video = playtest with revised p5js sketch / Bottom Video = Testing the LED Matrix + P5js)

 

Musical Fireflies

Full Screen link  // P5js Editor link  

Description: Catch the fireflies on the screen to create a melody. As you catch them on the screen, they are caught from the “sky” and appear in real time in a mason jar (as illuminated LEDs). Originally envisioned as a chord building exercise and evolved into a Chance Aesthetics style Melody creation activity.

Once you’ve collected them, you can release them back into the sky by clicking on “release fireflies” button. When clicked, all the caught “fireflies” in the mason jar are released back into they sky (turning off the LEDs/resetting the LED matrix).

Still working on a 2nd button called “Listen to my melody,” that will play through the collected array of notes/fireflies that exist in your jar. Will continue working on it this next week  : )

I think for my oscillator to play through my array song [], I will look into setInterval(), setTimeOut(), JavaScript Closure. Could also take another look at the p5js Song Example .

Screen Shot 2018-12-10 at 2.57.19 AM

 

 

 

1) Sketch Iteration

IMG_4549

1.1 – To build my sketch I rewatched & reworked through many Coding Train tutorials, starting back with Coding Train 3.2./ Bouncing Ball. 

 

1.2 – Then started to think about how to create arrays for my circles referencing Coding Train 7.3 Array of Objects / OOP & making a class of fireflies. This dove back into the playlist on Arrays & Objects. Thinking about how to have the flexibility to have 5 or 500 fireflies with the change of one variable.

 

1.3 – To have the fireflies eventually move in to digital (or physical) jar, needed to understand / have the firefly behavior change when clicked.  Coding Train’s 7.4 Mouse Interaction With Objects covers just that. When the firefly is clicked, the sketch checks to see if the mouse x,y coordinates overlap with the diameter of the circle.  If they do then can have  “Clicked on Firefly!” print to console.

 

1.4 – Office Hours [Jasmine] + Coding Train 7.1 – 7.5

I decided to sign up for office hours with Jasmine Soltani to help me process the needed building blocks for my sketch. I brought in a version of the sketch seen below in 1.5.  I became stuck on thinking about the best way to assign a musical note to each firefly, but also how to do a rollover blink accent as well as splicing and copying to a new array. She walked me through a sketch she made her first semester that covered various strategies including Coding Train 7.4 – Removing Objects from an Array. To select my frequencies in my I used the Musical note frequencies from a chart through Michigan Tech University here

 

1.5 – Office Hours [Jasmine & Allison] + PCOMP usertesting

Then I scheduled office hours with our ICM teacher Allison Parrish to also think about possible array and arrays of objects. Later that week in Danny Rozin’s PCOMP I received some valuable feedback during our initial play test, that if I was to have an actual physical  mason jar next to the sketch, that maybe the sketch should be more atmospheric verses displaying another jar. That idea made a lot of sense to me : )

In preparation of our ICM user testing, I found a placeholder image of a summer night sky and a creative commons ambient recording through freesound.org. (**jar image above was also a clipart placeholder image. If had gone that design route would have drawn one in a similar style ❤ )

 

 

1.6 – ICM Usertesting, Office Hours [Yeseul]

Our ICM usertesting class was extremely helpful. Classmates offered lots of great insight, advice, and ideas for future iterations:

  • Button aesthetic
    • confusing to user / unclear when or in what order to click. one playtester had an association with a pop-up add button, and thought maybe she needed to click out of the button first in order to interact with the rest of the screen.
  • Add real firefly image to help establish that it is a firefly
  • createCanvas(windowWidth & windowHeight); 
    • to help make the image fullscreen / the user feel more immersed in environment
  • enjoyed the yellow blink rollover & ambient sounds 
  • oscillator doesn’t have an envelope attached 
    • so when 2 fireflies have middle C as their tone consecutively, it feels like an error due to it being one continuous tone.
  • Restrict “Release Fireflies” button to only redraw max original amount ex:9
    • was redrawing an infinite amount vs returning a caught amount back to the screen

I also met with resident Yeseul to talk about LED/lighting solutions for the PComp aspect of the project. She encouraged me to take a look at DIY LED matrix tutorials ❤

 

 

 

1.7 – Revisions, Office Hours [Yeseul, Danny, Luisa],  Final class

In response to ICM usertesting, I found a creative common’s image of a firefly and inserted it over the Firefly Object dots [following coding train 7.8 Objects & Images]. I also worked further into splicing and pushing to a new array. When a Firefly is clicked it is assigned a random tone frequency from my array noteFrequency[]. It is spliced from the array fireflies[] and instead has the noteFrequency then pushed to a new array song[].

After reworking through some of Danny Rozin’s Serial communication examples, I got my 3x3x3 LED cube communicating serially with p5js to arduino. p5js sends a ‘1’ to the arduino to communicate which LEDs to light up. If the button “release fireflies” is selected, p5js will serialWrite(0); to the arudino side, resetting the LED matrix to all off.

Luisa and I talked through some ideas and approaches to sound for the next couple iterations. She also recommended a MUX shield to me. She also talked about how instead of using an oscillator, I could possibly play with repitching the same sound file down the line. She also reiterated that maybe Melody make more sense than chord building, even though originally I changed it due to struggling to make it polyphonic via code. That the experience of capturing a firefly really is usually one at a time.

She also mentioned for the animation style, that since the background is realistic? Maybe it would help if the firefly image reflected that as well. Or the flipside, having the background becoming more illustrative than photorealistic. Either way to try to match the visual language maybe? Will have to test it out ❤

 

 

1.8 – Post Finals Revisions + Office Hours [Luisa]

Was able to find a creative commons background image and apply it with windowWidth & windowHeight, allowing for a more responsive screen. Was able to get it running on the p5js side on an ipad by having both devices run on the same ip address (via Tom & Jeff’s External Screen tutorial ), but will still need the hardware components to make it interact with Arduino.

Was able to fix my Oscillator issue! I kept overwriting where I tried to connect it to an envelope in my firefly class. Now that the two are connected, even if  fireflies have the same musical note 3 times in a row, you can still hear each one as an individual. Luisa also helped walk me through how setInterval could work through an office hours example here. Excited to fold the concept into to my current sketch.

Screen Shot 2018-12-06 at 12.05.12 AM

 

 

 

 

2) User Testing / Folding in Feedback for Future

Screen Shot 2018-12-09 at 5.38.04 PM

A consistent and great piece of feedback I’ve received is to really think about firefly behavior, and how that might translate into light. I went more in depth on a post here .  When researching more into firefly light behavior, I came across a really wonderful post called   “Learn to Speak like a Firefly” by Science Friday’s Education Director Ariel Zych. She referenced a really great study called “Studies on the Flash Communication System in Photinus Fireflies” by James E. Llyod [Museum of Zoology/University of Michigan]. In it is the picture above, where they’ve distinguished different blinking patterns associated with different species.

The ones I’m most used to seeing is #8 on their diagram above, of the Big Dipper Firefly/Common Eastern Firefly[Photinus Pyralis]. It would be fun to try and incorporate that dimming, swooping ‘J’ pattern that’s associated with it. At least on the p5js side? Would also be fun to play with PWM / brightness levels to create more of firefly effect.

I also think that Daniel Shiffman’s Coding Train: Particle Systems could be a step in the right direction to create a more firefly like animation behavior?

 

3) Serial Communication + LED Matrix/Multiplexing  

IMG_4698IMG_4697

IMG_4550

  • when deciding which 3x3x3 tutorial to go with after trying a few, I decided to go with DIY 3X3X3 LED Cube with Arduino by Dilip Raja. I ended up stripping the Arduino code down to where for now it lights up 1 LED at a time. When all 9 are clicked, it enters a victory pattern where all nine are lit up in each layer, 1 layer at a time. It creates the feeling of synchronized fireflies

 

 

 

4) Inspiration & Resources

Screen Shot 2018-12-12 at 5.24.54 AM

Yeseul Song & Jeff Park – Liquid light (2016) https://vimeo.com/197146445

3d_cube_2.jpg

James Clar / 3D Display Cube (2001-2009)  http://www.jamesclar.com/portfolio_page/3d-display-cube-2002-2009/

 

A lot of my references and inspiration came as I was building. After I built the 3x3x3 Matrix (trying 3 different variations, going with  Dilip Raja’s tutorial), I started to find a lot of reference videos for similar Firefly Arduino projects. The idea was inspired by summers in TN & MO and also specific memories of a bike path in Forest Park in STL. It was a firefly breeding ground and you could see huge networks synchronous fireflies flashing and communicating.

___________________________________________________________________________________________

Code:

Arduino Side:

int numFlies = 0;
void setup()
{Serial.begin (9600);
for (int i=1;i<11;i++)
{
pinMode(i,OUTPUT); // PINS0-10 are set as output
}
pinMode(A0,OUTPUT); //PIN A0 set as output
pinMode(A1,OUTPUT); // PIN A1 set as output
pinMode(A2,OUTPUT); // PIN A2 set as output
digitalWrite(A0,HIGH); //pull up the A0 pin
digitalWrite(A1,HIGH); // pull up the A1 pin
digitalWrite(A2,HIGH); // pull up the A2 pin
/* add setup code here, setup code runs once when the processor starts */
}
void loop()
// have p5js communicate with arduino
{ if (Serial.available()){
int fromSerial = Serial.read();
if (fromSerial == 1) numFlies++;
if (fromSerial == 0) numFlies = 0;
}
// 1 firefly caught
digitalWrite(A0,LOW); //layer 1 of cube is grounded
if (numFlies == 1) {
digitalWrite(2,HIGH); //turn ON each LED one after another in layer1
}
else if(numFlies == 0){
digitalWrite(2,LOW);
}
// 2 fireflies caught
if (numFlies == 2) {
digitalWrite(3,HIGH); // turn ON each LED one after another in layer2
delay(200);
}
else if(numFlies == 0){
digitalWrite(3,LOW);
}
// 3 fireflies caught
if (numFlies == 3) {
digitalWrite(4,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(4,LOW);
}
// 4 fireflies caught
if (numFlies == 4) {
digitalWrite(5,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(5,LOW);
}
// 5 fireflies caught
if (numFlies == 5) {
digitalWrite(6,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(6,LOW);
}
// 6 fireflies caught
if (numFlies == 6) {
digitalWrite(7,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(7,LOW);
}
// 7 fireflies caught
if (numFlies == 7) {
digitalWrite(8,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(8,LOW);
}
// 8 fireflies caught
if (numFlies == 8) {
digitalWrite(9,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(9,LOW);
}
// 9 fireflies caught – light up 9th LED and then cycle through layer 2 & 3
if (numFlies == 9) {
digitalWrite(10,HIGH); // turn ON each LED one after another in layer2
delay(200);
digitalWrite(A0,HIGH);
delay(200);
digitalWrite(A1,LOW);
delay (200);
digitalWrite(A1,HIGH);
delay (200);
digitalWrite(A2,LOW);
delay (200);
digitalWrite(A2,HIGH);
}
else if(numFlies == 0){
digitalWrite(10,LOW);
}
}

 

___________________________________________________________________________________________

Code:

P5.js Side:

// Title: Musical Fireflies
// By: Becca Moore
// Class: ICM + PCOMP
// Description: Catch a firefly with the mouse to create a melody. Watch them illuminate //in the jar or listen to your collection
// If you click “Listen to my Melody”, the caught fireflies will illuminate and sing back the //melody you created by catching them.
// To release them back into the sky, click “Release Fireflies”
//
// Additional Hardware needed for experience:
// Headphones, Ipad or laptop, Mason Jar setup( microcontroller (Arduino Uno or Nano) + //LED matrix + 5v wall plug), Cardboard Box Scene (environmental setting)
//
//  – – – – – – – – – – – – – – – – – – – – – – – – – –
// Audio:
// 1) Summer Night Environmental Sounds
// from FreeSound.org
// rgbrobot__night-ambience-01-aug-2-2013-back-yard
//
// 2) Firefly tone
// from p5js’ built in sound library – oscilator + ADSR envelope
// frequencies selected based on c major scale through MTU chart // //http://pages.mtu.edu/~suits/notefreqs.html
//
//
// – – – – – – – – – – – – – – – – – – – – – – – – – – –
// Images:
// 1) Night Sky: Creative Commons / no attribution required image from Pixaby
//
// 2) Firefly: Creative Commons / no attribution required
// Originally found through a free clipart library that says they are
// in accordance with the Digital Millennium Copyright Act of 1998 //(http://www.copyright.gov/legislation/dmca.pdf)
// However, came across the same image on Scholastic’s site? so may dig a little bit more //to see
//
// – – – – – – – – – – – – – – – – – – – – – – – – – –
//
// Thank you for help & advice from:
// Allison Parrish, Jasmine Soltani, Luisa Pereira, Yeseul Song, Danny Rozin, Tom Igoe, //Coding Train/ Dan Shiffman
// & my larger ICM & ITP community for helping me learn so much!
//
/////////////////////////////////////////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////////////////////////////////////////
/////  Declare: global variables, arrays, & preload media) /////////
////////////////////////////////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////////////////////////////
// preload media
function preload() {
  img = loadImage(‘images/nightsky.jpg’);
  img2 = loadImage(‘images/greenglow.png’);
  img3 = loadImage(‘images/milkyway.jpg’);
  img4 = loadImage(‘images/trees.jpg’);
  fireflyImg = loadImage(‘images/fireflyTransparent.png’)
  net1Img = loadImage(‘images/net2.png’)
  summerSounds = loadSound(“summerSounds.mp3”);
}
//establish serial
let serial; // variable to hold an instance of the serialport library
let fromSerial = 0; //variable to hold the data
// declare global variables, arrays, & boolean logic
let img;
let img2;
// let img3;
let img4;
let fireflyImg;
let net1Img;
let summerSounds;
let osc;
let env;
let releaseTime = 0.5;
let buttonEmpty;
let buttonListen;
let fireFlyTone = 261.63
let playing = false;
let noteFrequency = [261.63, 329.63, 392.00, 523.25, 392.00, 329.63, 261.63, 196.00];
let fireflies = [];
let song = [];
///////////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////////
//////////////////          Setup           /////////////////////
////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////
function setup() {
  // createCanvas(1100, 800);
  createCanvas(windowWidth, windowHeight);
  summerSounds.play();
  summerSounds.loop();
  ///////////////////////////////////////////////////////
  ////// initialize Serial Commmunication to Arduino  //
  /////////////////////////////////////////////////////
  // ‘192.168.1.3’ = 11 Maple St for mobile
  serial = new p5.SerialPort(); // make a new instance of  serialport library
  serial.on(‘list’, printList); // callback function for serialport list event
  serial.on(‘data’, serialEvent); // callback for new data coming in
  serial.list(); // list the serial ports
  serial.open(“/dev/cu.usbmodem1421”); // open a port
  /////////////////////////////////////////////////////////////////////////////////
  ////// Firefly instantiation                                                  //
  ////// Draw 9 Fireflies to the screen w Firefly Class & push to fireflies[]  //
  ////// Ref = 7.1 – 7.4 – Coding train on Object Interaction                 //
  /////////////////////////////////////////////////////////////////////////////
  for (let i = 0; i < 9; i++) {
    let x = random(1100);
    let y = random(800);
    let r = (12);
    let f = new Firefly(x, y, r);
    fireflies.push(f);
  }
  ////// Create Firefly Sounds
  // Ref =  17.7 – Coding Train ADSR & Envelopes
  env = new p5.Envelope();
  env.setADSR(0.05, 0.1, 0.5, 1);
  env.setRange(1.2, 0);
  osc = new p5.Oscillator();
  osc.setType(‘sine’);
  osc.freq(fireFlyTone);
  osc.amp(env);
  osc.start();
  ///////////////////////////////////////////////////////////////////////
  /// Create Buttons                                            //
  /// – Release Fireflies                                      //
  ///                                                                       /
  /// – listen to my Melody (next iteration)  /
  //////////////////////////////////////////////////////////////////
  ///////////////////////////////////////////////////////////////////////////////////////////
  // “Release Fireflies” Button                                                 //
  // Essentially a Reset button                                                //
  // – create and use “Release” function                                /
  // – set int numFlies in arduino code back down to 0    /
  // – clear song back to empty []                                          /
  // – reset osc.amp to 0 or logic back to not playing       /
  // – return caught fireflies back to night sky                 /
  //////////////////////////////////////////////////////////////////////////////////////
  buttonEmpty = createButton(‘Release fireflies’);
  buttonEmpty.mouseClicked(release);
  if (buttonEmpty.mouseClicked(release)) {
  }
  function release() {
    fireflies.length = 0;
    song.length = 0;
    serial.write(0);
    for (let i = 0; i < 9; i++) {
      let x = random(windowWidth);
      let y = random(windowHeight);
      let r = (12);
      let f = new Firefly(x, y, r);
      fireflies.push(f);
    }
    console.log(“Fireflies released back into sky*~*”);
  }
  //////////////////////////////////////////////////////////////////////////////////////////
  //                                                                                               //
  // Listen to your collected Fireflies / melody                  //
  // or should text say “listen to my collection?”             //
  //- create fuction to play through notes collected in  //
  //  song[]                                                                              //
  /////////////////////////////////////////////////////////////////////////////////////
  // buttonListen = createButton(‘Listen to my Melody’);
  // buttonListen.position(150);
  // buttonListen.mousePressed(playSong);
  // function playSong() {
  //   for (let i = 0; i < song.length; i++) {
  //     osc.amp(0.5, 0.05);
  //      osc.freq(songArray);
  //     // console.log(osc.freq);
  //     // osc.stop(10.0);}
  //     console.log(“song playing”);
  //     console.log(osc.freq);
  //   }
  // }
}
//////////////////////////////////////////////////////////////////
/////////////////////////////////////////////////////////////////
///////////////////            Draw             //////////////
///////////////////////////////////////////////////////////////
//////////////////////////////////////////////////////////////
function draw() {
  background(255);
  // draw and scale night sky
  push();
  scale();
  image(img3, 0, 0, windowWidth, windowHeight);
  pop();
  //   push();
  //   imageMode(CORNERS);
  //   scale(0.5);
  //   image(net1Img, 500,500);
  //   pop();
  /////////////////////////////////////////
  // Firefly rollover blink //
  ////////////////////////////////////////
  for (let i = 0; i < fireflies.length; i++) {
    if (fireflies[i].rollover(mouseX, mouseY)) {
      fireflies[i].fireFlyBlink();
    } else {
      fireflies[i].noFireFlyBlink();
    }
    fireflies[i].move();
    fireflies[i].show();
  }
  //////////////////////////////////
  // main screen text //
  /////////////////////////////////
  fill(255);
  textSize(30);
  textAlign(CENTER);
  text(‘catch a firefly to create a melody’, windowWidth / 2, windowHeight / 2);
}
//////////////////////////////////////////////////////////////////////////////////////////////
// Sending Clicked Firefly from sky to Jar                           //
// – check to see if mouse is within diameter of firefly    //
// – if it is and is clicked                                                         //
//////////////////////////////////////////////////////////////////////////////////////////
function mousePressed() {
  for (let i = fireflies.length – 1; i >= 0; i–) {
    if (fireflies[i].rollover(mouseX, mouseY)) {
      fireflies[i].clicked(mouseX, mouseY);
      fireflies.splice(i, 1);
      console.log(“# of Fireflies left in sky”);
      console.log(fireflies.length);
      console.log(“Song”);
      console.log(song);
      serial.write(1);
    }
  }
}
/////////////////////////////////////////////////////////
//   Serial communication               //
//   print port list                              //
//   create a serialEvent function //
//////////////////////////////////////////////////////
function printList(portList) {
  for (var i = 0; i < portList.length; i++) {
    // Display the list the console:
    console.log(i + ” ” + portList[i]);
  }
}
function serialEvent() {
  // this is called when data is recieved
}
//////////////////////////////////////////////////////
/////////////////////////////////////////////////////
//////////////                            /////////////
/////////////  Firefly Class   /////////////
////////////                           /////////////
////////////////////////////////////////////////
///////////////////////////////////////////////
class Firefly {
  constructor(x, y, r) {
    this.x = x;
    this.y = y;
    this.r = r;
    this.a = 12
    this.n = random(noteFrequency);
  }
  //firefly behavior when clicked
  //plays oscilator tone
  clicked(px, py) {
    let d = dist(px, py, this.x, this.y);
    if (d < this.r) {
      console.log(“Firefly Caught in Jar! Note Frequency:”);
      if (!playing) {
        osc.freq(this.n);
        console.log(this.n);
        song.push(this.n);
        env.play();
      }
    }
  }
  // Firefly wiggle animation
  move() {
    this.x = this.x + random(-2, 2);
    this.y = this.y + random(-2, 2);
  }
  // Rollover firefly glow = fireFlyBlink() or noFireFlyBlink()
  rollover(px, py) {
    let d = dist(px, py, this.x, this.y);
    let val = d < this.r;
    return val;
  }
  fireFlyBlink() {
    this.a = 80;
  }
  noFireFlyBlink() {
    this.a = 12;
  }
  show() {
    push();
    noStroke();
    fill(255, 255, 0, 150);
    ellipse(this.x, this.y, this.a * 2);
    pop();
    noStroke();
    push();
    imageMode(CENTER);
    scale(0.5);
    image(fireflyImg, this.x * 2, this.y * 2);
    pop();
  }
}

 

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s