(Top video = playtest with revised p5js sketch / Bottom Video = Testing the LED Matrix + P5js)
Musical Fireflies
Full Screen link // P5js Editor link
Description: Catch the fireflies on the screen to create a melody. As you catch them on the screen, they are caught from the “sky” and appear in real time in a mason jar (as illuminated LEDs). Originally envisioned as a chord building exercise and evolved into a Chance Aesthetics style Melody creation activity.
Once you’ve collected them, you can release them back into the sky by clicking on “release fireflies” button. When clicked, all the caught “fireflies” in the mason jar are released back into they sky (turning off the LEDs/resetting the LED matrix).
Still working on a 2nd button called “Listen to my melody,” that will play through the collected array of notes/fireflies that exist in your jar. Will continue working on it this next week : )
I think for my oscillator to play through my array song [], I will look into setInterval(), setTimeOut(), JavaScript Closure. Could also take another look at the p5js Song Example .
1) Sketch Iteration
1.1 – To build my sketch I rewatched & reworked through many Coding Train tutorials, starting back with Coding Train 3.2./ Bouncing Ball.
1.2 – Then started to think about how to create arrays for my circles referencing Coding Train 7.3 Array of Objects / OOP & making a class of fireflies. This dove back into the playlist on Arrays & Objects. Thinking about how to have the flexibility to have 5 or 500 fireflies with the change of one variable.
1.3 – To have the fireflies eventually move in to digital (or physical) jar, needed to understand / have the firefly behavior change when clicked. Coding Train’s 7.4 Mouse Interaction With Objects covers just that. When the firefly is clicked, the sketch checks to see if the mouse x,y coordinates overlap with the diameter of the circle. If they do then can have “Clicked on Firefly!” print to console.
1.4 – Office Hours [Jasmine] + Coding Train 7.1 – 7.5
I decided to sign up for office hours with Jasmine Soltani to help me process the needed building blocks for my sketch. I brought in a version of the sketch seen below in 1.5. I became stuck on thinking about the best way to assign a musical note to each firefly, but also how to do a rollover blink accent as well as splicing and copying to a new array. She walked me through a sketch she made her first semester that covered various strategies including Coding Train 7.4 – Removing Objects from an Array. To select my frequencies in my I used the Musical note frequencies from a chart through Michigan Tech University here
1.5 – Office Hours [Jasmine & Allison] + PCOMP usertesting
Then I scheduled office hours with our ICM teacher Allison Parrish to also think about possible array and arrays of objects. Later that week in Danny Rozin’s PCOMP I received some valuable feedback during our initial play test, that if I was to have an actual physical mason jar next to the sketch, that maybe the sketch should be more atmospheric verses displaying another jar. That idea made a lot of sense to me : )
In preparation of our ICM user testing, I found a placeholder image of a summer night sky and a creative commons ambient recording through freesound.org. (**jar image above was also a clipart placeholder image. If had gone that design route would have drawn one in a similar style ❤ )
1.6 – ICM Usertesting, Office Hours [Yeseul]
Our ICM usertesting class was extremely helpful. Classmates offered lots of great insight, advice, and ideas for future iterations:
- Button aesthetic
- confusing to user / unclear when or in what order to click. one playtester had an association with a pop-up add button, and thought maybe she needed to click out of the button first in order to interact with the rest of the screen.
- Add real firefly image to help establish that it is a firefly
- createCanvas(windowWidth & windowHeight);
- to help make the image fullscreen / the user feel more immersed in environment
- enjoyed the yellow blink rollover & ambient sounds
- oscillator doesn’t have an envelope attached
- so when 2 fireflies have middle C as their tone consecutively, it feels like an error due to it being one continuous tone.
- Restrict “Release Fireflies” button to only redraw max original amount ex:9
- was redrawing an infinite amount vs returning a caught amount back to the screen
I also met with resident Yeseul to talk about LED/lighting solutions for the PComp aspect of the project. She encouraged me to take a look at DIY LED matrix tutorials ❤
1.7 – Revisions, Office Hours [Yeseul, Danny, Luisa], Final class
In response to ICM usertesting, I found a creative common’s image of a firefly and inserted it over the Firefly Object dots [following coding train 7.8 Objects & Images]. I also worked further into splicing and pushing to a new array. When a Firefly is clicked it is assigned a random tone frequency from my array noteFrequency[]. It is spliced from the array fireflies[] and instead has the noteFrequency then pushed to a new array song[].
After reworking through some of Danny Rozin’s Serial communication examples, I got my 3x3x3 LED cube communicating serially with p5js to arduino. p5js sends a ‘1’ to the arduino to communicate which LEDs to light up. If the button “release fireflies” is selected, p5js will serialWrite(0); to the arudino side, resetting the LED matrix to all off.
Luisa and I talked through some ideas and approaches to sound for the next couple iterations. She also recommended a MUX shield to me. She also talked about how instead of using an oscillator, I could possibly play with repitching the same sound file down the line. She also reiterated that maybe Melody make more sense than chord building, even though originally I changed it due to struggling to make it polyphonic via code. That the experience of capturing a firefly really is usually one at a time.
She also mentioned for the animation style, that since the background is realistic? Maybe it would help if the firefly image reflected that as well. Or the flipside, having the background becoming more illustrative than photorealistic. Either way to try to match the visual language maybe? Will have to test it out ❤
1.8 – Post Finals Revisions + Office Hours [Luisa]
Was able to find a creative commons background image and apply it with windowWidth & windowHeight, allowing for a more responsive screen. Was able to get it running on the p5js side on an ipad by having both devices run on the same ip address (via Tom & Jeff’s External Screen tutorial ), but will still need the hardware components to make it interact with Arduino.
Was able to fix my Oscillator issue! I kept overwriting where I tried to connect it to an envelope in my firefly class. Now that the two are connected, even if fireflies have the same musical note 3 times in a row, you can still hear each one as an individual. Luisa also helped walk me through how setInterval could work through an office hours example here. Excited to fold the concept into to my current sketch.
2) User Testing / Folding in Feedback for Future
A consistent and great piece of feedback I’ve received is to really think about firefly behavior, and how that might translate into light. I went more in depth on a post here . When researching more into firefly light behavior, I came across a really wonderful post called “Learn to Speak like a Firefly” by Science Friday’s Education Director Ariel Zych. She referenced a really great study called “Studies on the Flash Communication System in Photinus Fireflies” by James E. Llyod [Museum of Zoology/University of Michigan]. In it is the picture above, where they’ve distinguished different blinking patterns associated with different species.
The ones I’m most used to seeing is #8 on their diagram above, of the Big Dipper Firefly/Common Eastern Firefly[Photinus Pyralis]. It would be fun to try and incorporate that dimming, swooping ‘J’ pattern that’s associated with it. At least on the p5js side? Would also be fun to play with PWM / brightness levels to create more of firefly effect.
I also think that Daniel Shiffman’s Coding Train: Particle Systems could be a step in the right direction to create a more firefly like animation behavior?
3) Serial Communication + LED Matrix/Multiplexing
- when deciding which 3x3x3 tutorial to go with after trying a few, I decided to go with DIY 3X3X3 LED Cube with Arduino by Dilip Raja. I ended up stripping the Arduino code down to where for now it lights up 1 LED at a time. When all 9 are clicked, it enters a victory pattern where all nine are lit up in each layer, 1 layer at a time. It creates the feeling of synchronized fireflies
4) Inspiration & Resources
Yeseul Song & Jeff Park – Liquid light (2016) https://vimeo.com/197146445
James Clar / 3D Display Cube (2001-2009) http://www.jamesclar.com/portfolio_page/3d-display-cube-2002-2009/
A lot of my references and inspiration came as I was building. After I built the 3x3x3 Matrix (trying 3 different variations, going with Dilip Raja’s tutorial), I started to find a lot of reference videos for similar Firefly Arduino projects. The idea was inspired by summers in TN & MO and also specific memories of a bike path in Forest Park in STL. It was a firefly breeding ground and you could see huge networks synchronous fireflies flashing and communicating.
-
Chance Aesthetics exhibition at the Kemper Art Museum / I saw as a student in Saint Louis. The show really inspired me and my friends to think about randomness / Games & Systems of Random Ordering & Chance Operations
-
MITmedialab – The Musical Fireflies – Learning About Mathematical Patterns in Music Through Expression and Play /
-
“Studies on the Flash Communication System in Photinus Fireflies” by James E. Llyod [Museum of Zoology/University of Michigan] (especially the drawing!)
-
“Learn to Speak like a Firefly” blog post by Science Friday’s Education Director Ariel Zych.
-
1-7: Foundations of Programming with p5js Coding train / Daniel Shiffman – [especially 7.1-7.8 sections on Objects & Arrays]
-
DIY 3X3X3 LED Cube with Arduino by Dilip Raja // https://circuitdigest.com/microcontroller-projects/making-3X3X3-led-cube-with-arduino
-
Jar of Fireflies by KESO https://www.instructables.com/id/Jar-of-Fireflies/#intro
___________________________________________________________________________________________
Code:
Arduino Side:
int numFlies = 0;
void setup()
{Serial.begin (9600);
for (int i=1;i<11;i++)
{
pinMode(i,OUTPUT); // PINS0-10 are set as output
}
pinMode(A0,OUTPUT); //PIN A0 set as output
pinMode(A1,OUTPUT); // PIN A1 set as output
pinMode(A2,OUTPUT); // PIN A2 set as output
digitalWrite(A0,HIGH); //pull up the A0 pin
digitalWrite(A1,HIGH); // pull up the A1 pin
digitalWrite(A2,HIGH); // pull up the A2 pin
/* add setup code here, setup code runs once when the processor starts */
}
void loop()
// have p5js communicate with arduino
{ if (Serial.available()){
int fromSerial = Serial.read();
if (fromSerial == 1) numFlies++;
if (fromSerial == 0) numFlies = 0;
}
// 1 firefly caught
digitalWrite(A0,LOW); //layer 1 of cube is grounded
if (numFlies == 1) {
digitalWrite(2,HIGH); //turn ON each LED one after another in layer1
}
else if(numFlies == 0){
digitalWrite(2,LOW);
}
// 2 fireflies caught
if (numFlies == 2) {
digitalWrite(3,HIGH); // turn ON each LED one after another in layer2
delay(200);
}
else if(numFlies == 0){
digitalWrite(3,LOW);
}
// 3 fireflies caught
if (numFlies == 3) {
digitalWrite(4,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(4,LOW);
}
// 4 fireflies caught
if (numFlies == 4) {
digitalWrite(5,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(5,LOW);
}
// 5 fireflies caught
if (numFlies == 5) {
digitalWrite(6,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(6,LOW);
}
// 6 fireflies caught
if (numFlies == 6) {
digitalWrite(7,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(7,LOW);
}
// 7 fireflies caught
if (numFlies == 7) {
digitalWrite(8,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(8,LOW);
}
// 8 fireflies caught
if (numFlies == 8) {
digitalWrite(9,HIGH); // turn ON each LED one after another in layer2
}
else if(numFlies == 0){
digitalWrite(9,LOW);
}
// 9 fireflies caught – light up 9th LED and then cycle through layer 2 & 3
if (numFlies == 9) {
digitalWrite(10,HIGH); // turn ON each LED one after another in layer2
delay(200);
digitalWrite(A0,HIGH);
delay(200);
digitalWrite(A1,LOW);
delay (200);
digitalWrite(A1,HIGH);
delay (200);
digitalWrite(A2,LOW);
delay (200);
digitalWrite(A2,HIGH);
}
else if(numFlies == 0){
digitalWrite(10,LOW);
}
}
___________________________________________________________________________________________
Code:
P5.js Side:
30 KM/s 