Finished writing, now back to the code

23 Apr 2012

Meemoo started at a prototype in the January 2011 Interactive Cinema class at Media Lab Helsinki. It looked quite different back then, and had one function: cutting around triggers of multiple web video players.

As this project progressed, I wanted the videos to be in a different browser widow than the controllers. I made a text format for control signals to be sent between windows with window.postMessage(). This browser function could only be used to send strings, but around the middle of the year (Gecko 6.0) the function expanded to allow all kinds of data to be passed from window to window. At this point the web video remixer idea became the web remixer remixer (the metamedium metaremixer).

I realized that this idea could encompass most of my web/interactive experiments, so I made it my thesis project. In the written thesis I looked at design for hackability, and how that could be positive for education.

Other services for learning code (like Code Academy) start with "this is a var." You complete the lessons, get badges, and maybe get a fundamental concept of programming, but what can you really do with it? Meemoo starts with "this is a stop motion application." Use it as is, or look under the hood and hack it if you like. I think that this design for hackability feeds curiosity and encourages learning in a more natural way, much closer to how I and many other actually learned coding.

At every step in the process, people can instantly share their apps and creations for the world to see and modify. Or not, and keep it all local.

Lev Vygotsky said that learning happens in the space between what can do by ourselves and what we can do with guidance from someone with more skill. He called this the "zone of proximal development." The Internet is the ultimate manifestation of this metaphorical space. Most web coders learned with "view source," and this project seeks to extend this concept to creative web apps.

Feel free to read the full version of the thesis:
(1.6MB PDF)

Since submitting the paper I have been happy to get back into the code. The biggest change to the UX has been with the wires, adding draggable plug ends. You can see this in action on the Processing example, dragging a wire from one reflow module to the other.

Meemoo has also accepted it's first code from another developer! This is a big moment for any open source project. Automata added the foundations of a history stack, so if you delete a module you can press ctrl-z to undo the deletion.

Live animation with Meemoo as VJ software

12 Feb 2012

I was invited to do visuals for a Zodiak's Side-Step dance festival club night. I used the gig as an opportunity to push Meemoo development and pressure-test the live-animation features.

Meemoo Live Animation at Zodiak (on Youtube)

Many thanks to Aino Riiho (and other partygoers) for the clay sculpting, Juho Santasalo for the videography, and Heikki Sillanpää (dkstr) for the music.

This is what you get for hiring a web developer to VJ your party (o_-)

As the party started and I was still coding furiously, adding features to the world module. Thirty minutes later the music tempo picked up, inviting people to the dance floor, and I made myself declare the coding done for the night. It was a thrill to see the first sprite hit the dance floor: multicolored glitter swirling in water.

We used clay and construction paper (and some glitter) as the basic building blocks of the visuals. I'm attracted to the textures and imperfections that come from using materials like these. Using the taptempo module, I synced the sprites' animation to the rhythm. It was fun to build these tiny animations and then throw them onto the screens around the dance floor.

For the gig I made some special modules for creating a "world" into which I could insert animated sprites. On the software development side, I'm happy that I decided to make two modules (Controller and World) share the same Backbone model. Each module has its own view of the same model, so the data passed through the wire will be the same on both sides.

There are lots of improvements and ideas that came up in the evening:

• Camera: I used a Sony Eyetoy webcam, and the color was pretty bad. I chose kid's art supplies with rich colors, but most of the color was washed out in the first step. Next time I'll do some tests to find a better webcam, or use the camera on my phone, or a real digital camera somehow.
• Audience participation: I planned to use Kinect to get silhouettes of people dancing into the world, but ran out of time. I was imagining using different animated textures for specified depth ranges.
• Flocking: I only had time to implement the tiled animation. The original concept was that sprites could be individual or flocks that would move around the screens.
• UX tweaks: Confirm dialog on every delete got annoying when juggling around modules. Directly un/replugging wires is a suggestion that is now high on the to-do list.
• I made a hack to open the World module in a new window to view it fullscreen on the projectors. I plan on making this a built-in feature for any module.

Despite these limitations, I got a lot of good feedback about the visuals. People were interested in what I was doing, and came around to play with the art supplies. Doing visuals powered by a web browser (Firefox Aurora, by the way) was a fun experiment, and with a few more display options I think that the limitations would have been less aesthetically obvious. Performing under pressure was a good way to test the system.

Only once in the evening did a JavaScript warning pop up on the dance floor. I consider that a victory, and it made me laugh a lot when it happened.

Friction-Free Post-Scarcity Creative Economies

24 Jan 2012

Lately I have been imagining a post-scarcity friction-free creative economy.

Music, film, and publishing industries are thrashing around trying to readjust to their post-scarcity reality. Object design will be next to be "post-scarce" with the rise of 3D printers. How can we design a creative economy that better matches reality? Couldn't we design a system that is more like the internet, and less about censorship and control?

Link = credit = deferred tip.

If financial transactions were friction-free (no fees or percentages, like Dwolla transactions <$10), then a creative work could specify percentages of tips to other parties. If you tip an band, then they could split that tip among label and members as they see fit. Those credit links would be the opposite of paid links (rel="nofollow"), they would be paying credits (rel=credit(10%) ?).

Interactive Model of Deferred Tipping (powered by Meemoo)

If you could tip a filmmaker as easily as a Facebook "like" or Flattr, and they could defer part of that tip to the musician from whom they used a CC-licensed track, wouldn't that be a good way to provide value to a Creative Commons economic system?

Now, if you Flattr a blog post that embeds a video, the blog owner gets the full value of the Flattr. If they could defer 99% of the Flattr to the film, then the filmmaker would be encouraged to CC-license their stuff so that it could be shown by any venue interested. The venue could be a bike-powered back-alley film festival and it could work the same way.

What if the blog owner is greedy and only defers 5% of the tip to the movie? The amount of the credit should be clearly displayed with the tip widget, along with the breakdown of how your tip will be distributed. The tipper could then decide to just click through to the movie page to tip it directly.

Of course, it should be distributed, not tied to just Flattr or Dwolla. Maybe "friction-free creative economy" certification and logo for services are in order.

A friction-free app store.

I was imagining a Meemoo App Store, but thought that people might be adverse to others profiting off of their open-source modules. If this were to slow the amount of module building and sharing, it would undermine the point of the project.

A deferred-tip system which defers credit from an app's tips to the modules it uses would be an interesting experiment. You could also provide credit when forking a module. Would this encourage open-source collaboration or get in the way?

Like this idea?

Mozilla WebFWD summit

20 Dec 2011

Mozilla WebFWD's first summit just came to a close. It was great to see the offices and meet a cross-section of the people that make Mozilla what it is.

I made a simpler webcam to animated gif example to pitch the concept.

Here is the presentation that I gave on Friday, with some added notes:

New Example

11 Dec 2011

I'm making steady progress with the interface. In order to understand the tool more I'll be making more example modules and compositions. This one goes from a Flash webcam through some image-bashing transformations, JPG and canvas.

There are some design challenges that are becoming apparent through this process. It is fun to imagine the potential of what this will become.

Dragging wires

14 Nov 2011

This step takes the UI much closer to being ready: draggable wires. Now you drag a wire (from processing to glitch, for example).

Try it.

I also added a "source" button. Making source viewable and hackable is a major design goal of this project.

WebFWD and thesis progress

10 Nov 2011

Meemoo is now supported by Mozilla's new WebFWD Innovation Accelerator! "It's about creating open solutions that push the Web forward, together." That sounded good to me. So far they have hooked my project up with some nice tech offers, and we have weekly learning sessions and calls with all of the involved projects. In December they are bringing us together for a face-to-face summit at Mozilla's HQ in California.

Last week I made a "first paper" presentation of Meemoo for my thesis class. The instructor and my peers seemed to get the idea, and it looks like Meemoo will become a part of Media Lab Helsinki courses and research.

Work on the graphing interface continues. I just made some changes that make dragging and resizing modules smoother. This work is in preparation for the next hurdle, the UX of connecting and disconnecting wires. Once that is done, import/export of compositions and saving to local storage (easy) will be all that is left before I'll start creating more demos to test/show what the system can do.

Zoomed out

Last night a friend suggested that I implement zooming in the interface. I pinched my trackpad, and the interface zoomed out. Done! Building software in the web browser is cool like that.

multithreaded javascript?

28 Sep 2011

When first designing the video remix version of Meemoo, I put the video players in an iframe page, separate from the main application. This made it possible to "pop-out" the players to be shown on a projector. The communication between the app and the players happens with window.postMessage(). This is how the new framework works as well: iframes communicating with postMessage.

blocking module

At first I thought that this would have the added benefit of running the JavaScript of each module in a separate thread, but it turns out that it only works like this in Opera. I made a test by adding a module with a blocking script. In all browsers except Opera, pressing the button stops the processing and glitch modules. Unfortunately, Opera's JavaScript performance is noticably chunkier than the primary browsers that I'm testing: Chrome, Firefox, and Safari.

I wonder if window.open() windows run in separate threads. This is worth a test.

using browser developer tools profiles

25 Sep 2011

Meemoo's architecture lets you easily see how much processing time modules are using. Using Chome's Developer Tools' Profiles, you can get a view like this, which breaks down which functions in the composition are using the most resources:

running the current development example at 60fps

moving image data

24 Sep 2011

It is easy to move image data from module to module for manipulation:

image glitch module demo

live (try resizing the glitch window), glitch source

multiple ins and outs

22 Sep 2011

The development example now shows multiple ins and outs:

multiple ins and outs

Todo, to have the basic framework ready:

• add module by url
• drag from port to port to create edge
• disconnect edges
• widgets on input ports for adjusting values directly
• export/import graph JSON and lock down the v1 schema
• save to local storage

a processing.js module

17 Sep 2011

I made a simple Processing.js module to add some visuals to the demo. It maps the /beat signal from the metronome to the mousePressed() function in Processing. It works on my 'droid phone in Mobile Firefox.

module source, processing source

http://meemoo.github.com/iframework/

I'm working on named inputs now.

Proved

12 Sep 2011

I have been making proof-of-concepts and alpha versions of this idea since January, so I'm ready to say that I have graduated to working on the alpha version now. I have been working on the UI, which isn't necessarily the best thing to be polishing at the beginning, but I feel like I need to be able to interact with it on a basic level before understanding which features to implement. Creating a web app is more like sculpting clay than marble... you can always add and take away more if needed.

http://meemoo.github.com/iframework/... Don't forget to view source!

Modular JavaScript multimedia framework

02 Aug 2011

I like the immediacy of wiring-metaphor multimedia frameworks like Quartz Composer and Pure Data. On the other hand, logic controls that would be simple to code can get quite complicated to wire, and messy tangles of wires can quickly cover the screen.

I have decided to abstract out the components of Meemoo (metronome, pattern sequencer, video players) into reusable JavaScript instruments that can communicate with each other using OSC-like messages. I have made a proof-of-concept (server 1, server 2) that shows how this can work with very little latency:

iframes communicating with OSC-like strings through window.postMessage()

To go these 10 hops, Firefox 5.0.1 takes ~42ms, Chrome 12 takes ~12ms, and Safari 5.1 takes ~4ms. I'm not sure why my test is showing Safari as the winner of window.postMessage() speed, but they all seem fast enough.

The JS of each frame runs in a separate thread, so if something slows down in one (fancy prosessing.js or WebGL visuals?), it shouldn't affect the others (master metronome). A simple web application will save the state of all of the frames, and the graph data of which iframe is sending messages where. Forking compositions and modules will be simple and encouraged by the design of the project. Stay tuned...

(Ab)using html <video>

08 May 2011

When I set out to write a web application for remixing web video, I did some tests and figured out that I could write it all in html+js, and it has been a great learning experience and mostly smooth. However, there are always going to be differences in browsers. While I haven't needed to write any browser workarounds (thanks to JQuery and ignoring older browsers), the <video> element works differently in the 3 browsers that I'm testing.

The main functionality of Meemoo is setting up and skipping to time-based triggers in video, which is as easy as video.currentTime = 5. Because of the different ways that browsers handle video caching, performance can be quite different.

Chrome can skip to triggers around the current playhead with great response time, but skipping backwards more than a few seconds causes a pause. Safari skips forward fine, but skipping backwards causes the video element to report itself not buffered after that time, which blocks skipping forward for a moment. Firefox with WebM video seems to be the best combination for now, with very responsive seeks all over the length of the video.

The YouTube player is Flash, and skipping feels a little less responsive than with the HTML video players, but is fast enough. Triggers close to the end of the video sometimes cause the video to rebuffer, which isn't good.

I also had a hair-pulling mystery bug that caused ghost players to be playing in the background, found a workaround.

Introducing Meemoo

03 May 2011

Meemoo is a web application that allows experimenting with mixing web video with tools familiar to people who create digital music: patterns and pattern sequencers. Please read about the idea and how to try it yourself at meemoo.org.

I'm currently looking for support to create a community site around sharing mixes and promoting remixable video.

Please get in touch with bugs, suggestions, and exported compositions.