JavaScript
Browser Bits

@BenjaminBenBen

I'm Ben

Freelance developer

based in Oxford

(available for projects!)

Data, Visualisation, Realtime, Performance, VR, Interaction, Stuff

Binary data in JavaScript

Accessing raw image & audio data

Mashing data together (demos)

Binary Numbers

🙌

10 Digits

Base 10

Decimal

🤖

2 Digits

Base 2

Binary

Decimal 42

Binary 0b101010

Octal 0o52

Hexidecimal 0x2a

How to JavaScript

Outputting


              (42).toString(10) == '42'
              (42).toString(2)  == '101010'
              (42).toString(8)  == '52'
              (42).toString(16) == '2a'

Parsing


              42 == parseInt('42',     10)
              42 == parseInt('101010', 2)
              42 == parseInt('52',     8)
              42 == parseInt('2a',     16)

Integer Literals


              42 == 42
              42 == 0b101010
              42 == 0o52
              42 == 0x2a

OR, AND, NOT, SHIFT

& | ~ << >>

& (AND)


                 0b0000000011111111
              &  0b0000111111110000

              == 0b0000000011110000

| (OR)


                 0b0000000011111111
              |  0b0000111111110000

              == 0b0000111111111111

~ (NOT)


              ~  0b0000000011111111

              == 0b1111111100000000

              == -256

<<, >> (Bit shifting)


                 0b0000000011111111

              >> 4

              == 0b0000000000001111

Use cases


              const FLAG_1 = 0b0001
              const FLAG_2 = 0b0010
              const FLAG_3 = 0b0100
              const FLAG_4 = 0b1000


              const v = FLAG_1 | FLAG_4


              if(v & FLAG_1) …
              if(v & FLAG_1 && v & FLAG_3) …

Tricks


              const floor = 3.145|0


              if(~str.indexOf(word)) {
                // doesn't contain word
              }

Implementing functions

hex_to_rgb('#abcdef') == 'rgb(11, 205, 239)'


              // '#abcdef'

              const from_hex =
                s => parseInt(s.slice(1), 16)

              const to_rgb =
                c => `rgb(${c>>16}, ${c>>8&255}, ${c&255})`

              const hex_to_rgb =
                h => to_rgb(from_hex(h))

*

* it's not always quite that simple

/

Bits

Numbers

Binary operators

Handling bigger things

00010101010101010101010101010101010100001010101010 10101010101010101010101000010101010101010101010101 01010101010000101010101010101010101010101010101000 01010101010101010101010101010101010000101010101010 10101010101010101010101000010101010101010101010101 01010101010000101010101010101010101010101010101000 01010101010101010101010102010101010000101010101010 10101010101010101010101000010101010101010101010101 01010101010000101010101010101010101010101010101000 01010101010101010101010101010101010000101010101010 01010101010101010101010101010101010000101010101010 10101010101010101010101000010101010101010101010101 01010101010000101010101010101010101010101010101000 01010101010101010101010101010101010000101010101010

Split it up into numbers

Int8Array, Uint8Array, Uint8ClampedArray, Int16Array, Uint16Array, Int32Array, Uint32Array, Float32Array, Float64Array

ArrayBuffers

  • u8 = new Uint8Array(48)
  • buffer = u8.buffer
  • u8_2 = new Uint8Array(u8.buffer, 6, 6)
  • u32 = new Uint32Array(u8.buffer)
  • dv = new DataView(buffer)

Great for

Memory usage

Central state

Performance

/

Typed Arrays

Array Buffers

DataViews

Accessing raw data

Images

<img src="joanie.jpg" />

joanie.jpg → Raw pixel values


              const {width, height} = image_element

              const canvas = document.createElement('canvas')
              canvas.width = width
              canvas.height = height

              const ctx = canvas.getContext('2d')

              ctx.drawImage(image_element, 0, 0, width, height)

              const imageData = ctx.getImageData(0, 0, width, height)
            


              {
                width: 500,
                height: 500,
                data: [ r, g, b, a, … ] // 1M
              }
            


              ctx.putImageData(imageData, 0, 0)
            

                // setting components

                const {data} = imageData

                for(let i = 0; i < data.length; i += 4) {
                  data[i] = 0
                }

                ctx.putImageData(imageData, 0, 0)
              

                // switching components

                const {data} = imageData

                for(let i = 0; i < data.length; i += 4) {
                  data[i] =  255-data[i+2]
                }

                ctx.putImageData(imageData, 0, 0)
              

                // Using other functions

                const {data} = imageData

                for(let i = 0; i < data.length; i += 4) {
                  data[i] = Math.sin(i/800000) * 255
                }

                ctx.putImageData(imageData, 0, 0)
              

                // Mashing stuff together

                const {data} = imageData

                for(let i = 0; i < data.length; i += 4) {
                  data[i] = Math.cos(i/800000) * 255
                  data[i+1] *= 1.8
                  data[i+1] += Math.sin(i/10) * 200
                }

                ctx.putImageData(imageData, 0, 0)
              

                // Messing stuff up

                const {data} = imageData

                for(let i = 0; i < data.length; i += 4) {
                  data[i] = data[i+2] *= data[i] ^ data[i+1]
                }

                ctx.putImageData(imageData, 0, 0)
              

                // Aligning to pixels

                const data = new Uint32Array(
                  imageData.data.buffer
                )

                // data.fill(0xffcc00ff)

                // data.sort((a, b) => (a & 0xff) - (b & 0xff) )

                ctx.putImageData(imageData, 0, 0)
              

                const data = new Uint32Array(
                  imageData.data.buffer
                )

                const r = 0x000000ff
                const g = 0x0000ff00
                const b = 0x00ff0000
                const a = 0xff000000

                for(let i = 0; i < data.length; i++) {
                  data[i] = data[i] & r | a
                }

                ctx.putImageData(imageData, 0, 0)

/

Images

Canvas2D, ImageData

Audio

<audio src="bach.ogg" controls />

bach.ogg → Raw audio data

Web Audio


              const ctx = new AudioContext()

              const source = ctx.createMediaElementSource(audio_element)
              const analyser = audioCtx.createAnalyser()

              source.connect(analyser)
              analyser.connect(ctx.destination)
            

              const timeData = new Uint8Array(analyser.fftSize)
              analyser.getByteTimeDomainData(timeData)

              var freqData = new Uint8Array(analyser.frequencyBinCount)
              analyser.getByteFrequencyData(freqData)

Analyser

Time Domain Data

[]


              const ctx = canvas.getContext('2d')
              ctx.scale(canvas.width / timeData.length, canvas.height / 255)

              const draw = () => {
                requestAnimationFrame(draw)

                analyser.getByteTimeDomainData(timeData)


                ctx.clearRect(0,0,timeData.length,255)

                ctx.beginPath()
                timeData.forEach((value, i) => {
                  ctx.lineTo(i, value)
                })
                ctx.stroke()
              }
              draw()

Analyser

Time Domain Data


              analyser.getByteFrequencyData(freqData)

Analyser

Frequency Data

[]

Analyser

Frequency Data

Generating Audio

Output

BufferSourceNode

Input + Output

ScriptProcessor


            const ctx = new AudioContext()

            const audio_buffer = ctx.createBuffer(
              1, ctx.sampleRate, ctx.sampleRate
            )

            const channel_data = audio_buffer.getChannelData(0) // Float32Array!
            

            for(let t in channel_data) {
              channel_data[t] = -1…1
            }

            const source = ctx.createBufferSource()
            source.buffer = audio_buffer
            source.connect(ctx.destination)
            source.start()
            


              for(let t in channel_data)
                channel_data[t] = (Math.random() - .5) * .2
            


              for(let t in channel_data)
                channel_data[t] = Math.sin(t / 30)
            


              for(let t in channel_data)
                channel_data[t] =
                  Math.sin(t/30) *
                  Math.sin(t/channel_data.length * Math.PI)
            


              let f = Math.random() * 80
              for(let t in channel_data)
                channel_data[t] =
                  Math.sin(t / f) *
                  Math.sin(t / channel_data.length * Math.PI)
            


              let f = Math.random() * 80
              for(let t in channel_data)
                channel_data[t] =
                  Math.sin(t / f) *
                  Math.sin(t / f * 2) *
                  Math.sin(t / f * 4) *
                  Math.sin(t / channel_data.length * Math.PI)

/

WebAudio

Audio Graph, AnalyserNode, BufferSourceNode, ScriptProcessor

Mixing stuff together

Frequency → Image?

Do they fit?


                const image = ctx.createImageData(frequencies.length / 4, canvas.height)

                let idx = 0
                const render = () => {
                  requestAnimationFrame(render)
                  analyser.getByteFrequencyData(frequencies)

                  image.data.set(frequencies, frequencies.length * idx)

                  ctx.putImageData(image, 0, 0)

                  idx = (idx + 1) % canvas.height
                }
                requestAnimationFrame(render)

spectrograph

No processing!

Though maybe we could do with some

1/ Data Density

image.data

[r, g, b, a, r, g, b, a, r, g, b, a, …]
[f, f, f, f, f, f, f, f, f, f, f, f, …]

new Uint32Array(image.data.buffer)

[rgba, rgba, rgba, rgba, …]
[f,    f,    f,    f,    …]

2/ Colour utilisation


              const to_color = (v) => {
                let r = -Math.sin(v * PI74)
                let g =  Math.sin(v * PI74)
                let b = -Math.cos(v * PI74)

                if(r < 0) r = 0
                if(g < 0) g = 0
                if(b < 0) b = 0

                r = r * 255 & 255
                g = g * 255 & 255
                b = b * 255 & 255

                return r  | (g << 8) | (b << 16) | 0xff000000
              }
            


              frequencies.forEach((v,i) => {
                image32[i + offset] = to_color(v)
              })

Networked Data

Fetch


              fetch('/object.stl')
                .then(res => res.arrayBuffer())
                .then(buffer => new Uint8Array(buffer))

WebSockets


              const ws = new WebSocket(url)
              ws.binaryType = 'arraybuffer'

              ws.onmessage = (event) => data = new Uint8Array(event.data)

              ws.send(buffer)

Websocket endpoints

Route voice data to a server


              const {In, Out} = nexmoGraph({audioCtx})

              navigator.mediaDevices
                .getUserMedia({ video: false, audio: true })
                .then(stream => audioCtx.createMediaStreamSource(stream))
                .then(node => node.connect(In))

              Out.connect(audioCtx.destination)

/

Networked data

Summary

Binary operators, Typed Arrays, ArrayBuffers

ImageData, AnalyserNode, Fourier transforms, BufferNode

Sharing Data, Fetch, WebSockets, Voice data

Thanks for listening

github.com/benfoxall/js-browser-bits

@benjaminbenben

Ben Foxall