{"id":1155,"date":"2015-08-21T03:42:22","date_gmt":"2015-08-21T07:42:22","guid":{"rendered":"http:\/\/blog.agilityfeat.com\/?p=1155"},"modified":"2015-08-21T03:42:22","modified_gmt":"2015-08-21T07:42:22","slug":"how-to-spectrum-analyser-for-webrtc-audio-stream","status":"publish","type":"post","link":"http:\/\/34.200.113.64\/en\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/","title":{"rendered":"How to: Spectrum Analyser for WebRTC Audio Stream"},"content":{"rendered":"<p><script src=\"https:\/\/s3-us-west-2.amazonaws.com\/blogs-scripts\/util\/code-scripts-1.js\"><\/script><\/p>\n<p><script src=\"https:\/\/platform.vine.co\/static\/scripts\/embed.js\"><\/script><\/p>\n<p>I&#8217;m writing a series over at <a href=\"http:\/\/realtimeweekly.co\/\" target=\"_blank\" rel=\"noopener noreferrer\">Real Time Weekly<\/a> about using <a href=\"http:\/\/realtimeweekly.co\/how-to-use-webrtc-on-your-ios-projects-part-i\/\" target=\"_blank\" rel=\"noopener noreferrer\">WebRTC on iOS<\/a>, specifically to document the research we&#8217;ve been conducting in order to release a couple of iPhone apps, based on WebRTC, under AgilityFeat&#8217;s brand. One of these apps is a remote spectrum analyzer based on <strong>real-time<\/strong> audio streams.<\/p>\n<p><script src=\"https:\/\/s3-us-west-2.amazonaws.com\/blogs-scripts\/vine\/vine-2015-08-005.js\"><\/script><\/p>\n<p>In this tutorial we&#8217;ll go over the steps to build a spectum analyzer using WebGL and GLSL shaders in plain HTML and using the <strong>getUserMedia<\/strong> function to access the microphone.<\/p>\n<h2>ThreeJS and GLSL Shaders<\/h2>\n<p>I&#8217;ve written about shaders before on <a href=\"http:\/\/www.agilityfeat.com\/blog\/2015\/08\/using-webgl-in-ios-without-phonegap-or-ionic\" target=\"_blank\" rel=\"noopener noreferrer\">my guide to using WebGL in iOS without PhoneGap or Ionic<\/a> and why they&#8217;re so useful for mobile platforms. The quick version of it is that using Vertex and Fragment Shaders forces your application to use GPU (graphics chip) rather than CPU (processor) to generate graphics.<\/p>\n<p>ThreeJS provides a very simple API to pass GLSL Shaders to it. When creating 3D objects you can assign them different types of materials. One of the available materials which ThreeJS offers is the <code>THREE.ShaderMaterial<\/code>.<\/p>\n<p>Shaders are written in a flavor of C++ called GLSL. Instead of writing Javascript which gets translated to GLSL calls, ThreeJS took the wise decision of interpreting GLSL shaders by accepting them as a good ol&#8217; string.<\/p>\n<p>In order to interact with the shaders, in this case for animation purposes, we use a few pre-disposed variables which ThreeJS exposes to us as properties of the <code>ShaderMaterial<\/code>.<\/p>\n<p>The two properties we&#8217;ll be looking at are the <code>attributes<\/code> and the <code>uniforms<\/code>.<\/p>\n<p>A couple important notes:<\/p>\n<p><strong>Shader attributes<\/strong> are parsed only once, which means that even though you can change their values after rendering the first frame, once rendering has begun, the shader will only use the initial value you set.<\/p>\n<p><strong>Shader uniforms<\/strong> on the other hand are dynamic and will be re-interpreted on each frame render.<\/p>\n<p>Both attributes and uniforms are strongly typed, and even though you&#8217;ll see standard data types such as <code>floats<\/code> and <code>integers<\/code>, you&#8217;ll also see some GLSL specific types such as <code>vec3<\/code> which are meant to be used for multi-dimensional arithmetic.<\/p>\n<p>For a full set of supported data types take a look at the official list on Github:<\/p>\n<p><a href=\"https:\/\/github.com\/mrdoob\/three.js\/wiki\/Uniforms-types\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/mrdoob\/three.js\/wiki\/Uniforms-types<\/a><\/p>\n<h2>Preparing a Spectrum GLSL Shader<\/h2>\n<p><a href=\"https:\/\/agilityfeat.com\/wp-content\/uploads\/2015\/08\/spectrum-bars.png\"><img loading=\"lazy\" src=\"https:\/\/agilityfeat.com\/wp-content\/uploads\/2015\/08\/spectrum-bars.png\" alt=\"Chart depicting how bars are displaced using GLSL shaders.\" style=\"max-width:375px; max-height:250px;\" width=\"375\" height=\"250\"\/><\/a><br clear=\"left\"\/><\/p>\n<p>As you can see in chart above, what I&#8217;m going to do, for simplicity&#8217;s sake, is to move the bars up and down based on the frequency data, one vertex at a time, instead of resizing them or moving just the top two vertices of each bar. I talk about vertices and not shapes because shaders work like that, applying transformations to each vertex.<\/p>\n<p>Let&#8217;s start with the most basic of vertex shaders, with no transformations being applied whatsoever:<\/p>\n<pre><code class=\"javascript\">THREE.SpectrumShader = {\n  defines: {},\n  \n  attributes: {\n    \/\/ We'll add our attributes here\n  },\n  \n  uniforms: {\n    \/\/ We'll add our uniforms here\n  },\n\n  vertexShader: [\n\n    \"varying vec3 vNormal;\",\n\n    \"void main() {\",\n      \n      \"gl_PointSize = 1.0;\",\n      \n      \"vNormal = normal;\",\n      \n      \"vec3 newPosition = position + normal;\",\n      \n      \"gl_Position = projectionMatrix * modelViewMatrix * vec4(newPosition, 1.0);\",\n\n    \"}\"\n\n  ].join(\"\\n\"),\n\n  fragmentShader: [\n\n    \"void main() {\",\n      \n      \"gl_FragColor = vec4(0.0, 0.0, 0.0, 1.0);\", \/\/ R, G, B, Opacity\n\n    \"}\"\n\n  ].join(\"\\n\")\n};<\/code><\/pre>\n<p>This should be placed on a separate Javascript file, as a good practice.<\/p>\n<p>So I&#8217;m going to need an attribute for each vertex to hold the maximum amount of pixels they will be pushed down, and a uniform to hold a relation from <code>0.0<\/code> to <code>1.0<\/code> which I&#8217;ll modify on a frame-to-frame basis. Later I&#8217;ll take these values and multiply them together to get the final amount I will move down the vertices of each bar.<\/p>\n<p>We modify the following lines of our shader:<\/p>\n<pre><code class=\"javascript\">attributes: {\n  \/\/ We'll add our attributes here\n},\n\nuniforms: {\n  \/\/ We'll add our uniforms here\n},<\/code><\/pre>\n<p>And we&#8217;re left with:<\/p>\n<pre><code class=\"javascript\">attributes: {\n  \"translation\": { type:'v3', value:[] }, \/\/ The values are of type Vector3 so we can\n                                          \/\/ translate each vertices X, Y, and Z positions\n  \"idx\": { type:'f', value:[] } \/\/ I set this as a float value since integer attributes\n                                \/\/ are not allowed\n},\n\nuniforms: {\n  \"amplitude\":  { type: \"fv1\", value: [] } \/\/ This is a regular array of float values\n},<\/code><\/pre>\n<p>The <code>idx<\/code> attribute you see there has to do with the fact that WebGL is built on top of an older implementation of OpenGL and GLSL, and as such it doesn&#8217;t provide and id to know which vertex we are looking at on each iteration.<\/p>\n<p>On our main Javascript we can instantiate our <code>ShaderMaterial<\/code> the following way:<\/p>\n<pre><code class=\"javascript\">material = new THREE.ShaderMaterial( THREE.SpectrumShader );<\/code><\/pre>\n<p>Once our material is instantiated, we can access the attributes and uniforms like this:<\/p>\n<pre><code class=\"javascript\">material.attributes.translation.value.push(new THREE.Vector3(0.0, 145.0, 0.0));<\/code><\/pre>\n<h2>Drawing bars in a ThreeJS geometry<\/h2>\n<p>The awesome thing about using GLSL shaders is that through clever vertex-per-vertex transformations you can treat a single geometry as if it was multiple objects.<\/p>\n<p>I will be drawing 50 bars on screen, but in reality I am going to be generating a single geometry which will hold 50 clusters of 4 vertices and 2 faces each, as such:<\/p>\n<p><a href=\"http:\/\/i.giphy.com\/l41lL7A6u4BrgAXOU.gif\"><img loading=\"lazy\" src=\"http:\/\/i.giphy.com\/l41lL7A6u4BrgAXOU.gif\" alt=\"Constructing multiple bars in a single ThreeJS geometry.\" style=\"max-width:450px; max-height:450px;\" width=\"450\" height=\"450\"\/><\/a><br clear=\"left\"\/><\/p>\n<pre><code class=\"javascript\">function addBar(x,width){\n  \/\/ Generate id on a per-bar basis\n  var idx = geometry.vertices.length \/ 4;\n  \n  \/\/ Each bar has 4 vertices\n  geometry.vertices.push(\n    new THREE.Vector3( x, 0, 0 ),\n    new THREE.Vector3( x + width - 1, 0, 0 ), \/\/ Minus one pixel to be\n    new THREE.Vector3( x + width - 1, 150, 0 ), \/\/ able to differentiate bars\n    new THREE.Vector3( x, 150, 0 )\n  );\n  \n  \/\/ Each bar has 2 faces\n  var face = new THREE.Face3( idx * 4 + 0, idx * 4 + 1, idx * 4 + 2 );\n  geometry.faces.push( face );\n  face = new THREE.Face3( idx * 4 + 0, idx * 4 + 2, idx * 4 + 3 );\n  geometry.faces.push( face );\n  \n  \/\/ As well as two faceVertexUvs (one per face)\n  \/\/ Useful if applying image textures\n  geometry.faceVertexUvs[0].push([\n    new THREE.Vector2( 0, 1 ),\n    new THREE.Vector2( 1, 1 ),\n    new THREE.Vector2( 1, 0 )\n  ]);\n  geometry.faceVertexUvs[0].push([\n    new THREE.Vector2( 0, 1 ),\n    new THREE.Vector2( 1, 0 ),\n    new THREE.Vector2( 0, 0 )\n  ]);\n  \n  \/\/ For every vertex we create a translation vector\n  \/\/ an id (every 4 vertices share the same id)\n  \/\/ and an amplitude uniform for each bar\n  for(var i = 0; i < 4; i++){\n    material.attributes.translation.value.push(new THREE.Vector3(0.0, 145.0, 0.0));\n    material.attributes.idx.value.push(idx);\n    if ((i % 4) === 0) material.uniforms.amplitude.value.push(1.0);\n  }\n}\n\n\/\/ Generating 50 bars across a length of 250 pixels\nfor(var i = 0; i < 50; i += 1){\n  addBar( -((10 * i) - 250) - 10, 10 );\n}<\/code><\/pre>\n<h2>Pull audio from microphone using WebRTC<\/h2>\n<p>Now, on to the WebRTC of it all, we need to fetch the audio from the microphone:<\/p>\n<pre><code class=\"javascript\">function start_mic(){\n  if (navigator.getUserMedia) {\n    navigator.getUserMedia({ audio: true, video: false }, function( stream ) {\n      \/\/ The audio context and the analyser pull the audio data and\n      \/\/ allow us to store it in an array of frequency volumes\n      audioCtx = new (window.AudioContext || window.webkitAudioContext)();\n      analyser = audioCtx.createAnalyser();\n      \n      \/\/ Our source is our WebRTC Audio stream\n      source = audioCtx.createMediaStreamSource( stream );\n      source.connect(analyser);\n      \n      analyser.fftSize = 2048;\n      bufferLength = analyser.frequencyBinCount;\n      \n      \/\/ We will pull the audio data for each bar from this array\n      dataArray = new Uint8Array( bufferLength );\n      \n      \/\/ Let's begin animating\n      animate();\n    }, function(){});\n  } else {\n    \/\/ fallback.\n  }\n}<\/code><\/pre>\n<p>and feed it to the shader through the <code>amplitude<\/code> uniform:<\/p>\n<pre><code class=\"javascript\">function animate(){\n  requestAnimationFrame( animate );\n  \n  analyser.getByteFrequencyData(dataArray);\n    \n  for(var i = 0; i < 50; i++) {\n    material.uniforms.amplitude.value[i] = -(dataArray[(i + 10) * 2] \/ 255) + 1;\n  };\n  \n  render();\n}<\/code><\/pre>\n<h2>Using our GLSL Shader to Animate the Spectrum Bars<\/h2>\n<p>Everything we've done so far is fine and dandy, but we need to tell the shader what to do with the audio data.<\/p>\n<p>Every vertex shader is composed at least by 3 positional elements. The <strong>projection matrix<\/strong> (the position of a vertex relative to the camera), the <strong>model view matrix<\/strong> (the position of a vertex relative to the object it belongs to), and the <strong>position matrix<\/strong> (the position of a vertex relative to the screen).<\/p>\n<p>In this case we are going to be altering the <strong>position matrix<\/strong>:<\/p>\n<pre><code class=\"javascript\">vertexShader: [\n  \n  \"attribute vec3 translation;\",\n  \n  \"attribute float idx;\",\n  \n  \"uniform float amplitude[ 50 ];\",\n\n  \"varying vec3 vNormal;\",\n  \n  \"varying float amp;\",\n\n  \"void main() {\",\n    \n    \"gl_PointSize = 1.0;\",\n    \n    \"vNormal = normal;\",\n    \n    \"highp int index = int(idx);\",\n    \n    \"amp = amplitude[index];\",\n    \n    \"vec3 newPosition = position + normal + vec3(translation * amp);\",\n    \n    \"gl_Position = projectionMatrix * modelViewMatrix * vec4(newPosition, 1.0);\",\n\n  \"}\"\n\n].join(\"\\n\"),<\/code><\/pre>\n<p>As promised, you can see how it all (the attributes and uniform) come together on these lines:<\/p>\n<pre><code class=\"javascript\">\"amp = amplitude[index];\",\n    \n\"vec3 newPosition = position + normal + vec3(translation * amp);\",<\/code><\/pre>\n<p>The <code>newPosition<\/code> vector holds the new <code>Y<\/code> coordinate based on the amplitude uniform we are passing it (basically the volume of the frequency this bar represents).<\/p>\n<p>The final shader looks like this:<\/p>\n<pre><code class=\"javascript\">THREE.SpectrumShader = {\n  defines: {},\n  \n  attributes: {\n    \"translation\": { type:'v3', value:[] },\n    \"idx\": { type:'f', value:[] }\n  },\n  \n  uniforms: {\n    \"amplitude\":  { type: \"fv1\", value: [] }\n  },\n\n  vertexShader: [\n  \n    \"attribute vec3 translation;\",\n    \n    \"attribute float idx;\",\n    \n    \"uniform float amplitude[ 50 ];\",\n\n    \"varying vec3 vNormal;\",\n    \n    \"varying float amp;\",\n\n    \"void main() {\",\n      \n      \"gl_PointSize = 1.0;\",\n      \n      \"vNormal = normal;\",\n      \n      \"highp int index = int(idx);\",\n      \n      \"amp = amplitude[index];\",\n      \n      \"vec3 newPosition = position + normal + vec3(translation * amp);\",\n      \n      \"gl_Position = projectionMatrix * modelViewMatrix * vec4(newPosition, 1.0);\",\n\n    \"}\"\n\n  ].join(\"\\n\"),\n\n  fragmentShader: [\n  \n    \"varying float amp;\",\n\n    \"void main() {\",\n      \n      \"gl_FragColor = vec4(-(amp) + 1.0, 0.5 + (amp\/2.0), 0.0, 1.0);\",\n\n    \"}\"\n\n  ].join(\"\\n\")\n};<\/code><\/pre>\n<p>As you can see on the <code>fragmentShader<\/code> I reuse the <code>amp<\/code> variable to get a nice fade to orange as the volume of a bar reaches its max level. The sharing of variables between shaders is possible due to the <code>varying<\/code> keyword.<\/p>\n<p>As always, the code for this demo is available on github:<\/p>\n<p><a href=\"https:\/\/github.com\/agilityfeat\/WebRTC-Spectrum-Demo\/tree\/master\" target=\"_blank\" rel=\"noopener noreferrer\">https:\/\/github.com\/agilityfeat\/WebRTC-Spectrum-Demo\/tree\/master<\/a><\/p>\n<p>Be sure to drop me a line ifyou find this info helpful.<\/p>\n<p>Happy coding!<\/p>","protected":false},"excerpt":{"rendered":"<p>I&#8217;m writing a series over at Real Time Weekly about using WebRTC on iOS, specifically to document the research we&#8217;ve been conducting in order to release a couple of iPhone apps, based on WebRTC, under AgilityFeat&#8217;s brand. One of these apps is a remote spectrum analyzer based on real-time audio streams. In this tutorial we&#8217;ll [&hellip;]<\/p>","protected":false},"author":11,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"categories":[128,57,116],"tags":[146,117],"jetpack_featured_media_url":"","yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v15.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>How to: Spectrum Analyser for WebRTC Audio Stream - AgilityFeat Panama Software Test Center<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to: Spectrum Analyser for WebRTC Audio Stream - AgilityFeat Panama Software Test Center\" \/>\n<meta property=\"og:description\" content=\"I&#8217;m writing a series over at Real Time Weekly about using WebRTC on iOS, specifically to document the research we&#8217;ve been conducting in order to release a couple of iPhone apps, based on WebRTC, under AgilityFeat&#8217;s brand. One of these apps is a remote spectrum analyzer based on real-time audio streams. In this tutorial we&#8217;ll [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/\" \/>\n<meta property=\"og:site_name\" content=\"AgilityFeat Panama Software Test Center\" \/>\n<meta property=\"article:published_time\" content=\"2015-08-21T07:42:22+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/agilityfeat.com\/wp-content\/uploads\/2015\/08\/spectrum-bars.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\">\n\t<meta name=\"twitter:data1\" content=\"7 minutes\">\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/34.200.113.64\/#website\",\"url\":\"https:\/\/34.200.113.64\/\",\"name\":\"AgilityFeat Panama Software Test Center\",\"description\":\"AgilityFeat Panama offers customized, multilevel web and mobile software testing for a variety of industries.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/34.200.113.64\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/agilityfeat.com\/wp-content\/uploads\/2015\/08\/spectrum-bars.png\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/#webpage\",\"url\":\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/\",\"name\":\"How to: Spectrum Analyser for WebRTC Audio Stream - AgilityFeat Panama Software Test Center\",\"isPartOf\":{\"@id\":\"https:\/\/34.200.113.64\/#website\"},\"primaryImageOfPage\":{\"@id\":\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/#primaryimage\"},\"datePublished\":\"2015-08-21T07:42:22+00:00\",\"dateModified\":\"2015-08-21T07:42:22+00:00\",\"author\":{\"@id\":\"https:\/\/34.200.113.64\/#\/schema\/person\/c34873e9bee22a4aab19b8a0e800aad0\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/agilityfeatpanama.com\/blog\/2015\/08\/how-to-spectrum-analyser-for-webrtc-audio-stream\/\"]}]},{\"@type\":\"Person\",\"@id\":\"https:\/\/34.200.113.64\/#\/schema\/person\/c34873e9bee22a4aab19b8a0e800aad0\",\"name\":\"Jean\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/34.200.113.64\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/?s=96&d=mm&r=g\",\"caption\":\"Jean\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/posts\/1155"}],"collection":[{"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/comments?post=1155"}],"version-history":[{"count":0,"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/posts\/1155\/revisions"}],"wp:attachment":[{"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/media?parent=1155"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/categories?post=1155"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/34.200.113.64\/en\/wp-json\/wp\/v2\/tags?post=1155"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}