Accessing the User’s Camera with JpegCamera and Canvas

This article was peer reviewed by Dan Prince. Thanks to all of SitePoint’s peer reviewers for making SitePoint content the best it can be!

Over the past decade, browser vendors have introduced various new APIs that enable us as programmers to create richer and more fluent experiences. One of these is the getUserMedia API, which enables access to the user’s audio and video devices. However, it’s still not quite there yet in terms of browser compatibility.

With this in mind, Adam Wróbel wrote JpegCamera, a library that takes into account the different caveats among browsers for interacting with user’s camera and provides fallbacks for those cases where access to client’s media is not supported.

In this article we’ll see how by using JpegCamera, together with HTML canvas element capabilities, we can build a clone of Instagram’s Layout app:

The demo Layout-like application

The source code for the demo can be downloaded from Github.

The JpegCamera Library

JpegCamera allows you to access the user’s camera as part of your application, gracefully degrading to a Flash fallback if the browser does not support getUserMedia().

The first step to is to include the necessary scripts in your project.

The library depends on the SWF Object and Canvas to Blob libraries, both of which come as part of the zip download from the project’s Github page. However, in the same zip there’s a with dependencies version of the script, which provides the same functionality as having the three scripts loaded in the page.

With this in mind, you can either include the three needed scripts.

<script src="/jpeg_camera/swfobject.min.js" type="text/javascript"></script>
<script src="/jpeg_camera/canvas-to-blob.min.js" type="text/javascript"></script>
<script src="/jpeg_camera/jpeg_camera.min.js" type="text/javascript"></script>

Or just go with the one script alternative.

<script type="text/javascript" src="js/libs/jpeg_camera/jpeg_camera_with_dependencies.min.js"></script>

For production environments, the later seem to be the way to go, unlike during development.

Once the library is included you can use the global JpegCamera object to check the camera availability, and choose how to manage the fallback if not.

If the access is granted, you can setup a listener for when the camera is ready with the JpegCamera() constructor.

The JpegCamera() constructor takes a CSS selector as an argument which should identify the container to be used for the camera stream.

The snippet below shows the code that does this:

(function() {
    if(!window.JpegCamera) {
      alert('Camera access is not available in your browser');
    } else {
      JpegCamera('.camera')
        .ready(function(resolution) {
          // ...
        }).error(function() {
          alert('Camera access was denied');
        });
    }
})();

This way, you can setup your application to only start once the camera is ready, or let the user know that they either require a different browser or need to enable camera access for the application.

Inside the ready callback function, the device’s camera resolution is passed as the first argument. This can be useful if the application we are building relies on the device’s camera quality (i.e.: to make HD capture available or not).

Meanwhile the error callback receives as an argument a string message explaining what happened. If you need to show the user an explanation in case of an error, you can use the message the library supplies.

In addition to this, the JpegCamera API provides the following methods:

• capture(): This is the method that takes a picture. It returns the image itself as a Snapshot object (the class that JpegCamera uses for images).
• show(): Once you take the picture, the Snapshot object that you obtain allows you to display the image in the page, by invoking its show() method. The image will be displayed inside the same container you specified when initializing the camera.
• showStream(): If a snapshot is currently displayed in the container, showStream() hides the image and displays the stream.
• getCanvas(): Takes a callback function as a parameter, which will receive as an argument the canvas element with the captured image.

Let’s dive into an example application that illustrates what JpegCamera allows us to do.

Building the Application

The demo application emulates (sort of) what Layout does: it allows the user to take photos and generates new images by combining them. In our version, the combined images can be downloaded by clicking on them.

The application structure is based on the Module Pattern. This pattern gives us a couple of benefits:

1. It allows to have a clear separation between each of the application components.
2. It keeps our global scope clean by only exposing methods and properties that are strictly required by the others. In other words, we get to use private attributes.

You’ll notice that I pass three parameters into the self invoked functions:

(window, document, jQuery)

And these arguments are received:

function(window, document, $)

The reason for passing window and document is for minification purposes. If we pass these as arguments, then each of them can be replaced for a single character. If we had just referenced these global objects directly, the minifier would not be able to substitute them with shorter names.

With jQuery, we do it to avoid conflicts with other libraries that might also using $ as their main function (i.e.: Prototype).

At the top of the Layouts and Custom modules you’ll see something along these lines:

if(!window.LayoutApp) {
  window.LayoutApp = {};
}

This is for two reasons:

1. We prevent the modules from generating errors in case we did not include the scripts properly in index.html.
2. We keep our global scope clean by making the modules part of a main one and only available for it once the application starts.

The application logic is divided into three modules:

• The App module
• The Layouts module
• The Custom module

These three modules together with our libraries must be included in our index.html as follows:

<!-- index.html -->
<script type="text/javascript" src="js/libs/jquery-1.12.1.min.js"></script>
<script type="text/javascript" src="js/libs/jpeg_camera/jpeg_camera_with_dependencies.min.js"></script>
<script type="text/javascript" src="js/src/custom.js"></script>
<script type="text/javascript" src="js/src/layouts.js"></script>
<script type="text/javascript" src="js/src/app.js"></script>

And there’s one more small piece of code to start the application.

<!-- index.html -->
<script type="text/javascript">
  (function() {
    LayoutApp.init();
  })();
</script>

Now, let’s review the modules one by one.

The App module

This module holds the main application logic. It manages the user interaction with the camera, generates the layouts based on the pictures taken and enables the user to download the generated images.

Everything starts in the App module, with the init method.

// App module (app.js)
initCamera = function () {
  if (!window.JpegCamera) {
    alert('Camera access is not available in your browser');
  } else {
    camera = new JpegCamera('#camera')
      .ready(function (resolution) {})
      .error(function () {
      alert('Camera access was denied');
    });
  }
},

bindEvents = function () {
  $('#camera-wrapper').on('click', '#shoot', capture);
  $('#layout-options').on('click', 'canvas', download);
};

init: function () {
  initCamera();
  bindEvents();
}

When invoked, ìnit() kicks off the application by calling the following methods:

1. initCamera() starts the camera, if available, or shows an alert.
2. bindEvents() sets up the necessary event listeners: 1. The first one to take the photos upon clicking the Shoot button. 2. The second one to generate the download when clicking on one of the combined images.

capture = function () {
  var snapshot = camera.capture();
  images.push(snapshot);
  snapshot.get_canvas(updateView);
},

When the user clicks on Shoot, capture() is invoked. capture() uses Snapshot’s class method getCanvas() passing as the callback updateView() function.

updateView = function (canvas) {
  canvas.selected = true;
  canvases.push(canvas);

  if (!measuresSet) {
    setCanvasMeasures(canvas);
    measuresSet = true;
  }

  updateGallery(canvas);
  updateLayouts(canvas);
},

In turn, updateView() caches the new canvas object (see updateGallery()) and updates the layouts with the new image by invoking updateLayouts(), which is the method that does the magic.

updateLayouts() relies on the following three methods:

• setImageMeasures(): This one defines an adequate width and height for the images, considering how many have been taken.
• setSourceCoordinates(): By checking the image measurements, this returns the coordinates for the center of the image.
• setTargetCoordinates(): This one takes into account the index of the image to be drawn and returns the coordinates of where the images will be drawn on the target canvas.

In addition to this, calculateCoeficient() takes care of keeping the proportions between the orginal image and the one to be generated, by comparing the source and the target canvas measures.

Finally, updateLayout() draws the image in the new canvas by using context.drawImage() with the data from the four functions above. The implementation to be used will be the one that uses its eight parameters. Meaning that we specify the source coordinates, source measures, target coordinates and target measures.

The Layouts module

The Layouts module provides the basic layout data, together with some helper functions.

Since we want to keep our scopes clean and just share with the other modules what’s strictly necessary, the Layouts module gives access to the attributes the App module needs through its getters.

// Layouts module (layouts.js)
var CANVAS_MAX_MEASURE = 200,
    LAYOUT_TYPES = {
      HORIZONTAL: 'horizontal',
      VERTICAL: 'vertical'
    },
    LAYOUTS = [
      {
        type: LAYOUT_TYPES.VERTICAL
      },
      {
        type: LAYOUT_TYPES.HORIZONTAL
      }
    ];

return {

   getCanvasMaxWidth: function() {
     return CANVAS_MAX_MEASURE;
   },

   getLayouts: function() {
     return LAYOUTS.concat(Custom.getCustomLayouts());
   },

   isHorizontal: function(layout) {
     return layout.type === LAYOUT_TYPES.HORIZONTAL;
   },

   isVertical: function(layout) {
     return layout.type === LAYOUT_TYPES.VERTICAL;
   },

   isAvailable: function(layout, totalImages) {
     return !layout.minImages || layout.minImages <= totalImages;
   }
 }

As seen above, none of the modules can mutate what is inside the Layouts module, but all that’s needed for the application to work is readily available.

Here’s what each of these methods contribute to the application:

• getCanvasMaxWidth(): In order to keep the images tidy I determined a default width for them and assigned it to CANVAS_MAX_MEASURE. This value is used in the App module to define the combined image measures. See the snippet below for the actual math within the App module.

// App module (app.js)
setCanvasMeasures = function (canvas) {
    measures.height = canvas.height * MAX_MEASURE / canvas.width;
},

This way our combined images can have any measure we like, independently from how big the ones we get from JpegCamera are and we prevent any stretching or shrinking by maintaining the aspect ratio from the picture just taken.

• getLayouts(): Returns the layouts that generate the combined images from the pictures taken by the user. It returns both the application default layouts, together with the custom ones that can be added to the Custom module (more on this later).
• isHorizontal() and isVertical(): The default layout in the application are defined by setting its type attribute, which takes its values from the LAYOUT_TYPES. By receiving a layout object as an argument and relying on this constant, these two methods evaluate layout.type === LAYOUT_TYPES.HORIZONTAL and layout.type === LAYOUT_TYPES.VERTICAL. Based on the return values of these functions, the App module defines the measures, source coordinates and target coordinates for the combined images.
• isAvailable(): Depending on how many images the user took and considering the layout’s minImages attribute, this function determines if the layout should be rendered or not. If the user has taken as many images or more than the ones set as the minimum, then the layout will be rendered. Otherwise, if the user has not taken as many photos or the layout does not have a minImages attribute defined, then the combined image will be generated.

The Custom module

The Custom module allows the addition of new layouts with their own implementation of the applications main three methods: setImageMeasures(), setSourceCoordinates(), and setTargetCoordinates().

This can be achieved by having a new layout object added to the Custom module’s CUSTOM_LAYOUTS array with its own implementation of the above three methods.

// Custom module (custom.js)
var CUSTOM_LAYOUTS = [
  /**
  * Place your custom layouts as below
  */
  // ,
  // {
  //   setImageMeasures: function (layout, targetCanvas, imageIndex) {
  //     return {
  //       height: 0,
  //       width: 0
  //     }
  //   },
  //   setSourceCoordinates: function (canvas, layout, imageWidth, imageHeight, imageIndex) {
  //     return {
  //       x: 0,
  //       y: 0
  //     }
  //   },
  //   setTargetCoordinates: function (targetCanvas, layout, imageWidth, imageHeight, imageIndex) {
  //     return {
  //       x: 0,
  //       y: 0
  //     }
  //   }
  // }
];

Each of the overriden functions in the application, will check if the layout being drawn contain a function for this.

See how it’s done in App.setImageMeasures():

// App module (app.js)
setImageMeasures = function (layout, targetCanvas, imageIndex) {
  if (isFunction(layout.setImageMeasures)) {
    return layout.setImageMeasures(layout, targetCanvas, imageIndex);
  } else {
    if(Layouts.isVertical(layout)) {
      return {
        width: $(targetCanvas).width(),
        height: $(targetCanvas).height() / images.length
      };
    } else if(Layouts.isHorizontal(layout)) {
      return {
        width: $(targetCanvas).width() / images.length,
        height: $(targetCanvas).height()
      };
    }

    return {
      width: $(targetCanvas).width(),
      height: $(targetCanvas).height()
    };
  }
}

Here we simply check if the custom layout has its own implementation of a function to define the image measures and, if so, invoke it.

This is achieved by the isFunction() helper, which checks if the argument received is actually a function by checking its type.

// App module (app.js)
isFunction = function(f) {
  return typeof f === 'function';
}

If the current module does not contain its own implementation of setImageMeasures(), the application goes on and sets the measures according to the layout type (either HORIZONTAL or VERTICAL).

The same flow is followed by setSourceCoordinates() and setTargetCoordinates().

The new layout can determine the size of the section to be cropped from the image taken, from which coordinates, and where it will be placed on the target canvas.

One important detail is to keep in mind that the custom layout methods must return objects with the same attributes as the original methods do.

To be clearer, your custom implementation of setImageMeasures() should return something in this format:

{
  height: 0, // height in pixels
  width: 0 // width in pixels
}

Creating a custom layout

Let’s have a go at creating a custom layout. You can see the full code listing for this file here.

As seen in the Layouts module section, layouts can have a minImages attribute defined. In this case, let’s set it to 3. Let’s also have the first image taken cover 60% of the target canvas, while the next two will split the remaining 40%:

{
  minImages: 3,
  imageData: [
    {
      widthPercent: 60,
      heightPercent: 100,
      targetX: 0,
      targetY: 0
    },
    {
      widthPercent: 20,
      heightPercent: 100,
      targetX: 120,
      targetY: 0
    },
    {
      widthPercent: 20,
      heightPercent: 100,
      targetX: 160,
      targetY: 0
    },
  ],
  // ...

To achieve this, let’s apply a simple rule of three, using the targetCanvas measures:

// Custom module (custom.js)
setImageMeasures: function (layout, targetCanvas, imageIndex) {
  var imageData = this.imageData[imageIndex];
  if( imageData) {
      return {
        width: imageData.widthPercent * $(targetCanvas).width() / 100,
        height: imageData.heightPercent * $(targetCanvas).height() / 100
      };
  }

  return {
    height: 0,
    width: 0
  }
},

Since all of the functions receive as an argument the number of the image (imageIndex) currently being processed, we can arbitrarily determine the size, the source coordinates for cropping, and the coordinates where original image’s section will be placed in the target canvas for each of the photos.

In the case where there’s no data associated with a particular imageIndex , we can just return an object with both of the attributes set to 0. This way, if the user takes more pictures than those that are defined in the custom layout, the combined image will still look good.

Let’s override the other two functions:

setSourceCoordinates() Given that we want to include the center of the image with all of its vertical content, we will return an object with x set to 50 and y set to 0.

setSourceCoordinates: function (canvas, layout, imageWidth, imageHeight, imageIndex) {
  return {
    x: 50,
    y: 0
  }
},

setTargetCoordinates() Since we know the canvas’ measures, let’s just manually define where they’d be placed in the target canvas.

setTargetCoordinates: function (targetCanvas, layout, imageWidth, imageHeight, imageIndex) {
  var imageData = this.imageData[imageIndex];

  if (imageData) {
    return {
      x: imageData.targetX,
      y: imageData.targetY
    }
  }

  return {
    x: 0,
    y: 0
  }
}

As you may agree, there’s plenty of room for improvement on this example, but this should be enough to get you started.

Conclusion

As we saw, JpegCamera takes the pain out of using the user’s camera in your application without needing to worry about cross-browser compatibility.

Including it as part of your project is as simple as adding the needed scripts to the page, and to use it requires understanding just four API methods. Writing a fun application takes only slightly more than a couple hundreds lines of code!

How about you, have you ever needed to work with user’s media? Are you keen to try implementing your own layout? Let me know in the comments!

Frequently Asked Questions about JPEGCamera and Canvas