Slurping data into your webapp

What is the best way to get data into a webapp for visualisation?

I use webapps to build data visualizations for scientific users. I like webapps because the web-browser is, without doubt, the nicest platform to make graphics and user interfaces, especially for cross-platform situations.

When I think of webapps, I sometimes use just a static HTML [web-page with javascript. These are supremely portable as I can zip up a web-page, and send it to anyone, because everyone has a browser.

Now whether I am dealing with a static web-page, or a tight client/server embrace, or an AJAX cross-site talk, one problem I keep on coming across is - how do I get data into my web-app?

Now obviously, it’s JSON.

Or is it? In my command-line programs, I pretty much use JSON first as good policy. But with web-apps, JSON can be a pain.

Why? Loading JSON works great if you have a client/server setup on the same IP. But if you create a static web-page, it doesn't work. If you use cross-origin websites without fancy CORS settings, it doesn't work.

So what are the options?

Always run a coupled client/server. Only joking. Actually not joking. I was in a recent workshop doing some three.js tutorials. When we tried to do some texture loading on our local machine, we actually had to spin up a python server to load data.

In the past, I've loaded data as variables in a script. Yuck! But it works though you've massively polluted the global space. If you now load multiple data objects, you'll be doing the dance of variable name juggling.

I used JSONP for a while. But writing JSONP handlers is unweildy. As well, JSONP has been superceded by CORS for a lot of use-cases.

I finally stumbled on a solution that works for me. The solution is to embed your data into an AMD module and use require.js. To do this, I transform any data.json file into yourdata.js as an AMD module:

define(function() {
  return { 
    'json': 'literal'
  }
})

Then in your web-page, make sure you include require.js:

<script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.5/require.js"></script>

And then in your scripts:

require(['./yourdata.js'], function(data) {
  // do something with data
  console.log(JSON.stringify(data, null, 2))
})

Why is this great? It works everywhere, even in your local files. You can use relative file-names, or you can use cross-origin urls. It doesn't matter.

You might ask why AMD modules and require.js? It's because that's the only way that modules will work in browsers without extensive transpiling workflows.

By using the require function, you've controlled scoping of your data variables, and you can guarantee execution order.

And nope, you won't have to spin up a server just to load data anymore!