After a successful first session, we are happy to share the results of our second Le lab session, which happened a week ago at TailorDev. We worked a lot (a bit too much to be honest) on Monod to make it usable in real-life.

TailorDev Le lab

Previously, we started to learn React by building a basic Markdown editor (Monod) in five days. For this second Le lab session, we wanted to tackle two problems we faced using Monod:

  1. no way to use Monod offline (and we love offline applications);
  2. no ability to share our writings.

Additionally, we wanted to provide a few templates to make final contents even more awesome! In short, templates are (stateless) React components designed to render content with style, along with metadata: YAML Front Matter FTW!

As you might imagine, this was quite a lot of new features to ship. Did we succeed? Yes. Did we really succeed? Nope, we actually failed to deliver “on time”, that is on Day 4. We really wanted all these features, and the problem was that we simply could not split such features into smaller chunks. We had to either ship each feature as a whole, or not work on it at all. Because we use Monod at TailorDev, we decided to spend an extra day working on the code.

Hurry to use this new release?

Try Monod now!

Offline-First with Service Workers

We live in a disconnected & battery powered world, but our technology and best practices are a leftover from the always connected & steadily powered past.
Source: http://offlinefirst.org/

Offline-First is a sort of paradigm that considers network failure or connectivity issue not like errors, but as a fact of life: it happens, but it is okay. We can deal with it, and we can even create beautiful offline-first applications, e.g., minutes.io, and now: Monod.

Making Monod offline-first was relatively easy since it had no backend: everything was stored locally (thanks to localForage). What we had to do was to have all its assets cached somewhere to make Monod available in the browser without any Internet connection. There are two ways of doing this: Application Cache (or AppCache) and Service Workers.

AppCache allows to specify which files the browser should cache and make available to offline users. That sounds great except that AppCache is a Douchebag. Nonetheless, it is still possible to rely on it (by fixing it with an iframe for instance), which is cool because it is widely supported.

Service Workers are scripts that are run by the browser in the background. While Service Workers are not really widely supported yet, they are really powerful and much better than Application Cache to provide an offline experience. For instance, Service Workers allow to build applications following the Application Shell Architecture.

We leveraged this Webpack offline-plugin to implement both Service Workers and AppCache (as fallback). We had to override a few things but the plugin integrates well with Webpack, and it was handy to add all the assets to the CACHE section of the manifest, or to deal with the Service Worker version.

The second major issue we faced lied in how we designed Monod during the first Le lab session: we were not able to create more than one document at the time, and those document were not shareable.

Sharing Documents

In order to share documents with others, there are a few things to take into consideration. First, we should be able to deal with different documents. Thereafter, we would need a central server to share documents among all the users. And because we care a lot about privacy, everything should be encrypted at some point. Bring the Offline-First paradigm into the game, and you should now understand why we failed to ship this feature in 2 days.

We started by adding the ability to deal with multiple documents, identified with a UUID. We did not add any document management system to list or retrieve existing documents, so one has to bookmark (or retain somewhere) the URLs of the documents à la Pastebin. Next step was to be able to encrypt/decrypt the documents.

On Encryption and Zero Knowledge

A basic technique to encrypt data in a safe and reliable manner is to perform the encryption/decryption operations on the client, not on the server. This is often called Zero Knowledge, which can be defined as the inability of the server to access plain text data while maintaining its duties to store and secure that data in our case. All Monod data, either stored locally or remotely, or exchanged over the network, are encrypted using AES (256 bits). As of now, when you create a new document on Monod, identified by a unique identifier <uuid>, a random 256 bits key is generated in the browser (<secret>), and put into a fragment. Because this part of the URL is client-side only, the key is never transmitted to the server, which therefore cannot decrypt anything.

Monod: zero-knowledge cloud storage

We relied on the Stanford JavaScript Crypto Library (SJCL), a small, secure, and powerful cross-browser library for cryptography in JavaScript. As we decided to follow the Offline-First approach, we started by making this part fully working before working on the server-side.

The Server-Side

The server (or “backend”) we implemented is an Express application that exposes an API, allowing to retrieve and store documents on the file system. It does not have any logic per se, except validating inputs.

When one clicks a Monod URL, the application loads, and then starts by looking into its local database. If it cannot find any document for the given UUID, it sends a request to the server API. If the server returns a document (whose content is encrypted), Monod decrypts it using the secret key part of the URL, and finally renders the content. Otherwise, you will get a message saying that the document cannot be retrieved or decrypted. This also means that, in order to share a document, you have to copy the full URL.

Monod: loading local/remote documents

The API has been fully covered by tests thanks to supertest. Wiring Monod and the API did not seem to be a big deal at first glance, except that we needed some sort of synchronization

Synchronization: Highway to Hell

Our goal was not to create a collaborative editor in a week, but when we thought about Monod sharing feature, we felt the need for a “simple” synchronization layer. It was probably a mistake, yet we are now happy with the final result.

We identified a couple use cases, but most of them actually led to many edge cases, especially with our Offline-First approach. At the end, the Pull Request for the sharing feature turned into a long discussion, and a lot of checkboxes! To be honest, we thought it would have been easier to write.

You might wonder why we did not choose any existing solution, e.g., Hoodie, PouchDB, or Kinto). We failed at understanding whether Hoodie was a good fit for us. PouchDB was designed to integrate with CouchDB on the server (even though there is an Express middleware), it seemed too much. Kinto seemed great except that data are owned by authenticated users.

In Monod, there is no notion of “user”, we only manage documents. Anyone with a full URL should be able to open the related document, and synchronize it for further use (Offline-First approach). In addition, we only want to synchronize documents that have been opened, not all the documents created on Monod. Therefore, we designed our own synchronization mechanism. We wrote a first version that seemed fine, but it is never fine when it is not (easily) testable, so we rewrote the whole damn thing.

We did not use redux or any Flux framework for two reasons: i. it was likely overkill, and, ii. we wanted to understand how things were supposed to work under the hood. This Kinto.js + React project boilerplate was a gold mine though, and we decided to follow their simple yet efficient architecture.

Our final implementation relied on Immutable.js, Facebook library providing many Immutable data structures, which was really great to avoid side-effects. We decided to write a 100% Promise-based API, instead of mixing promises and sequential code (hint: this does not work). We tried to keep one level of promises as much as we could, and, above all, to think in term of functional-ish programming. Every JavaScript developer MUST know what a pure function is, why we have pure and impure functions, and why immutability is interesting. These are likely the only things that will not change in the JavaScript ecosystem in the upcoming weeks ;-)

Rethinking the way we wrote this piece of software allowed us to extensively test it. Given that this part interacted with a server, we used faux-jax to fake the API HTTP responses. Here is an example for describing successive API calls using generators:

const responses = function* () {
	yield { status: 200, body: { last_modified: 1 } }
	yield { status: 200, body: { last_modified: 2 } }
}();

fauxJax.on('request', (request) => {
	const response = responses.next().value;

	request.respond(
		response.status,
		{ 'Content-Type': 'application/json' },
		JSON.stringify(response.body)
	);
});

The synchronization mechanism implemented at the moment allows to get the latest version of any document you are viewing, but also to fork the current document if there are any conflicts (i.e. the document has been modified by two persons), so that you never lose your work. We might miss a few more edge cases, but it seemed to work well for us. This feature is now available to you for live testing!

Document Templates

As we already wrote in our previous Le lab session report, we extensively use Markdown for our documents. Hence, we designed document templates that one need to checkout, fill and compile to generate a PDF (after installing a bunch of stuff). But this belongs to the past now that Monod supports document templates!

Front-Matter

As you can see from the snapshot below, there is a template selection form in the upper-right corner of the editor:

Monod: document templates

Once selected, the template is loaded with default values for variables, waiting for front-matter inputs. These inputs should be defined on top of your document and respect the YAML front-matter syntax:

---
key: value
foo:
  bar: lol
---

The rest of your document is considered as the “main content”. It is the designer’s responsibility to place it wherever it is consistent in the template.

Templates are (Stateless) React Components

For demonstration purpose, we shipped a few template samples with this release. Each template is a React component bundled with its own (inline) styles:

import React from 'react';
import BaseTemplate from './Base';

/**
 * Letter template
 */
export default class Letter extends BaseTemplate {

  getDefaultData() {
    return {
      date: '[date]',
      location: '[location]',
      // [...]
    }
  }

  render() {
    const data = this.getData();
    const
      letterStyle = {
        fontSize: '12pt',
        fontFamily: 'Palatino, "Times New Roman", Times, serif'
      },
      addressFromStyle = {
        marginTop: '0'
      },
      // [...]
      ;

    return (
      <article style={letterStyle}>
        <header>
          <address style={addressFromStyle}>
            <strong>{data.addressFrom.name}</strong><br/>
            {data.addressFrom.street}<br/>
            {data.addressFrom.zipCode}&nbsp;
            {data.addressFrom.city}<br/>
            {data.addressFrom.country}
          </address>
          // [...]
          <div style={locationDateStyle}>
            {data.date}, {data.location}
          </div>
        </header>
        <section style={contentStyle}>
          {this.props.content}
        </section>
        <footer style={signatureStyle}>
          <div>
            {data.signature}
          </div>
        </footer>
      </article>
    )
  }
}

Can I Haz PDF?

Yes! Print your current document (from your web browser) into a file and tadaaa!

One More Thing

As we guess what your next question will be, the answer is: YES, it will! We are going to open source Monod in a few days :tada: :balloon: :star2:

Stay tuned.


In the meantime, you can safely use our public instance, which we also use at TailorDev: https://monod.lelab.tailordev.fr. We are also happy to read your thoughts on Twitter!