For this fifth Le lab session, we wanted to build something that was really missing on a daily basis at TailorDev: a backend for Watson, our wonderful CLI to track our time.

TailorDev Le lab #5 crick.io

Wow, it’s been a looong time since our last Le lab session (almost a year). Shame on us, it’s getting harder and harder to book a full week dedicated to these hack sessions. As explained in a previous blog post, we decided to have Le lab sessions every Quarter. This session is for Q1 and we already have booked 3 more weeks until the end of this year.

Anyway, if you are not familiar with our Le lab concepts, we invite you to read the blog post of our first Le lab session. TL;DR: Le lab is a week devoted to set up a PoC with new tools or technologies we want to learn.

Introducing Crick

If you have been following our story, you probably know that the idea of Crick is not new at TailorDev. From Day 1, when we discussed the idea of a time-tracker working as a CLI and synchronized with a backend with Yannick, we chose to call this couple Watson (the CLI) & Crick (the backend). Watson came to life first and we postponed the backend development.

Everything went well for months. Thanks to the community, Watson became a reliable professional tool. We use it everyday to track every project we are working on.

At the same time, however, it was annoying to (i) ask people from the team to report the time spent for a customer (or internal) project, (ii) do the math for those reports and finally, (iii) generate an invoice and analyze our activity.

We needed a minimalist backend able to consolidate multiple users data to generate a global report for a project.

That’s precisely the scope of this Le lab session. Keep reading!

Architecture & implementation

Crick has a classical architecture that relies on two main components: a HTTP API designed as a Go application and talking to a PostgreSQL database, and a React/Redux web application consuming this API.

The Crick API

The Crick API is a Go 1.8 (at the time of writing) application. It relies on the great httprouter, a bunch of middleware and sqlx + pq for the database layer.

Below is the file structure of the api/ directory in the crick mono-repository. We use dep to manage the dependencies, hence the Gopkg.* files. For now, we decided to put the vendor/ directory under version control to ensure perfect reproducibility. Think it is dumb? Tell us.

.
├── Gopkg.lock
├── Gopkg.toml
├── config/
├── handlers/
├── main.go
├── middleware/
├── migrations/
├── models/
└── vendor/

The models/ package contains the data layer, including the SQL queries and Go structures to query and manipulate the Crick data. A Go interface named Repository exposes the different methods available to manipulate these data, and one concrete implementation of that interface is the DatabaseRepository. This design has been very helpful for testing the API (more on that later).

We use migrate to manage the (one-way) database migrations, which are written in plain old SQL and stored in the migrations/ directory. One-way means we do not allow rollbacks.

The handlers package contains the HTTP handlers a.k.a. the “controllers” in some other languages/frameworks, and the middleware package contains the authentication middleware.

The Crick API has two authentication mechanisms: the first one leverages Auth0 (OpenID Connect) for the web application and the other one is “API token”-based for Watson. These two different authentication schemes implement a uniform API so that the handlers are not aware of which authentication layer was used.

The User is added to the context by the middleware, and one can retrieve it via a function exposed by the middleware package in a handler: no direct access to the context.

import (
	"net/http"

	"github.com/TailorDev/crick/api/middleware"
	"github.com/julienschmidt/httprouter"
)

func MyHandler(w http.ResponseWriter, r *http.Request, ps httprouter.Params) {
	user := middleware.GetCurrentUser(r.Context())

	...
}

Finally, the main.go file contains the logic to connect everything together, including Uber Zap logger and systemd activation listener. We will cover this part as well as how we deploy this API in another blog post, so stay tuned!

During this session, we made the mistake to hack on the API before having entirely specified it, which was slightly counter-productive. Repeat after me: documentation-first is the only way to go! Talking about it, we have used Apiary.io one more time and the whole documentation is available at: docs.crickapi.apiary.io.

A note on testing

The Crick API is a basic Go application: its job is to fetch data into a database and to expose them in JSON over HTTP(S). Testing such an application should not be complicated. It should be a matter of booting a database, inserting fixtures, running the test suite, and cleaning everything up right after.

This process is actually cumbersome and we wanted to focus on testing specific parts of the API. Le lab sessions are time-boxed so we have to take shortcuts sometimes. The most important API endpoint is /frames, which takes many different optional query parameters and returns a set of frames along with varying meta data.

Let’s take two examples. If you pass a projectId query parameter then you expect only the frames for that project along with a project object in the meta data:

{
  "meta": {
    "project": {
      "id": "<UUID>",
      "name": "project-name"
    }
  },
  "frames": [
    ...
  ]
}

You can also retrieve the frames from different projects by specifying a list of project names (projects query parameter) and then you would get the frames along with a projects meta array that time:

{
  "meta": {
    "projects": [
      {
        "id": "<UUID 1>",
        "name": "project-name"
      },
      {
	"id": "<UUID 2>",
        "name": "another-project-name"
      }
    ]
  },
  "frames": [
    ...
  ]
}

Some parameters are mutually exclusive but other can just play nice together. We wrote a (SQL) QueryBuilder to process all the query parameters and return the corresponding and valid SQL query. It is also able to slightly modify the query to count the number of results for that query because we also support pagination (it would have been too easy otherwise I guess…).

Testing this QueryBuilder was straightforward, but the combination of the Repository interface and this QueryBuilder allowed us to write test cases that looked like this one:

func TestGetFramesWithProjectID(t *testing.T) {
	r := &MockRepository{}
	h := handlers.New(r, zap.NewNop())

	router := httprouter.New()
	router.GET("/test-endpoint", h.GetFrames)

	req, err := http.NewRequest("GET", fmt.Sprintf("/test-endpoint?projectId=%s", uuid.NewV4()), nil)
	if err != nil {
		t.Fatal(err)
	}

	req = req.WithContext(AddUserToContext(req.Context(), GetFakeUser()))

	rr := httptest.NewRecorder()
	router.ServeHTTP(rr, req)

	// assert status code, etc.

	query := r.QueryBuilder.ToSQL()
	expected := `<EXPECTED SQL QUERY>`

	if query != expected {
		t.Fatalf("invalid SQL query, expected: `%s` but got: `%s`", expected, query)
	}
}

The MockRepository is another implementation of the Repository interface. Its structure contains different members, one of them being a QueryBuilder pointer. The GetFramesWithQueryBuilder(qb QueryBuilder) method of the repository has been implemented so that it retains qb into the MockRepository, thus making it available in the test case. We can now test the whole logic to transform a HTTP request into a valid SQL query :sunglasses:

The Crick Web app

The frontend of Crick has been designed as a SPA powered by the React.js/Redux combination. This is the third official React application made by TailorDev, and we might say that it’s certainly the cleanest one we’ve built so far.

The Crick web application does not look like the previous ones: we have enforced Flow types almost everywhere, we have used a logical component architecture, started to use selectors to avoid direct access to the state, and used redux-api-middleware.

Logical component architecture, you said?

The logical component architecture as we call it is inspired by Marmelab directory structure and Jack Hsu’s rules for structuring a Redux application. In short, everything goes into a src/<Component> directory. We follow the Ducks convention for our Redux logic, which is bundled into the reducer.js file. The index.js is the connected React component, bridging the UI and the data. The presenter.js is the main React component and it is often a pure (stateless) component. Other components used by the presenter have proper names, e.g. List.js or Item.js. The style of the logical component can be found in a index.css file. Last but not the least, the test files are located in a sub-directory named __tests__/.

src/Auth
├── __tests__
│   ├── presenter.js
│   └── reducer.js
├── index.css
├── index.js
├── presenter.js
└── reducer.js

We mentioned the use of selectors. That was not something we have done in the past. The idea is to avoid direct access to the state. Each reducer exposes its state type (e.g., AuthState) and, at least, one state-level selector to access it. In the near future, we will have more specialized selectors to access some of the data.

// reducer.js

export type AuthState = {
  id: ?string,
  token: ?string,
  // ... other data types
};

export const selectAuthState = (state: State): AuthState => {
  return state.auth;
};

The AuthState allows to type the reducer() function, which is helpful to avoid returning bad states in one of the case statements processing the actions:

// reducer.js

export default function reducer(state: AuthState = initialState, action: Action = {}): AuthState {
  // reducer logic
};

The selectors are used in the connected components. Here is an example of the selectAuthState() selector in the index.js of the Auth component:

// index.js

const mapStateToProps = (state: State) => {
  const auth = selectAuthState(state);

  // ... return props
};

Last, State type is declared in a global src/types.js file and it looks like this:

export type State = {
  auth: AuthState,
  projects: ProjectsState,
  // ... other states
};

Material-UI

For efficiency/pragmatic concerns, we decided to use a set of React-ready components for the user interface (UI) instead of writing our own ones based on a CSS framework. Material-UI was our first choice to implement Crick’s very first UI. We only tuned it with TailorDev’s colors:

// TailorDev MUI theme for Crick
const crickTheme = getMuiTheme({
  palette: {
    primary1Color: '#2e354f',
    primary2Color: '#060f27',
    primary3Color: '#585f7b',
    accent1Color: '#269272',
    accent2Color: '#91f6d1',
    accent3Color: '#5ec3a0',
    textColor: '#333',
  },
});

const store = configureStore();

ReactDOM.render(
  <MuiThemeProvider muiTheme={crickTheme}>
    <Provider store={store}>
      <Router>
        <App />
      </Router>
    </Provider>
  </MuiThemeProvider>,
  document.getElementById('root')
);

Tip: you can easily design your own theme with the Material Design Color Tool :muscle:

A quick overview of the service

Words stay words and you might feel a bit frustrated not having the opportunity to view or test the service as we did not set up a public instance of Crick. To apologize, we have crafted nice gifs to highlight some features of the interface.

Login with Auth0

Thanks to Auth0, logging in to Crick is blazing fast once you authorized Auth0 to use your third-party account (like GitHub, Bitbucket or GitLab). Once logged in, users are invited to configure Watson with their Crick credentials:

Crick login

Project report

After having run watson sync to push your frames to Crick, users are able to explore nice reports per project. A calendar heatmap à la GitHub shows the activity on a given project over the past year (last 300 days actually), and it is possible to get different reports by specifying date spans and/or some tags:

Crick crick project report

Team report

Team reports accumulates users frames for shared projects in a team and provides similar features as the project report view. This is the core feature of Crick we wanted to achieve during this Le lab session:

Crick teams

That’s it!

Conclusions

To wrap up this Le lab session, here is a summary of what we’ve learnt:

  1. Before getting your hands dirty, intensively think about your API usage and design it consequently; we have lost too much time revamping the specification/implementation. Using a tool like Apiary to define your API blueprint is required;
  2. Getting off the ORM-road and take full advantage of your database is the way to go to avoid unnecessarily complex application architectures. Come on, SQL is not that hard. You know it. :muscle:
  3. Material-UI is a pretty good choice to quickly implement an interface for your service. We did not regret it during this week;
  4. Dealing with deadlines is hard. Develop a POC in a week and communicate about it is really short considering the rest of our activities both at a professional (emails, customers, etc.) and personal (kids, sport, etc.) levels. For the next sessions, we might consider a team retreat for 5 days in a neutral place, far from the rest of world (but with access to internet :grin:).

Having this new Le lab session was not that easy because we decided to develop a project that apparently nobody wanted (see our first lean experiment), except us! At this time, we do not have a precise idea of what Crick will become. For now, we will use it internally, improve the UI and increase the test suites before opening it to the community. Yes, the only absolute certainty is that Crick will be an Open Source project, like every Le lab session results.

If you raise your hands and hail this project, we could also set up a public instance of Crick. What do you think? Would it be of interest for you? Don’t be shy and get in touch with us :wink:

Update: we have opened a PR on Watson to fix the watson sync command.


PS: want to know more about the Crick project or want us to write about a given topic? Please contact us via Twitter or email, we are friendly.