One of the goals we set ourselves for 2016 was the re-architecture of our Fundraising & registrations platform and moving to micro services. The team has been busy on that this year and the work will continue well into 2017 too. Here is a small rundown of our journey so far.

Why?

Our previous monolith was Zend 1 based and it was getting harder to support, upgrade, extend and secure easily. Technical debt had crept in over 4 years and it was getting harder to separate the logic between different layers. A simple case was when we wanted to separate the presentation layer from the service layer so as to plug and play newer front end tools, and also upgrade one layer without affecting the other.

From a mere maintenance perspective, it was getting harder to upgrade the PHP version of the application, use other open source libraries and also improve the base framework without spending too many cycles testing and reworking, which stagnated innovation. Over time, all our practices and processes had also got centered around the nuances of our old platform and basically we were walking backwards in time. Our speed of delivery was getting affected and deployment processes were running the risk of being insecure.

It was time that we created a system which was extensible, scalable, easy to maintain, and fun to work with. Micro services fit that bill.

The start

Choices

Starting with a clean slate, we were able to revisit everything from our methodologies, to ways of working and inherent team structure as well. We had a relatively new project team on board but they had a grasp of the existing monolithic application to be able to determine what areas needed to be separated out. We also matched this separation with long term goals of re-use and repurposing some the layers for other purposes. It also opened up exciting areas of innovation based on the APIs we were inherently creating as part of these micro services.

With all the possibility of available choices for micro services and processes, we decided to pace ourselves and settled on having a hybrid architecture first before completely moving over to micro services so as to meet a few demands of our ongoing campaigns. Our team size is basically a core strength of 3 devs, and yes that does not necessarily fit Conway’s Law for an ideal micro services architecture but it fits the bill for now and as we involve more teams, we expect the ideal structure to take shape as well.

We began by setting ourselves deadlines for settling on our technology choices for each micro service. We prototyped, discussed, and documented the various choices as a team, before making a decision. We won’t go into the detail of the decision making process in this blog post for each layer, although will cover some of the main technologies chosen. Hopefully we can delve into details in future blog posts.

For the presentation layer we debated between Angular, Ember, and React.JS. We chose React.JS and set out to build an isomorphic application. Here are our notes from the comparison exercise we did in April 2016.

Angular 2.x Ember 1 or 2 React 15 or 0.14
Browser support All modern & IE9+ Ember 2: all modern & IE9+

Ember 1: all modern & IE8+

React 15: all modern & IE9+

React 0.14: all modern & IE8+ with additional ES5 shims

Community support Seems really good. Things are well documented, and in my opinion the whole project has a stability-focused & mature approach to new versions that’s still unusual in the JS framework world.

As a result many Ember 2 features have been back-ported to Ember 1, making it still a completely viable choice today with comprehensive deprecation warnings and so a very smooth upgrade path to Ember 2.

Ember 2.4 is the project’s first Long-Term Support version, with a commitment to critical bug fixes for 18 months after its release.

In wide use for a long time without drastic language changes.

Moved to semver this month (April) with version 15 and seem to be taking a similar approach to Ember with deprecations – “If your code is free of warnings when running under React 0.14, upgrading should be easy.”

Not sure how much has been ported back from 15 to 0.14 but it’s likely to still be at least as viable a choice as Ember 1.x. React 15 only came out very recently. Leaderboard prototype is on React 0.14.

Facebook / vendor support includes browser extensions for Chrome & Firefox.

Projects stats GitHub: 11k stars, 2.8k forks

Not stable yet but already looks like a couple of hundred addons on js.coach directory and >500 npm matches

GitHub: 16k stars, 3.4k forks.

2.3k addons.

GitHub: 41k stars, 6.8k forks

Can’t find an authoritative addons total but there seem to be some long standing popular ones for e.g. form handling & server rendering and lots on e.g. js.coach directory and npm.

Security Ember’s a stable project with a sensible disclosure policy for security issues.

Latest version logs console messages about the Content Security Policy, and you can lock down access to any external resources to ensure only trusted ones are loaded.

Found 2 historical CVE entries but there seem to be more not tied to Ember explicitly in the CVE database. However they also seem to have been taken seriously and patched quickly in general.

As the second link in the above § points out, Ember doesn’t enforce a sandbox policy for developer code. This doesn’t make us more vulnerable to XSS than e.g. jQuery does with our code today – it just means we’re not automatically better protected.

This presentation (slide 110) gives Ember more points out of the 3 for some security considerations, and awards it the highest ‘score’ overall. However this is a bit subjective, and Angular 2 might take a different approach e.g. on advisories once it’s stable.

I’ve only found 1 historical CVE mentioned anywhere for React itself – an XSS issue from December 2013. It seems to have been disclosed responsibly and React core appears to have a good track record.

React takes an approach which in general strongly discourages unsafe operations that are likely to lead to XSS problems in apps – although not checking input types can still lead to XSS problems like this 2015 vulnerability disclosed to HackerOne.

Performance New(ish) Glimmer engine greatly improved rendering performance. It was released + back-ported to Ember 1.13 in June 2015.

It’s still slower than React in some respects but quicker in layout & paint tests. There are some detailed benchmarks & explanations in this article.

Right now there still seem to be some Android performance problems & Discourse have built a custom renderer wrapping virtual-dom to get round it. But some comments are suggesting that ‘Glimmer 2’ (which should be in stable soon) will improve this greatly for Ember 2 anyway.

LinkedIn have developed & shared a Chrome extension for analysing & improving Ember apps’ performance.

Good performance.

React creates an in-memory data structure cache (Virtual DOM), computes the resulting differences, and then updates the browser’s displayed DOM efficiently.

React also provide performance tool API called ReactPerf, it is a profiling tool to overview app’s performance.

Server side rendering Angular Universal doesn’t seem to be attempting client-JS-free rendering currently (even in the starter example) or doing form processing. Seems unfinished & feature light even compared to e.g. Ember FastBoot. More limited than I’d hoped. FastBoot only works with Ember 2.3+, and doesn’t yet cover form processing. On its own it’s not a complete solution for stripped back IE8 (or non-JS) support – and probably still won’t be in the next few months.

Detailed notes here.

While React doesn’t try to do anything like this on its own, addons & modules to support universal JS (e.g. within Redux) and form processing (e.g. newforms) seem relatively mature and usable compared to the other frameworks considered.

Getting this working fully for our requirements may be a bit involved, but should definitely be possible.

Accessibility Angular 1 had ngAria – A2 is providing sensible ARIA behaviour built in, based on this (see surrounding slides). Notes & commits on this GitHub issue detailed some steps we can take to make an Ember app as accessible as possible, including determining which containers are focusable and announcing route / title changes.

a11y-testing addon can help catch some common a11y issues in templates – potentially making it easier to ensure a good screen reader UX than when using plain HTML, and making maintaining this standard a part of acceptance tests.

React uses custom html attribute to render web accessibility attribute for components. Additionally, react-a11y addon can scan react elements and warns about potential accessibility issues.

Looks like react-a11y can also be quite easily tied into acceptance tests, similar to what the Ember module does by default.

React Native also provide Accessibility API to enable support for iOS and Android.

Testing Angular2 favours jasmine+karma for unit testing and protractor for end to end testing. Quite a few articles like this one and this one explain how it works and provide step-to-step guide. Ember CLI builds a project structure and tool chain that gives you JsHint checks for free and makes unit & acceptance tests very easy and intuitive to add.

Having the data models separate in Ember Data also makes it easy to test specific, isolated aspects of their functionality. The relatively large number of pieces/ concepts does have some advantage in that each layer is independently testable.

Seems like with Jest you have quite a simple starting point for functional / black-box testing of each component in isolation. There also seem to be a range of other viable approaches for component testing.

There are also useful notes on React testing approaches here.

Testing may not be quite as prominent in docs & CLI tooling as the other frameworks but certainly has been thought about.

Pros Less tooling hurdle: Angular2 offers more opinions out of the box, which helps developers get started more quickly without feeling overwhelmed by decisions. This enforces consistency.

Simple yet more powerful than Angular1: in Angular 2 everything is encapsulated in a Component

Mobile first: it is designed from the ground-up for mobile and optimised for memory efficiency and less CPU cycles. Also better support for touch events and gestures.

TypeScript:

‘Stability without stagnation’. A more considered and developer-friendly approach to long term framework evolution than e.g. Angular.

Ember 1 is a viable choice for full JS support in IE8.

Ember Data seems more advanced than other frameworks’ model approaches and having the client ‘know’ more about our data is likely to ease smarter validation & form handling.

Favouring convention over configuration, especially re. where each file lives, means it’s easy to get things working while following a standardised, logical project structure. This keeps code cleaner, and makes it much quicker to get started on a project for anyone who’s worked with Ember before.

Enough big corporate users (+ Apple apparently!) that development is unlikely to ever stagnate without a viable fork appearing. LinkedIn have contributed back their own tools for working with it.

Takes a more complete approach than e.g. React, doing more for you along the way – while also being reasonably simple to get started with by having conventions that handle most parts of an app automatically.

Really good CLI tool which includes running tests + JSHint and (with an addon) compiling SASS/ LESS/ Compass styles.

React 0.14 is a viable choice for full JS support in IE8.

One-way data binging to keeps UI consistent and make data flow easy to follow.

Probably the fastest option on aggregate right now in terms of overall draw / update performance. Likely to be among the fastest for the foreseeable future.

JSX: an optional tree structure that’s fast, provides XSS protection, looks familiar to web developers from other technologies & is ‘good for you‘, separating concerns by module rather than language.

Smallest footprint by far, including Redux.

Cons Angular 2 currently with version 2.0.0-beta.15 and official release date yet unknown. The development of Angular 2 Universal (server-side rendering) is also unclear.

Lack of documentation and addons in general because it’s new and still in development phase.

Steep learning curve

Using Ember 1 today could leave us unable to use an increasing number of addons. Using Ember 2 gives us no easy route to IE8 fallback support, since FastBoot is not advanced enough to be a full solution.

Ember Data’s cleverness also makes it a bit more complicated to understand at first.

The whole framework is also much bigger than e.g. React, so there are far more concepts to learn overall.

Smaller scope and ‘conceptual footprint’ also means we have more responsibility for the design & handling of our data structure, and considerations like securing our JS against XSS.

For PHP micro-frameworks we chose SLIM 3. Zend Expressive, and Silex were the other choices considered.

Zend Expressive Silex Slim 2 & 3
Pros Docs are already quite good

CLI tool makes swapping DI container / router etc. easy at install time

Has been designed with a middleware / PSR-7 focus from the start

One of the fastest micro RESTful PHP frameworks, arguably TOP 5

Built on Symfony components and modular, use only what you need

Good documentation plus 3rd party community support

Excellent for large projects ( Organization of controllers can be done in many ways for example )

Routes grouping and routes collection are excellent, giving the ability to have routes configured via YAML files. Ready for using to provide Restful service, as it provide JSON handling method for its views

Supporting middlewares

Most installs (2.4m vs 1.4m for Slim)

One of the fastest micro RESTful frameworks available – recent benchmarks suggest it beats Silex

Based on its name it’s really SLIM. Good documentation (it could be better), and then again and 3rd party community ( though it’s less than Silex)

Routes grouping is a great feature

Supporting middlewares – Slim 3 has been PSR-7-centric from the outset

Most GitHub stars (6.6k vs 3.1k for Silex), maybe most active development long-term?

Cons Very new compared to Slim & Silex – less mature, much smaller user base. 18k installs (vs millions), 300 GitHub stars.

Not included in most performance benchmarks I’ve seen for this reason – but AFAIK it’s not claiming to be the fastest

Dependencies are on Zend components – mostly less widely used than Silex & Slim’s choices

A bit heavier than Slim, as a result of doing slightly more (e.g. routing config) out the box

Slightly more tied to 1 bigger framework (Symfony) in its choices of dependencies

Missing the ability of route configurations make it very hard to to work with routes in case of large projects

(Slim 2 – now fixed) needs small tweaks to provide Restful service which may lead to a not very clean code

We have not been so lucky with some of our choices in some other areas and were limited by suitability with our current managed PaaS and hybrid architecture. For example, we used Gearman in the legacy application and moved to RabbitMQ (much better) as we had capability for it and had better TLS support. We will revisit such choices next year as we optimise and fine tune our system.

PaaS

The move to a PaaS was not an easy decision on top of the micro services re-architecture. We have traditionally been IaaS-based and we were quite comfortable with the control we had. However, PaaS gives more control over to developers and allows them to ensure their choices and stack and replicated all the way to production as they expect them to be. The decision was taken to use Pivotal CloudFoundry as our PaaS of choice primarily as our Donations team have been using CloudFoundry successfully for many years. When we realised it was currently complex to build a self hosted CloudFoundry on our existing infrastructure (as the supporting services on CloudFoundry were fast moving and had compatibility complexities), we decided to go for a managed environment on Pivotal CloudFoundry, who have also been generous in offering an almost-free service. The solution is great from an application developer’s perspective and it allows developers to have complete control over their application and dependencies. In our experience, CloudFoundry is still rapidly evolving, is a great PaaS in the making even if PHP was not the first language supported (the buildpack is under active development and we have had extensions added to it). However, we do not entirely plan to stick to PHP and will choose more appropriate languages of choice in our micro services as we develop the capability in-house.

We use Docker for our test environments and deploy to Pivotal PWS using Concourse CI pipelines. We also have some of our monolith service layers on AWS whilst we are currently in a hybrid architecture; so in a way we use a Multicloud setup, which will continue to evolve as we progress into 2017.

Continuous Delivery

While moving to a hybrid microservice architecture, we improved our CI system to be able to test and deliver applications faster and easier, independent of each other. We moved all our CI tasks from Jenkins to Concourse CI, which is an excellent systems using the concept of pipelines, nicely integrated with Cloud Foundry and docker, and managed via code.  

API Testing

As we plan to re-use several micro services for integrations with other services elsewhere, API testing formed a major part of testing these services as products themselves. Our main challenge so far has been choosing the right manual and automation testing software.

For automation testing, we currently use the PHP Behat Mink Framework as it also supports BDD practices. We also have an integration testing framework that will send requests to the target API and verify the responses from the API. We have incorporated these automation tests into the CI pipeline which will automatically be triggered for any commit to the API. We are still in the early stages and should aim to induct more practices to test functionality to more granular levels.

It is also equally important to test the APIs manually. Currently the team is using Postman which provides an environment to write and run the tests for requests and responses. We are still evolving the testing framework and aim to get granular and accurate whilst increasing coverage in a nimble way.

Team

In saying everything about the move to a whole different architecture and set of practices, none of these would have been possible if the team were not willing and engaged from the start. Micro services is not just a change of technology and architecture, but also a change to the associated processes, team make-up, and culture. None of this will be possible without a great set of developers and a supporting team. Kudos for their adaptability, skill and patience.

Product

Finally, the product we are talking about that we started migrating to micro services is already live for Red Nose Day 2017 https://my.rednoseday.com/user/login/. It is far from complete as it stands at the time of this writing, and in the spirit of being Agile, we release as early and often as we can.

Future

The world of APIs have opened up all sorts of integration possibilities and innovative ideas. We are also equally excited about Serverless architecture and the power this whole landscape provides. We have tons of data to share and analyse, all of which are now made possible with the new architecture. We just have to catch-up now to build and secure our APIs and open it to other teams to integrate and use.

Challenges

As mentioned earlier, we sometimes feel we are in uncharted territory as a PHP app on CloudFoundry, and have uncovered a few challenges that we are working towards resolving:

Monitoring and status tools

As of this writing we have not found a set of monitoring tools that cover our use cases entirely and allow us to control all our services easily.  The existing Sleuth and Zipkin implementations in CloudFoundry are what we are using predominantly as of now. There is the potential to complement them with the NewRelic service as well. However, we do intend to expand on their capabilities to cover everything we want to track.

Infrastructure monitoring

Whilst this goes against the managed PaaS proposition, we see a current need in our evolution to have a grasp of load balancer stats around number of requests queued (app saturation), peak inbound and outbound simultaneous connections, and max errors from the backend. However most of Pivotal CloudFoundry metrics available now is application-level focused and does not help with external dependency checks as well, which for us is required as we have part of the application running on AWS and need health checks on them as well. Adding infrastructure-level information is in the long term future goals.

Custom health checks

Deep application level health checks is something the team have been used to and would like to have, being risk averse. This would help catch false positives so as to stop production deployments, and Night of TV monitoring so that we catch any issues that we may have missed testing.

MySQL

PWS do not have a good MySQL offering so we would continue to use AWS RDS.

Logging

Regardless of your choice of cloud provider, moving to distributed microservices makes reliable centralised logging more important than ever. We spent some time comparing managed and self-hosted services for this, and decided that a paid hosted solution was well worth the investment to get the insight and monitoring that we need for these apps.

Building & securing new services

Moving to distributed apps that cannot rely on private networking creates some challenges in terms of the services we need, and making sure that sensitive data never leaves a private network unencrypted. As our developers aren’t experienced with provisioning these, we had to reach out to our busy WebOps team to get some services like RabbitMQ set up on AWS in a way that runs quickly and securely with the new infrastructure.

With the dev team – Noel Light-Hilary, Ayan Ozturk, John Levermore, Venu Botla, Marco Maglio, Jessie Wang, Ahmed Raschad, and Mohammed Labib

Written by Girish Nair

Head of Engineering at Comic Relief. I lead a team of talented engineers who take pride in creating quality software of value.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s