One of the goals we set ourselves for 2016 was the re-architecture of our Fundraising & registrations platform and moving to micro services. The team has been busy on that this year and the work will continue well into 2017 too. Here is a small rundown of our journey so far.
Our previous monolith was Zend 1 based and it was getting harder to support, upgrade, extend and secure easily. Technical debt had crept in over 4 years and it was getting harder to separate the logic between different layers. A simple case was when we wanted to separate the presentation layer from the service layer so as to plug and play newer front end tools, and also upgrade one layer without affecting the other.
From a mere maintenance perspective, it was getting harder to upgrade the PHP version of the application, use other open source libraries and also improve the base framework without spending too many cycles testing and reworking, which stagnated innovation. Over time, all our practices and processes had also got centered around the nuances of our old platform and basically we were walking backwards in time. Our speed of delivery was getting affected and deployment processes were running the risk of being insecure.
It was time that we created a system which was extensible, scalable, easy to maintain, and fun to work with. Micro services fit that bill.
Starting with a clean slate, we were able to revisit everything from our methodologies, to ways of working and inherent team structure as well. We had a relatively new project team on board but they had a grasp of the existing monolithic application to be able to determine what areas needed to be separated out. We also matched this separation with long term goals of re-use and repurposing some the layers for other purposes. It also opened up exciting areas of innovation based on the APIs we were inherently creating as part of these micro services.
With all the possibility of available choices for micro services and processes, we decided to pace ourselves and settled on having a hybrid architecture first before completely moving over to micro services so as to meet a few demands of our ongoing campaigns. Our team size is basically a core strength of 3 devs, and yes that does not necessarily fit Conway’s Law for an ideal micro services architecture but it fits the bill for now and as we involve more teams, we expect the ideal structure to take shape as well.
We began by setting ourselves deadlines for settling on our technology choices for each micro service. We prototyped, discussed, and documented the various choices as a team, before making a decision. We won’t go into the detail of the decision making process in this blog post for each layer, although will cover some of the main technologies chosen. Hopefully we can delve into details in future blog posts.
For the presentation layer we debated between Angular, Ember, and React.JS. We chose React.JS and set out to build an isomorphic application. Here are our notes from the comparison exercise we did in April 2016.
|Angular 2.x||Ember 1 or 2||React 15 or 0.14|
|Browser support||All modern & IE9+||Ember 2: all modern & IE9+
Ember 1: all modern & IE8+
|React 15: all modern & IE9+
React 0.14: all modern & IE8+ with additional ES5 shims
|Community support||Seems really good. Things are well documented, and in my opinion the whole project has a stability-focused & mature approach to new versions that’s still unusual in the JS framework world.
As a result many Ember 2 features have been back-ported to Ember 1, making it still a completely viable choice today with comprehensive deprecation warnings and so a very smooth upgrade path to Ember 2.
Ember 2.4 is the project’s first Long-Term Support version, with a commitment to critical bug fixes for 18 months after its release.
|In wide use for a long time without drastic language changes.
Moved to semver this month (April) with version 15 and seem to be taking a similar approach to Ember with deprecations – “If your code is free of warnings when running under React 0.14, upgrading should be easy.”
Not sure how much has been ported back from 15 to 0.14 but it’s likely to still be at least as viable a choice as Ember 1.x. React 15 only came out very recently. Leaderboard prototype is on React 0.14.
|Projects stats||GitHub: 11k stars, 2.8k forks||GitHub: 16k stars, 3.4k forks.||GitHub: 41k stars, 6.8k forks|
|Security||Ember’s a stable project with a sensible disclosure policy for security issues.
Latest version logs console messages about the Content Security Policy, and you can lock down access to any external resources to ensure only trusted ones are loaded.
As the second link in the above § points out, Ember doesn’t enforce a sandbox policy for developer code. This doesn’t make us more vulnerable to XSS than e.g. jQuery does with our code today – it just means we’re not automatically better protected.
This presentation (slide 110) gives Ember more points out of the 3 for some security considerations, and awards it the highest ‘score’ overall. However this is a bit subjective, and Angular 2 might take a different approach e.g. on advisories once it’s stable.
|I’ve only found 1 historical CVE mentioned anywhere for React itself – an XSS issue from December 2013. It seems to have been disclosed responsibly and React core appears to have a good track record.
React takes an approach which in general strongly discourages unsafe operations that are likely to lead to XSS problems in apps – although not checking input types can still lead to XSS problems like this 2015 vulnerability disclosed to HackerOne.
|Performance||New(ish) Glimmer engine greatly improved rendering performance. It was released + back-ported to Ember 1.13 in June 2015.
It’s still slower than React in some respects but quicker in layout & paint tests. There are some detailed benchmarks & explanations in this article.
Right now there still seem to be some Android performance problems & Discourse have built a custom renderer wrapping virtual-dom to get round it. But some comments are suggesting that ‘Glimmer 2’ (which should be in stable soon) will improve this greatly for Ember 2 anyway.
LinkedIn have developed & shared a Chrome extension for analysing & improving Ember apps’ performance.
React creates an in-memory data structure cache (Virtual DOM), computes the resulting differences, and then updates the browser’s displayed DOM efficiently.
React also provide performance tool API called ReactPerf, it is a profiling tool to overview app’s performance.
|Server side rendering||Angular Universal doesn’t seem to be attempting client-JS-free rendering currently (even in the starter example) or doing form processing. Seems unfinished & feature light even compared to e.g. Ember FastBoot.||More limited than I’d hoped. FastBoot only works with Ember 2.3+, and doesn’t yet cover form processing. On its own it’s not a complete solution for stripped back IE8 (or non-JS) support – and probably still won’t be in the next few months.||While React doesn’t try to do anything like this on its own, addons & modules to support universal JS (e.g. within Redux) and form processing (e.g. newforms) seem relatively mature and usable compared to the other frameworks considered.
Getting this working fully for our requirements may be a bit involved, but should definitely be possible.
|Accessibility||Angular 1 had ngAria – A2 is providing sensible ARIA behaviour built in, based on this (see surrounding slides).||Notes & commits on this GitHub issue detailed some steps we can take to make an Ember app as accessible as possible, including determining which containers are focusable and announcing route / title changes.
a11y-testing addon can help catch some common a11y issues in templates – potentially making it easier to ensure a good screen reader UX than when using plain HTML, and making maintaining this standard a part of acceptance tests.
|React uses custom html attribute to render web accessibility attribute for components. Additionally, react-a11y addon can scan react elements and warns about potential accessibility issues.
Looks like react-a11y can also be quite easily tied into acceptance tests, similar to what the Ember module does by default.
React Native also provide Accessibility API to enable support for iOS and Android.
|Testing||Angular2 favours jasmine+karma for unit testing and protractor for end to end testing. Quite a few articles like this one and this one explain how it works and provide step-to-step guide.||Ember CLI builds a project structure and tool chain that gives you JsHint checks for free and makes unit & acceptance tests very easy and intuitive to add.
Having the data models separate in Ember Data also makes it easy to test specific, isolated aspects of their functionality. The relatively large number of pieces/ concepts does have some advantage in that each layer is independently testable.
|Seems like with Jest you have quite a simple starting point for functional / black-box testing of each component in isolation. There also seem to be a range of other viable approaches for component testing.
There are also useful notes on React testing approaches here.
Testing may not be quite as prominent in docs & CLI tooling as the other frameworks but certainly has been thought about.
|Pros||Less tooling hurdle: Angular2 offers more opinions out of the box, which helps developers get started more quickly without feeling overwhelmed by decisions. This enforces consistency.
Simple yet more powerful than Angular1: in Angular 2 everything is encapsulated in a Component
Mobile first: it is designed from the ground-up for mobile and optimised for memory efficiency and less CPU cycles. Also better support for touch events and gestures.
|‘Stability without stagnation’. A more considered and developer-friendly approach to long term framework evolution than e.g. Angular.
Ember 1 is a viable choice for full JS support in IE8.
Ember Data seems more advanced than other frameworks’ model approaches and having the client ‘know’ more about our data is likely to ease smarter validation & form handling.
Favouring convention over configuration, especially re. where each file lives, means it’s easy to get things working while following a standardised, logical project structure. This keeps code cleaner, and makes it much quicker to get started on a project for anyone who’s worked with Ember before.
Takes a more complete approach than e.g. React, doing more for you along the way – while also being reasonably simple to get started with by having conventions that handle most parts of an app automatically.
Really good CLI tool which includes running tests + JSHint and (with an addon) compiling SASS/ LESS/ Compass styles.
|React 0.14 is a viable choice for full JS support in IE8.
One-way data binging to keeps UI consistent and make data flow easy to follow.
Probably the fastest option on aggregate right now in terms of overall draw / update performance. Likely to be among the fastest for the foreseeable future.
JSX: an optional tree structure that’s fast, provides XSS protection, looks familiar to web developers from other technologies & is ‘good for you‘, separating concerns by module rather than language.
Smallest footprint by far, including Redux.
|Cons||Angular 2 currently with version 2.0.0-beta.15 and official release date yet unknown. The development of Angular 2 Universal (server-side rendering) is also unclear.
Lack of documentation and addons in general because it’s new and still in development phase.
Steep learning curve
|Using Ember 1 today could leave us unable to use an increasing number of addons. Using Ember 2 gives us no easy route to IE8 fallback support, since FastBoot is not advanced enough to be a full solution.
Ember Data’s cleverness also makes it a bit more complicated to understand at first.
The whole framework is also much bigger than e.g. React, so there are far more concepts to learn overall.
|Smaller scope and ‘conceptual footprint’ also means we have more responsibility for the design & handling of our data structure, and considerations like securing our JS against XSS.|
For PHP micro-frameworks we chose SLIM 3. Zend Expressive, and Silex were the other choices considered.
|Zend Expressive||Silex||Slim 2 & 3|
|Pros||Docs are already quite good
CLI tool makes swapping DI container / router etc. easy at install time
Has been designed with a middleware / PSR-7 focus from the start
|One of the fastest micro RESTful PHP frameworks, arguably TOP 5
Built on Symfony components and modular, use only what you need
Good documentation plus 3rd party community support
Excellent for large projects ( Organization of controllers can be done in many ways for example )
Routes grouping and routes collection are excellent, giving the ability to have routes configured via YAML files. Ready for using to provide Restful service, as it provide JSON handling method for its views
Most installs (2.4m vs 1.4m for Slim)
|One of the fastest micro RESTful frameworks available – recent benchmarks suggest it beats Silex
Based on its name it’s really SLIM. Good documentation (it could be better), and then again and 3rd party community ( though it’s less than Silex)
Routes grouping is a great feature
Supporting middlewares – Slim 3 has been PSR-7-centric from the outset
Most GitHub stars (6.6k vs 3.1k for Silex), maybe most active development long-term?
|Cons||Very new compared to Slim & Silex – less mature, much smaller user base. 18k installs (vs millions), 300 GitHub stars.
Not included in most performance benchmarks I’ve seen for this reason – but AFAIK it’s not claiming to be the fastest
Dependencies are on Zend components – mostly less widely used than Silex & Slim’s choices
|A bit heavier than Slim, as a result of doing slightly more (e.g. routing config) out the box
Slightly more tied to 1 bigger framework (Symfony) in its choices of dependencies
|Missing the ability of route configurations make it very hard to to work with routes in case of large projects
(Slim 2 – now fixed) needs small tweaks to provide Restful service which may lead to a not very clean code
We have not been so lucky with some of our choices in some other areas and were limited by suitability with our current managed PaaS and hybrid architecture. For example, we used Gearman in the legacy application and moved to RabbitMQ (much better) as we had capability for it and had better TLS support. We will revisit such choices next year as we optimise and fine tune our system.
The move to a PaaS was not an easy decision on top of the micro services re-architecture. We have traditionally been IaaS-based and we were quite comfortable with the control we had. However, PaaS gives more control over to developers and allows them to ensure their choices and stack and replicated all the way to production as they expect them to be. The decision was taken to use Pivotal CloudFoundry as our PaaS of choice primarily as our Donations team have been using CloudFoundry successfully for many years. When we realised it was currently complex to build a self hosted CloudFoundry on our existing infrastructure (as the supporting services on CloudFoundry were fast moving and had compatibility complexities), we decided to go for a managed environment on Pivotal CloudFoundry, who have also been generous in offering an almost-free service. The solution is great from an application developer’s perspective and it allows developers to have complete control over their application and dependencies. In our experience, CloudFoundry is still rapidly evolving, is a great PaaS in the making even if PHP was not the first language supported (the buildpack is under active development and we have had extensions added to it). However, we do not entirely plan to stick to PHP and will choose more appropriate languages of choice in our micro services as we develop the capability in-house.
We use Docker for our test environments and deploy to Pivotal PWS using Concourse CI pipelines. We also have some of our monolith service layers on AWS whilst we are currently in a hybrid architecture; so in a way we use a Multicloud setup, which will continue to evolve as we progress into 2017.
While moving to a hybrid microservice architecture, we improved our CI system to be able to test and deliver applications faster and easier, independent of each other. We moved all our CI tasks from Jenkins to Concourse CI, which is an excellent systems using the concept of pipelines, nicely integrated with Cloud Foundry and docker, and managed via code.
As we plan to re-use several micro services for integrations with other services elsewhere, API testing formed a major part of testing these services as products themselves. Our main challenge so far has been choosing the right manual and automation testing software.
For automation testing, we currently use the PHP Behat Mink Framework as it also supports BDD practices. We also have an integration testing framework that will send requests to the target API and verify the responses from the API. We have incorporated these automation tests into the CI pipeline which will automatically be triggered for any commit to the API. We are still in the early stages and should aim to induct more practices to test functionality to more granular levels.
It is also equally important to test the APIs manually. Currently the team is using Postman which provides an environment to write and run the tests for requests and responses. We are still evolving the testing framework and aim to get granular and accurate whilst increasing coverage in a nimble way.
In saying everything about the move to a whole different architecture and set of practices, none of these would have been possible if the team were not willing and engaged from the start. Micro services is not just a change of technology and architecture, but also a change to the associated processes, team make-up, and culture. None of this will be possible without a great set of developers and a supporting team. Kudos for their adaptability, skill and patience.
Finally, the product we are talking about that we started migrating to micro services is already live for Red Nose Day 2017 https://my.rednoseday.com/user/login/. It is far from complete as it stands at the time of this writing, and in the spirit of being Agile, we release as early and often as we can.
The world of APIs have opened up all sorts of integration possibilities and innovative ideas. We are also equally excited about Serverless architecture and the power this whole landscape provides. We have tons of data to share and analyse, all of which are now made possible with the new architecture. We just have to catch-up now to build and secure our APIs and open it to other teams to integrate and use.
As mentioned earlier, we sometimes feel we are in uncharted territory as a PHP app on CloudFoundry, and have uncovered a few challenges that we are working towards resolving:
Monitoring and status tools
As of this writing we have not found a set of monitoring tools that cover our use cases entirely and allow us to control all our services easily. The existing Sleuth and Zipkin implementations in CloudFoundry are what we are using predominantly as of now. There is the potential to complement them with the NewRelic service as well. However, we do intend to expand on their capabilities to cover everything we want to track.
Whilst this goes against the managed PaaS proposition, we see a current need in our evolution to have a grasp of load balancer stats around number of requests queued (app saturation), peak inbound and outbound simultaneous connections, and max errors from the backend. However most of Pivotal CloudFoundry metrics available now is application-level focused and does not help with external dependency checks as well, which for us is required as we have part of the application running on AWS and need health checks on them as well. Adding infrastructure-level information is in the long term future goals.
Custom health checks
Deep application level health checks is something the team have been used to and would like to have, being risk averse. This would help catch false positives so as to stop production deployments, and Night of TV monitoring so that we catch any issues that we may have missed testing.
PWS do not have a good MySQL offering so we would continue to use AWS RDS.
Regardless of your choice of cloud provider, moving to distributed microservices makes reliable centralised logging more important than ever. We spent some time comparing managed and self-hosted services for this, and decided that a paid hosted solution was well worth the investment to get the insight and monitoring that we need for these apps.
Building & securing new services
Moving to distributed apps that cannot rely on private networking creates some challenges in terms of the services we need, and making sure that sensitive data never leaves a private network unencrypted. As our developers aren’t experienced with provisioning these, we had to reach out to our busy WebOps team to get some services like RabbitMQ set up on AWS in a way that runs quickly and securely with the new infrastructure.
With the dev team – Noel Light-Hilary, Ayan Ozturk, John Levermore, Venu Botla, Marco Maglio, Jessie Wang, Ahmed Raschad, and Mohammed Labib