Engineers, engineers, engineers

Today and tomorrow on the 22-23 of November Bonnier Broadcasting will attend the engineering career fair THS Armada at KTH Royal Institute of Technology

We will talk to engineering students about how we are building the next generation TV-experience on TV4 and C More through our tech development.

Ask us questions and share your views on our technologies and platforms. Talk to us about our cloud platforms, programming languages and how our engineers build our streaming services TV4 and C More on one shared tech platform.

We will also have fun quizzes with nice prices and some great cupcakes that you definitely want to taste.

Also students will have an opportunity to refine our suggestions or pitch own ideas for exciting master thesis projects next year at Bonnier Broadcasting.

Test-Driven Development is not about testing

An early morning last month (October 2016) a mix of software developers from several teams of Bonnier Broadcasting gathered in the cosy offices of Agical in Gamla Stan to spend a full day listening to the wise words of Mr. J. B. Rainsberger on the topic of TDD — Test-Driven Development.

Here follows an attempt to distil some of my reflections and take-aways from the event in the form of a combined summary and short intro to the topic.

What is a test; what is TDD?

As developers we are human, and as humans, we err. Writing tests is a way for us to eliminate, or at least reduce, errors by ensuring that what we build functions in the way we expect. If we write code that can add two numbers, we might write a test that asserts that given one plus two, we get three.

TDD takes the idea of testing a step further by in essence letting the test define correct behaviour, as opposed to verifying correct behaviour after the fact. Writing tests early on turns out to have positive effects not only on the correctness of our code, but also its design, with reduced coupling and better separation of concerns — perhaps this is the greatest benefit of TDD.

Take-aways

Many things were covered during the event and it would have been odd if everyone went home with the exact same insights. Here are just a couple of take-aways that struck me as having particular value.

Design

Design is hard. Done well it brings many benefits and can be intellectually rewarding. Done wrong it can be the cause of high costs both financially and in the form of lowered productivity and motivation.

TDD is not a silver bullet that magically solves all design problems, and I will not claim that TDD is essential to designing things well, but it is a valuable tool that if used right allows us to find design mistakes early so that they may be corrected, before they become expensive. Rainsberger relates to the concept of risk exposure.

E = ∑([pm * cm, …])

Exposure is the sum of the probability of each possible mistake multiplied by the cost incurred should it happen.

To reduce exposure we may choose to reduce the probability of a mistake — in essence figuring out the potential mistake before it happens. If it is true that design is hard, then predicting future mistakes is not a trivial task.

Alternatively, we could assume that mistakes will be made and instead try to reduce their cost.

Code that is easy to test tends to be better designed, since — and one could almost get away with giving the inverse as the reason — well-designed code tends to be easier to test. It is like a virtuous circle, and also a concept within TDD; write a test, implement, test, refactor (improve the design), loop. This way TDD helps keep a clean design. If we only take care to watch for the tell-tales indicated by how straightforward and clear the tests are, it is easier to, at an early stage, spot potential flaws in our design.

Contracts

When designing we make certain assumptions. That code that adds two numbers might assume that its arguments are numbers. In a statically typed language this particular case is of course automatically enforced. In a dynamically typed language, it has to be assumed or checked. This is a precondition for the code to work correctly and essentially becomes part of its contract, whether explicitly documented or not.

Another example is the Read method as defined by the interface io.Reader of the Go standard library. Read is used to read bytes from a data source by copying it into a given buffer p. Its contract states “if some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more”. It is the responsibility of the caller to be aware of this behaviour.

Assumptions always exist about how the code we write will be used, and about how code we use behaves. A clear contract not only defines what can and cannot be expected, but it helps define to what extent the code should be tested. While some things are difficult to capture in a test (such as unexpected behaviour), writing tests can be a nice way to explicitly demonstrate what kind of usage we expect and support.

Unit tests vs. integration tests

A unit test serves to test a small part of a system in isolation.

A unit test:

  • …is isolated
  • …is simple
  • …is small
  • …runs fast
  • …is clear on what is being tested
  • …makes it easy to pinpoint the cause of specific failure
  • …drives design

An integration test serves to test how multiple layers of a system work together as a whole.

An integration test:

  • …tests many parts of the system as a whole
  • …might make calls to things outside of our control
  • …can fail in many different ways
  • …is slow

Side note: Rainsberger would correct the term “integration test” replacing it with “integraTED test”, defining it such that an integraTION test tests isolated points of contact between two parts, whereas an integraTED test is, quote, “any test whose result (pass or fail) depends on the correctness of the implementation of more than one piece of non-trivial behaviour“. While these definitions are clear in themselves, the use of two similar-sounding terms to mean different things tends to cause confusion (cf. authentication vs. authorization — both conveniently abbreviated ‘auth’), so I choose to simply walk around the definition swamp by using the term “integration test” to mean what Rainsberger calls “integrated test”, and for the time being avoid this other definition of “integration test”. How to name things: one of the hard problems in programming.

Rainsberger metaphorically compares the two methods of testing saying that the former is like painting a wall with a brush, while the latter is more like trying to cover all of it by throwing buckets of paint at it.

He describes the latter kind of tests as a scam and drew for us a vicious circle, illustrating that the more we start to rely on these tests, the more we lose the ability to iteratively improve our design, which in turn negatively affects the ability to write isolated unit tests, in turn resulting in even more end-to-end tests being added.

Perhaps one can extend the paint metaphor by saying that the more buckets of paint we throw at the wall, the harder it gets to move closer to it without getting our feet covered in paint.

Above being said, it should be pointed out that acceptance tests for the purpose of verifying behaviour from the end user’s point of view are perfectly fine, as long as they are not used as a substitute for tests which are usable in driving the design.

Conclusion

Even for someone already at least somewhat familiar with the idea of TDD I found it refreshing to during this event be able to reiterate on some of the concepts.

For the past two-or-so weeks we tried forcing ourselves to use TDD at one of our mob stations. In the context of mob programming it turned out to be quite useful in defining something of a road map of what to do next, and also as a means to keep focus on a particular task.

The most important insight that I gained from this day deserves its own line:

TDD is not about writing tests, it is about writing better code

Lastly, some things noteworthy.

  • – Prepare for design change from any direction.
  • – Not prepared for any particular change, but prepared for change
  • – Test integrations with external parties in isolation.
  • – Abstract external parties using interfaces, exposing only what is needed.
  • – Avoid mocking types that you don’t own.
  • – Do not pass nil to things intentionally.
  • – Complex mocks can be an indication of bad design.
  • – Make the documentation clear, entertaining, and small, with code examples.
  • – 100% test coverage does not necessarily mean that all logic is being tested.
  • – A worker should never talk directly to the client in multi-threaded code.
  • – Test the worker in isolation; it should not know whether or not it is being called asynchronously.
  • – Avoid large anonymous functions, because they are hard to test in isolation.

I shall also add that the above are my own interpretations, and as such do not necessarily correspond exactly to the ideas presented during the event.

Thank you for reading.

A platform for our platform

In June this year (2016), a new team was formed within Bonnier Broadcasting with a mission to build a realtime data platform. The purpose being to address a set of high priority data needs we have identified, but also to support the growing data driven culture we see in both C More and TV4. A realtime data platform has of course no formal definition, and it can mean slightly different things to different people. To us, it means an accessible data warehouse where all our data sources are consolidated and integrated. These data sources range from relatively static sources, such as user or video databases, to event streams from apps and services, ingested in realtime. From all these sources we need to create data models, analysis, and visualizations that help the rest of the organization do a better job.

So, when starting out on this journey, one of the first questions to arise was, where should we build this platform? What platform should we choose as the foundation for our platform? This post recounts the reasoning behind our choice, and our experiences with the result so far.

TL;DR

We went with Google Cloud Platform for our data platform and we are mostly happy with it.

A cloud platform

For the majority of our system components (infrastructure services, databases, video platform) we have already made the move to the cloud. Several years ago we started using AWS and Heroku (and we are still migrating legacy services). At the time there were not many options, and we are still happy with AWS for our core needs.

This year, however, when discussing our future data platform, the cloud landscape has evolved substantially from when we picked AWS, and we saw an opportunity to take a step back and evaluate our options. And there are a host of options these days, at various abstractions levels, and with various levels of managed offerings.

Apart from pure functional requirements, of which none is particularly unique, we have a few non-functional constraints that effectively narrow us down a bit.

First, we want as much control as possible. We want to avoid, or at least minimize, vendor lock-in and dependencies to third-party products and solutions. This rules out a number of packaged solutions, such as Cloudera, Databricks, MapR, and others.

Second, we are a small team with an ambitous task. We want to minimize, as much as possible, the cost of operation and maintenance that comes with doing too much ourselves. The most obvious path from there would perhaps be to go in the Apache direction, where Hadoop, Spark, and an abundance of other tools, await the willing, and set them them up on our AWS account. Much, if not all, of what we want to achieve has already been done with these tools, and many well-known companies have contributed to the toolbox over the years.

However, a third constraint is that we have picked Go to be our language of choice. This is a conscious choice, made from a number of reasons (which I would be happy to share in a separate post). As a consequence, we have invalidated (or at least made awkward to use) from our list of options the majority of open source data processing tools and frameworks, which are available to your everyday Java, or Scala based shop.

GCP

Google Cloud Platform (GCP) has been moving forward at tremendous speed during the last year(s) to catch up with AWS, trying to establish itself as a serious contender in the cloud market. And with Spotify announcing early that they are migrating their data processing platform to GCP, the risk associated with making the same move was substantially lowered.

As it turns out, what Google Cloud wants to be matches our needs and constraints to a very large extent. Most, if not all, of the infrastruture we need is there. The following four components are what we use the most today.

  • BigQuery is the managed data warehouse we needed but did not find in Redshift.
  • Container Engine (GKE) is the managed Kubernetes platform we don’t have to waste our own time on setting up and maintaining.
  • Dataflow is the managed data processing service that automatically runs and scales our stream and batch jobs, that we write with Apache Beam/Dataflow.
  • PubSub is the managed message bus that we use to pass data between the services above.

A key word here is managed, and it translates to a huge amount of time and effort that our small team does not have to spend on stuff (e.g. setting up, configure, monitor) that is not pushing our data platform forward.

Many people I talk to seem to look at GCP as simply a Google version (albeit less mature) of AWS or Azure, that aside from subjective bias and pricing, they have more or less the same offering. Having started out with AWS for our platform, we have found this assumption to be false, at least when it comes to our use case. The managed aspect of the GCP services is a differentiator.

What about Go?

The third constraint I mentioned above was the Go language. These days, the JVM is the ruler of data engineering, and to a large extent data science as well. As a consequence, we are not completely free from the JVM dependency, as much as we want it. What’s holding us back is Beam/Dataflow, where the stream processing SDK is still only available in Java. The Python SDK is moving along, however, and there are rumors that a Go SDK is in the works somewhere inside Google. Hopefully we can contribute to that once it is released to the community.

However, since all Google Cloud API’s have official Go client libraries (of which most are hand crafted and idiomatic, and not generated from spec), we can do most of our plumbing and infrastructure in the language we feel gives us most speed and quality.

The flip side

When I earlier wrote about what Google Cloud wants to be, I was hinting at the major downside we have experienced with our choice of cloud platform. Several must-have’s for us in terms of functionality are still in beta, or even alpha, and one or two GA components are still lacking (I’m looking at you, Stackdriver). Each component is moving at a very high speed, and one shouldn’t invest too much effort in trying to work around a current limitation, as it may well be solved two months later.

So, while moving in the right direction at high speed generally is perceived as a good thing, it is also frustrating when you try to build something real on top of it. We wouldn’t want it the opposite way though, and all in all we are happy with the choice we made.

Flash och Silverlight ersätts av ny HTML5 spelare

2010 skrev Steve Jobs ett öppet brev till Adobe där han dömde ut framtiden för att spela video i Flash. Problem då som nu är alla säkerhetshål som funnits i produkten genom åren. Säkerhetsproblemen i Adobe Flash har vuxit med tiden och var som högst förra året. Det har tagit nästa sex år för verkligheten att komma ikapp den visionära Steve Jobs, men nu händer det överallt.
Säkerhetsproblemen i Adobe Flash

Från och med Chrome 55, som beräknas släppas i december, så kommer användare bli tvungen att aktivt aktivera flash i browsern. Redan i Augusti så började Firefox blocka visst Flash-innehåll i  browsern. Och i den senaste OS versionen från Apple, MacOS Sierra, så kräver Safari att du aktivt godkänner att du vill se innehåll i Flash och att du känner till de säkerhetsproblem som finns med Flash. Microsoft rör sig även som i samma riktning med deras nya Edge browser.

I vårt OVP team, inom teknikavdelningen, bygger vi just nu nästa generations videospelare för både TV4 och C More. Tanken är att vi ska ersätta dagens gamla videospelare i Flash på TV4 Play och den gamla Silverlightspelaren på C More med en ny spelare byggd i HTML5. Flash för video slog igenom stort i och med lanseringen av Youtube 2005 och har sedan dess använts av bolag världen över för att spela video på internet. TV4 Play lanserade flash på TV4 2009, efter att länge använt Windows media. Vi ligger idag långt fram jämfört med många andra broadcasters runt om i världen som fortfarande är 100% beroende av Flash för att tjäna pengar på video.

Från ett användarperspektiv så är detta jättepositivt. Användarna av TV4Play och C More kan förvänta sig snabbare laddningstider, förbättra säkerhet och lägre batterianvändning. Att vi byggt allt internt och på en ny kodbas möjliggör även en snabbare utvecklingstakt framåt kring spelarfunktionalitet som exempelvis ad-insertion eller integrerad statistik. Här söker vi fler utvecklare som vill jobba med spelarutveckling på Android, IOS och HTML5.

Ett stort fokus för projektet har också varit att bryta ut spelaren från den övriga webbapplikationen. Genom att göra detta så kan samma spelare och funktionalitet användas på både TV4 Play och C More. Detta kommer ytterligare snabba på utvecklingen inom Bonnier Broadcasting och innebär att vi inte behöver lägga dubbla tiden på att bygga spelare separat för resp varumärke och sajt. Det innebär också att vi kan bygga upp en central spelarkompetens inom teknik.

Den nya HTML5-spelaren kommer att tas i bruk på C More under November månad och förhoppningsvis så kommer den sen att rullas ut på TV4 Play innan jul.

Streaming Contribution Hack

Idag deltog vi på Bonnier Broadcasting på Eyevinns Streaming Contribution Hackday 

Temat på hackdayen var streaminglösningar i största allmänhet och vi fick se några fina implementationer och idéer för psykadeliska 360/VR-upplevelser, tumnagelgenerering för bättre användarnytta och Mp4-dataextrahering för att leverera stabilare signaler. Det laborerades också kring streaming över WebRTC samt matchning av QR-koder i reklamströmmar.

Huvudsyftet var att bidra tillbaka till communityt och att bygga egna lösningar på existerande open source-projekt.

Tack Eyevinn för ett trevligt event!