8  Conclusion of part 1

We’re at the end of part 1, and I need to congratulate you for making it this far. If you took the time to digest what we’ve learned up until now, you should be ready for what’s coming, which should be a bit easier, at least some of the parts.

But before continuing, let’s quickly summarise what we’ve learned so far.

We started our journey with two scripts that download and analyse data about housing in Luxembourg. We then learned about tools and programming paradigms that we will now use in part 2 to make our scripts more robust:

In some ways, you might think that we’ve made our life unnecessarily complicated for very little gain. For example, functional programming seems to be only about putting restrictions on how you code. Same with using trunk-based development; why make it so restrictive?

What you need to understand is that these restrictions actually play a role. They force us to work in a much more structured way, which then ensures that our projects will be well-managed and ultimately reproducible. So while these techniques come with a cost, the benefits are far greater.

We will start part 2 by rewriting our scripts using what we’ve learned, and then, we will think about approaching the core problem differently, and structuring our project not as a series of scripts (or R Markdown files in the case of literate programming) but instead as a pipeline. Because until now, there’s still no pipeline and let me remind you that this book has the word “pipeline” in its title.

We will also learn about tools that capture the computational environment that was used to set up this pipeline and how to use them effectively to make sure that our project is reproducible.