Lessons learned self-hosting Renovate
As I wrote a few weeks ago I'm a big fan of Renovate. I've been using Renovate for ~5 years as a user and as an operator, using a variety of hosted and self-hosted options, leading me to being seen as "the Renovate person".
Through this experience, I've learned a lot about Renovate and some tips to make operating it that little bit better.
My experiences with self-hosting Renovate
It may be useful to understand how I've been using Renovate over the years, before we start getting into what I've learned.
I was first introduced to it while at Capital One, where we used the Mend SAAS Platform for Software Composition Analysis (SCA), largely focussed on license and security remediation.
As part of Mend's platform, it also had an offering of Renovate, which we started trying out after one of the UK teams started shouting out about how it was improving a number of things for their software maintenance. At the time, as an organisation we were largely only updating dependencies when there were security issues, although there were pockets of senior engineers who tried to keep some things up-to-date, it was much more difficult when you're doing it manually!
One of the things that we found super useful with this offering was how fast it seemed.
We were running a self-hosted GitHub Enterprise instance, so my team had several GitHub organisations for our various applications. The speed was clear when we would create an <org>/renovate-config repo to start setting up presets, and in the time it takes to refresh the page, Renovate had already pushed an initial commit to the repo with a base configuration and README.
We continued to see this, for instance Renovate rebasing PRs immediately when asked or on other merges to default branches, and definitely helped make it a great experience, on top of some tips I wrote about having helped my team.
Around the same time, I decided that I wanted Renovate for my Open Source projects on GitLab.com, which I set up through the official Renovate runner pipeline. This offering didn't give me the immediate PR feedback that I'd gotten used to with the setup we had at Capital One, but I was more than happy with it, and wasn't exactly speeding through all the maintenance updates, so it wasn't holding me back at all.
Not long after I joined Deliveroo, we were starting to roll out Dependabot for our dependency updates. After years of using Renovate, I was finding it a little painful, especially having to hand-craft dependabot.ymls across each of our repos and then keeping them up-to-date.
After discussing with members of the team, I put together a proof of concept using Renovate's GitHub Action as a low-effort way to start getting some feedback from the team on the benefits and explore the developer experience improvements. As we started to want to roll it out a little more in our repos, and started to feel the pain of not having the PR feedback, especially compared to what we had with Dependabot, I found the (commercial, but free) Mend Renovate On-Prem (now Mend Renovate Community Edition). I spent some time getting it set up and then we started onboarding the whole organisation onto it, improving the ability to keep things up-to-date so much more easily.
The Community Edition works differently to the purely Open Source offerings by bundling in webhook processing, so you can get Renovate performing real-time actions, rather than waiting for the next schedule.
Off the back of using and enjoying the commercial offering, in August last year I moved to self-hosting Mend Renovate Community Edition on Fly.io for my personal projects, instead of using the GitLab CI-driven setup.
At Elastic, we've been using the free Mend Renovate SAAS for a good while (prior to me joining) and have been finding it works really well. However, part of the reason I'm writing this post is that we're considering self-hosted Mend Renovate Community Edition and I wanted to share some balanced views on it. We're considering moving away for some additional control that we can't get with the hosted version, at least right now.
Over the last few years I've been responsible for operating + supporting Renovate for organisations with ~2000 repos, of which include several large monorepos that get hundreds of changes merged per week (and maybe more often than that!).
My recommended journey
Let's say that you've discovered Renovate, and you're looking at how best to get started with it for your organisation.
I'd recommend that you take it step-by-step, moving up to the next stage if you feel the need. You're also welcome to skip some stages and move to an end state if you'd prefer to cut out some of the work.
If you can, start with the free hosted app by Mend, which allows you to focus on providing the value of Renovate to teams, instead of managing infrastructure and getting teams bought into it.
Start with GitLab CI/GitHub Actions
At this early stage, you may not be able to get approval to use the hosted version or to spin up the infrastructure associated with self-hosted Renovate, so instead we can start with using the Open Source project to gain buy-in.
Depending on what your source forge is, you may be able to use a pre-built setup like the GitHub Action or GitLab CI pipeline, which makes it straightforward to run Renovate and wire in some platform-specific settings.
Start by setting it up to work on a schedule, for instance once per weekday. This will make it a little clearer that there's not expected to be immediate feedback on PRs, but still allow you to get updates performed fairly often. Then, if you're finding this is still not ideal, trying to schedule it so it runs at i.e. 0800, 1200 and 1600 should allow for a few opportunities in the day for Renovate to update existing PRs, and allow teams to feel like things are still progressing.
When first rolling this out at Deliveroo, we only had ~30 of ~2000 repos onboarded to Renovate, which made Renovate understandably slow as it needed to scan each repo to work out if the repo was onboarded, and if so process it. For some of our very large monorepos, this took a long time to clone, and then end up throwing that data away.
After discussions upstream, I found that the optimizeForDisabled configuration would alleviate this performance impact, and so I'd recommend you switch it on too, especially until you get to having more repos onboarded than not.
Something you may also find is that, naturally, some repos are going to take longer to process. It's worth investigating how long the execution takes across your repos, and considering doing an up-front discovery of repositories and then passing them to Renovate. This allows you to also pre-select repos that have a Renovate configuration, and can be used to split them into buckets i.e. repos that start with [a-e] or splitting the large monorepos from the rest of your repos.
It's also worthwhile having the ability to schedule ad-hoc rebuilds, in the case that a scheduled run has failed, or if you've suddenly got a lot of things you want rescanned and don't want to wait until the next window.
Soon, you'll want faster feedback
My experience is that as you have more folks using Renovate on a schedule, they'll start to feel like it's very "slow" to get MRs merged or updated, because there's no immediate feedback and i.e. asking for Renovate to rebase a MR requires you waiting until the next scheduled run, which can be a pain.
Now, this isn't always true - I've spoken to a couple of people who have never run the Community Edition, which includes real-time processing using webhooks, as their organisations just haven't felt like they need it, which is cool to hear.
But if you are feeling this, you may want to move to the Community Edition, or the Hosted app, especially if you've now got more buy-in from the wider engineering organisation around how useful Renovate is.
As noted in my post about setting up the Community Edition on Fly.io, there's actually very little you need to do other than pulling the upstream Docker image and wiring in some environment variables, which is a pretty nice experience!
However, it's worth reading the lessons below before you get started, as it'll help gauge what additional things may be missing from the quick start guide.
Hosted
Alternatively, if you're happy with an external app being able to read your repositories, and possibly also reading your private packages, then I'd very much recommend the hosted app, especially as it's free 🤑
But this doesn't work if:
- You run anything non-GitHub.com
- You want to select repos to onboard on a case-by-case basis, as GitHub has a limit of 75 installations of an app - the alternative is to install the app on the whole org
- You want to use a custom Fine-Grained Personal Access Token to allow access at a very granular level
Lessons learned
Hopefully that gives you a good basis for my experience, and how I'd recommend to get started with self-hosting. But what about some things to watch out for when you're self hosting?
You may want a beefier machine
As noted in the docs, the Community Edition only scales vertically, not horizontally. That means you'll need to throw more CPU and memory at the single running instance to get it to operate under more load, or with larger repos.
There is the ability to split into more high-availability mode with the Enterprise Edition, but I've not (yet?) been able to give that a go, so can only talk about my experience with the Community Edition.
I know one thing I looked at in the past - which I didn't get round to doing - was to split the webhook-handling server from the schedule-running server. This would at least allow them to operate independently of each other and possibly reduce the overhead on all jobs when the schedule is running, but unsure if it'd help much.
Therefore, consider the resources you're allocating to Renovate, and keep an eye on your metrics. For my app on Fly.io, I've got 1024MB and 1 vCPU, which kinda works. But I'm also the only user, and am happy when it crashes.
Be careful about disk space
Speaking of keeping an eye on your metrics, you really need to watch out for disk usage!
Renovate will generally operate within /tmp/renovate directory (partially tuned by the cacheDir configuration, and tries to clean itself up as well as it can.
However, Renovate can only manage things it knows about, and because it'll call out to go, gradle and npm (to name a few) which store copies of dependencies in different places, you may have other locations on disk that are slowly filling up to the point that you're then unable to execute any jobs.
A workaround for this is to make sure that these tools move to specific locations in /tmp, and you attach external storage for the /tmp directory to reduce the risk of disk space issues.
For instance Go will default the GOPATH to ~/go (which in the container usually becomes /go). After processing many repositories, the GOPATH will get quite large, and then lead to the disk running out of space. One option to manage this better is to set the GOPATH:
# via https://gitlab.com/tanna.dev/renovate-runner/-/blob/9d2a6734935329a1c43668f28869477aeb0f382d/Dockerfile#L14
ENV GOPATH=/tmp/renovate/go
A slightly hacky solution to keep on top of disk space is to use the allowedPostUpgradeCommands and postUpgradeTasks configurations to create a script that runs - after each and every Renovate upgrade, so make sure it's fast! - to check whether a directory is getting too large and if so, clean it out.
Doing it with postUpgradeTasks means it shouldn't break any ongoing jobs, which may happen if you're periodically cleaning up the space in the background i.e. using a systemd timer.
Something you may also notice is that a lot of disk space is used just from what's already on the image - but as I found when I asked this, that's expected.
Understand how teams are using it
If you're running Renovate for your organisation, you're not just responsible for the infrastructure but also helping teams make the most of it. It can be really important to understand how teams are using Renovate, to see if there's anything the platform is doing that could be improved, but also to help support them in their usage.
There's great official documentation around how to reduce the noise that teams may be perceiving with Renovate, as well as the "best practices" guide.
Understanding cases where teams are repeating the same configuration is absolutely something that should lead into organisation-wide shared presets, as well as presets for a specific team. A few weeks ago I wrote a tool that allows you to scan for all Renovate configurations, and pop them into an SQLite database which allows for further filtering. I've already found this very useful, and noticed some teams doing some interesting things that we can promote into our shared presets.
It can also be useful to see if you can visualise how teams interact with Renovate as well as feeding the data into your Internal Developer Portal/Platform to bring it to where teams are.
Wire in secrets for private packages
Renovate boasts support for a huge number of ecosystems already, as well as crafting additional support through regexes. But one thing it can't do out-of-the-box is reach into your private artifact repositories.
Fortunately, it does have support for authenticating to many artifact repositories so you can i.e. update all your private Docker images, Maven packages, etc.
This is something that can be a huge benefit of going self-hosted, as you're able to connect it to things that are either VPN'd off from the outside world, or just that you'd be apprehensive of allowing the hosted service connecting to.
I've found that it can be useful to take advantage of the fact that when baking in a config.js for Renovate to execute against your repos, you can either reference environment variables (avoiding putting secrets in source control) or even take advantage of the fact that it's just JavaScript and have some code that reaches out to i.e. Vault or AWS Secrets Manager as part of the startup process, and then expose those credentials to Renovate.
Avoid exposing the status endpoint?
In the Mend Renovate On-Prem, the precursor to Mend Renovate Community Edition, there was an HTTP endpoint that listed what was currently being processed, or was up next.
Because you're likely exposing the Renovate app to the world - so i.e. GitLab can send webhooks to it - you'd also be exposing that endpoint to the world, which would not be ideal if you didn't want to disclose the names of repositories you had in your organisation, which could be a security/privacy risk.
As of the Mend Renovate Community Edition, this now requires explicit enabling, and authentication, so it's no longer a concern.
Conclusion
I hope the above doesn't dissuade you from self-hosting Renovate, but helps you have a more informed decision.
I've absolutely loved the ability to be able to self-host it (unlike other options in the dependency update space) and love that it's also available as a hosted SAAS app that allows you to get started with it without needing to run anything yourself.
And finally, some of the additional benefits you get from self-hosting that you don't get with the SAAS app (as far as I'm aware) are:
- Ensure all teams use internal base options (soon coming to hosted app)
- Sync the Renovate data with i.e. Backstage, for instance using secustor/backstage-plugins built by one of the Renovate maintainers
- Have all teams onboarded to Renovate by default for i.e. Docker dependencies, but then allow them to onboard for the full configuration (via)
Let me know what you think!