Managing dependencies in Rust
Lots of diffs have been committed since we at ANIXE began to hang our hat on Rust - we're proud to say that the life(time) of aclr8, our first production-grade Rust web service, began on February 24, 2016, and it has since crunched countless tons of HTTP traffic.
Managing dependencies in Rust
At the moment, we've got a handful of Rust applications working on production 7 days a week, 24 hours a day - some in Kubernetes, some on bare-metal Ubuntu - and an even greater number of hand-made libraries that keep abstractions at bay, allowing us to focus on one piece of the entire system at a time.
Now, we're writing this article not to brag about the size of our Git repositories but rather to share - to share our experience working with Rust's package manager, Cargo.
Git vs Alternate registry
Way back in 2016, Cargo didn't support alternate registries (which were stabilized in https://blog.rust-lang.org/2019/04/11/Rust-1.34.0.html) - in order to re-use private code (parsers, test tools and so on), one had to either keep all their crates together or separate them into different Git repositories; we've gone with the latter:
Apart from the self-evident advantage of requiring little to no setup, it proved to be quite valuable in developing cross-project features: say, when we've got to touch one of our internal core libraries, there's no need to actually upgrade it in all of the downstream crates until the entire feature is complete and working as intended - simply because for the time being you can switch branch = from dev to JIRA-123/dev, which works both for developers and CI's pipelines.
Unfortunately, there are no silver bullets when it comes to dependency management: versioning such projects proved to be arduous and complex, especially as the number of our crates (and thus cross-crate dependencies) grew. Eventually, the dust settled as we befriended Git tags (created manually together with each production deployment) and agreed on one simple rule: applications located on the master branch must work with libraries from the master branches too, and ditto for dev.
While we retained a poor man's versioning, we did end up with a more swift development process - the only nitpicking keeping us awake at night is cross-crate dependency-tango:
(in this case app depends on lib-a & lib-b, and lib-a itself depends on lib-b too - it's almost too easy to find oneself entangled like that when lib-b exposes some core functionality.)
If all those crates refer to their dependencies via branch = "dev", it's no sweat to create a situation in which app <- lib-b resolves to a different commit hash than app <- lib-a <- lib-b, leading to:
Now it's 2020, Cargo does support alternate registries, and there are even a few candidates to choose from - we've started to investigate https://github.com/Hirevo/alexandrie and got some positive results, so we'll certainly re-visit this topic in the future and see how it plays out.
Multi-repo vs Mono-repo
Sticks and stones may break my bones, but multi-repo never hurt me - someone, probably.
As you might have already inferred from the previous section, we lean on a multi-repo setup - it means that we've got lots of small repositories (more than 500, actually, but most of them aren't oxidized... yet) instead of a giant one.
Migrating to a mono-repo is one of those topics that likes to pop up & stay afloat from time to time, especially when someone's bravely fighting with a random dependency conflict.
Since the internet already contains lots of articles describing the differences between both setups, we'll content ourselves with saying that overall we're satisfied with our current modus vivendi. By keeping projects separate, we're able to work on them in tandem without imposing too much synchronization between teams, even if that means we have to solve some odd dependency issue now and then.
Semantic versioning
Cargo rules with a velvet glove as long as one sticks to semantic versioning - it's unfortunate that it doesn't apply to Git dependencies, though (that is: Cargo isn't able to follow and semantically-resolve versions from tags or branches).
While we strive to keep semantic versioning in some of our projects, it doesn't matter, nor does it provide any benefit - though we'll definitely have to revisit this policy after we migrate to a self-hosted registry.
Tricks
This section's title is clickbait since, in reality, we've got just a singular trick to share: at the moment, Cargo has a bug - https://github.com/rust-lang/cargo/issues/5478 - preventing it from applying patches that only change the branch of a Git dependency, e.g. following manifest would fail:
Luckily, there's an easy workaround - you can change git = to point at a different URL that's semantically the same resource, e.g. by specifying port:
This allows us to work on cross-project features (e.g. when you have to change something in a parser library and a downstream application) without forcing us to merge each and every minor change into dev.
It's also better than doing plain some_dependency = { path = "..." }, as it allows for CI to compile & test our feature-branches out of the box.
Summary
It's difficult to sum up architecture, infrastructure or best practices in just a few paragraphs. Nonetheless, we hope this article proves to be useful and that it sheds some practical light on the topics we've described.
ANIXE is continuously looking for Rust developers. Send your CV or use our mail - praca@anixe.pl if you are ready to solve real, high availability problems in Rust language!
Find out more about work in ANIXE >>
Read more news >>