This essay was written as part of the ID2012 Ubiquitous Computing Course at KTH Royal Institute of Technology Stockholm.

The task was to produce an analytic overview of a topic in the domain of Ubiquitous Computing or IoT. I chose Rust’s application for IoT out of my personal interest in Rust and previous experiences diving into IoT and specifically embedded devices using Rust.

The essay was finished and handed in by May 2021 and reviewing the field today already shows some minor changes. Therefore be warned, the field is evolving quickly and might look better depending on when you will read this.

If you want to support my work or reach out feel free under the emails below:

Yannik Sander (yannik@kth.se / contact@ysndr.de)

If you want to use this article, please consult me first. A PDF version is available here

Introduction

The IoT and Ubiquitous Computing already makes for a large share of software being written today. To prove secure writing such software should be ergonomic while staying performant and efficient, the potential for human error should be minimized.

Rust is a young programming language with a lot of promises, including being a viable platform for IoT of Ubiquitous computing. This essay will introduce Ubiquitous Computing and IoT, then, draw a picture of the current Rust ecosystem relevant for these fields and finally analyze how well this ecosystem can provide for the demands of the field.

Ubiquitous Computing and IoT

Introduction to the character of ubiquitous computing

The notion of Ubiquitous Computing, often and in the following simply referred to as Ubicomp, has been established by Mark Weiser in his 1991 paper envisioning “The Computer for the 21st Century”[1]. In a time where computing was visible, immobile and rear compared to today’s standards, Weiser spoke of a “disappearance” of technology. Surely, he did not speak about a decrease in computing, but the opposite. The disappearance was about the obvious presence of said technology. Weiser predicted a world where computing is omnipresent – ubiquitous – “weaved into everyday life.” On the one hand, it was about the mobility of computers, i.e. being able to take computing everywhere. Yet, this doesn’t cover it completely! Indeed, it was the unintrusive enhancement of people’s life that defined Ubicomp.

In the modern day, Ubicomp has become an important part of human-computer interaction (HCI) implementation and research. Especially being aware of the user’s context and acting upon that is an important aspect of Ubicomp relevant to HCI and carrying numerous opportunities for future advancements in computing [2]. There have been great developments in technology since Weiser formulated the concept of Ubicomp that enable many of his ideas. Poppe et.al. [3] pointed out critical developments in this regard.:

Context Awareness and Pro-Activeness

Multitudes of sensors both owned (or even worn) by the user or present in the public domain allow the greater possibility to take user context into account. This includes classifying user’s actions, emotions, health and location and allows them to provide services related to that. This might be sports tracking e.g. analytics and recommendations in tennis [4] or granting access to public transport without any interaction, enabled by face detection [5]. Further, context awareness empowers services to be gradually more proactive, likewise reducing the amount of interaction required by a user, although “mixed-initiative” is said to be more appropriate to HCI. Siri [6] and Google Assistant [7] are examples of such context-sensitive, mixed-initiative services. They provide information when they are queried by a user, most notably facilitating non-physical interaction with distributed devices (i.e. through voice commands handled by a supported client device). They might also present information on their own, based on context and need.

Adaptability

Need is interesting on its own. It might mean external factors such as emergency warnings, but often employs a different concept important to Ubiquitous computing, namely, adaptability. Evers et.al.[8] claim that “future computing systems must adjust to the user’s situations, habits, and intentions.” Pro-activeness, as mentioned above, is much more useful, if not only useful, if it supports and anticipates the user’s intentions and develops with their behaviour. As such, it’s imperative to not only collect user context but also user feedback on the actions anticipated.

Intelligence

Tangential to the ability to learn from user feedback is the perceived intelligence of a Ubiquitous system. As humans, we are used to expecting decent levels of intelligence in natural interaction with each other. In effect, to be perceived as a natural part of one’s environment, rather than being a tool, ubiquitous technology needs to show intelligence too. Adaptivity and pro-activeness, as discussed before, are some aspects of this. Additionally one might ask for additional criteria such as an ability to reflect and anticipate consequences, improve their behaviour and show diverse strategies as well as natural social competence.

Speaking of intelligence, one can discern different types of intelligence by the agent that shows it. Things or machines that display intelligence are typically considered robots. User software becomes a (software) agent or softbot. While softbots (for instance the aforementioned personal agents) can already be integrated into the environment and provide non-physical interaction, going a step further one can also separately distinguish smart environments. These are often referred to as implementations of ambient intelligence. We see intelligence embedded into objects in our environment [9] such as appliances, thermostats and similar devices. Research might go even further exploring rooms, that sense the user’s presence, can store and prepare different states for different users [10, p. 21] and be augmented virtually [1].

In fact, augmented reality was also envisioned by Weiser [1] as part of a ubiquitously computing future. Several projects are pushing the idea of AR implemented in projects of varying comfort in form of the Google Glasses and Microsoft Holo Lens or smartphone-based solutions like Google Lens.

Summary of the Dimensions of Ubicomp

Summarizing the nature of Ubiquitous computing one can distinguish advances on different dimensions. First, implementations of Ubicomp might have varying distance to the user. It can be public domain face detection based access control, or intelligent rooms. Closer to the user there are wearables or smart fabrics. In recent days project like Neuralink [11] picture a future with even closer integration of computing. Second, different grades of artificial intelligence are shown. Systems that interact close to the user are supposed to do so naturally, i.e. intelligently. The less direct the interaction the less the requirement for pro-activeness, talking about face detection as an example. Weiser himself coined another dimension, namely size:

“Inch scale, foot scale and yard scale devices.”

Computing nowadays can be as big or bigger than smart screens, down to tablet and smartphone/smartwatch size. Yet, even smaller computing is present in our environment. Credit cards, contactless keys, or glucose meters are examples of such smaller and certainly much more transparent areas of computation with medical devices reaching even greater records in smallness.

So far, Ubicomp has been mainly described as a form of HCI, providing ways for users to interact more or less directly with a greater service. Ubicomp itself while popular in academics (listing more than 1.5M search results on Google Scholar¹) hasn’t become nearly as present in the industry and everyday life as a term. Possibly due to its nature being more of a concept without clear and concrete borders as seen above. Instead, the term Internet of Things (IoT) experienced a phase of ubiquity in industry and consumer electronics. What is IoT then?

The Internet of Things

While Ubicomp as a concept seems to concentrate more on the connection with humans and the possibilities it offers to them, IoT has a greater impact as a marketing term for connected devices. In a way, it’s a more practical term than Ubicomp. Yet, that does not mean it is a more concrete one. Applications of IoT include Home Automatization, Smart Cities, Media Consumption and Transportation to name a few many of which are also part of Ubicomp. The difference to Ubicomp is that IoT describes actual systems/networks of devices that work together and communicate, as well as their protocols and standards.

As shown in an article in Business Horizons [12] numerous artefacts are part of the IoT. It is the nature of a system that qualifies it as part of the IoT. Data is produced at one end by wirelessly connected sensors, send over a network, processed by specific middleware running in the cloud and driving IoT based applications.

Only considering consumer applications the market for IoT is enormous. Home automation for example has evolved rapidly over the last years, with multiple applications by competing vendors reaching from lighting overheating to property security and more. The great interest in this market led to a substantial fragmentation of the market on nearly every layer of IoT. In the pursuit of standard multiple communication protocols evolved. Nowadays, Zigbee, Z-Wave, Bluetooth LE and WiFi are the dominant standards to build networks[13]. Yet, device protocols often remain largely incompatible still.

If building on open standards, the entrance to IoT has become quite simple. Hubs provide an interface to compatible connected devices or connect them to cloud-based services. Yet there is little standardization around the interaction between different devices’ specially if coming from different vendors.

Looking at industrial contexts, an apparent difference is the predominance of sensors at the bottom of the network becomes apparent. IIoT is characterized by a multitude of wirelessly connected actuators and sensors [14]. Unsurprisingly, IIoT generates a lot of data, which needs to be stored and processes or analyzed[15]. This fact strongly motivates cloud computing or even more immediate processing at the edge of the network. [16]. Additionally, the uniqueness of many applications implies that there are even incompatibilities between systems than in the consumer market.

To summarize, IoT consists of three main elements each of which can vary in complexity based on the application as crystallized by Jayavardhana Gubbi in a 2013 paper [17].

1. Hardware is the common term for sensors, actuators and communication drivers
2. Middleware provides intermediate analytics and data storage
3. Presentation conveys the findings to the end-user

One might additionally include Software, especially protocols, in the list. In the following, this essay focuses on software related to the first two points.

Rust

Rust is a relatively modern programming language that was first introduced in 2010 by Mozilla as a basis for their experimental browser engine Servo[18] parts of which are now driving the Firefox Browser[19]. Its trifecta of speed, safety and concurrency caught peoples interest early on. Since the beginning rust strived to provide greater safety through an advanced type system. By design, Rust disallows concurrent mutable access to the same data. Instead, it employs the concepts of data ownership and borrowing. At compile-time, Rust can resolve how long references are used and when they are cleared up. In effect, (modern) Rust does not implement a garbage collector. This and the fact that it is compiled to native code through LLVM put it in the same category as other unmanaged languages such as C/C++ and account for Rust’s performance. The first stable version of Rust was released in 2015[20]. Since then public interest grew starkly, due to its promises.

By now, Rust has been voted the “most-loved” language since 2016 by developers on StackOverflows yearly survey [21]. Its today’s most convincing features are summarized by Jake Goulding[22] in a blog post from January 2020. Firstly, its versatile and ergonomic type system enables very practical safety measures, for example, replacing null pointers for more expressive and safe Option<T> types and enforcing the handling of errors through a Result<Error, T>. Additionally, these are also examples of Rust’s capability of algebraic data types. The aforementioned garbage collection model – or the lack thereof – is as well highly appreciated by users of the language as it decreases the applications memory footprint dramatically. The possibility of safe direct memory access has likewise driven Rust to be an aspiring candidate for embedded{@} devices as well as recently becoming an officially supported option for Linux kernel module development[23], not at last because sticking to Rust’s compiler enforced rules drastically reduces the possibility of segfaults. Segfaults, typically occurring when accessing invalid memory, are by default prevented by Rust’s memory design.

Additionally to the language design, Rust has built a thriving ecosystem. Its standard build tool and package manager Cargo[24] is the pivoting point of this ecosystem. Using cargo one can easily manage dependencies, config feature flags, run tests and much more. It also offers great extensibility through custom commands and built-in integrability with IDEs [25]. Cargo links in and provides tools for publishing libraries on its package library crates.io[26]. These libraries referred to as crates are considered to be one of Rust’s most important features on their own. Traditional languages such as C/C++ do not have any standard package manager, libraries are typically installed as precompiled binaries that need to be linked at compile time or runtime in case of shared objects. This requires the developer to include header files that are only resolved using a basic preprocessor, install these libraries separately and track/require them using third party tooling with little control over the actual version being used leaving many security issues to be dealt with by the user of the software and OS maintainers. C++ recently added support for modules[27] solving some problems related to header files but remains fragmented in general. Rust got inspired by more modern and ergonomic solutions of more recent languages such as NPM[28].

The Rust Ecosystem for IoT

In the introduction, Rust’s ecosystem was outlined. Focusing on IoT one needs to take a deeper look into the accompanying tools and libraries. This essay will introduce key technologies and concepts that enable the development of IoT devices and related edge computing.

When speaking of IoT ARM is by far the leading manufacturer of Chipsets used at the edge of the IoT[29] and embedded devices such as sensors. As such, to be a viable option to cover the IoT space as a language, support for ARM-based processors is imperative! Hence special focus will lie on ARM support in Rust.

Rust on Microcontrollers

Tools

A major component of the ecosystem of a programming language are tools that simplify or automate the development processes. These processes can become highly complex even for rather simple projects. For instance programming, a common microchip, requires a debugger, a connector to the on-chip debugger and the programmer and the build tool to work together. The latter also needs to be configured for the programmed chip. Several tools that have been developed try to shrink the associated learning curve and strive to allow for greater productivity quicker.

Being based on LLVM Rust supports a multitude of platforms [30] including many ARM platforms. Cargo complements this by offering an interface to cross-compile to foreign architectures. Additionally, rustup[31] provides an interface to easily acquire toolchains for these architectures and simplifies keeping track of the fast-paced releases of the Rust language. Combining these tools, cross[32] has been developed by the rust-embedded working group which uses isolated docker containers to minimize the efforts required and possible failures of setting up a development environment by providing a managed prepackaged solution. With this running and testing code for different architectures becomes as easy as

$ cross test \
  --target mips64-unknown-linux-gnuabi64

Targeting microcontrollers, in particular, the knurling project [33] develops tools that make embedded development more seamless. probe-run is a project that integrates downloading binaries to controllers, and running code, as well as connecting debuggers with cargo and can therefore be easily integrated with IDE’s. defmt significantly reduces resource overhead of logging on microchips and has been found to offer a highly integrated debug process [34].

Abstraction layers

Speaking about abstraction layers one must first understand why they are needed. Programming microcontrollers is flooded with hardware-level interaction, unsurprisingly. While rust is capable of doing these accesses, in many cases some of Rusts safety measures need to be disabled. While more is possible in these unsafe environments, obviously one strives to reduce the use of unsafe. Besides safety, ergonomics and compatibility are more reasons to ask for abstractions. Rust is known for its capabilities to bring these virtues to its users in other areas already due to expressive Generics and its trait system. In the context of embedded programming, this has enabled people to create various levels of abstractions on top of the lowest levels of interaction with the hardware.

Accessing the hardware

Peripherals on microcontrollers are configured through so-called memory-mapped registers. Manipulating the state of these registers changes how the external connectors to the chip behave, whether they are inputs or outputs, digital or analogue. Also, internal structures can be controlled this way, e.g. timers can be set and reacted upon. Unfortunately, there is no common interface to these registers not only due to the number of different manufacturers but also different chip design and application.

While configuration and layout differ, it does not do so undocumented. In fact for long SVD files [35] are being made available by manufacturers describing the chip layout formally. In Rust, this is made use of to create so-called peripheral access crates (PAC). Using a svd2rust [36] one can generate a rust library that implements a safe interface to all of the specified registers including context-based functions, such as being able to write or read from pins, or start timers using a method rather than setting bits manually.

Abstracting hardware functions

PACs do a great job making raw hardware accessible by Rust in an automated and safe way. Building on top of this, one might perform standard operations such as communicating to peripherals connect to USB, enable timers and so on. While building this functionality from the ground up based on peripheral access, a safer and more portable solution is building on a shared abstraction. Such an abstraction is provided by the embedded_hal[37] crate.

The functionality provided by embedded_hal fulfils some important requirements:

1. It is independent of any specific chip
2. Does not make restricting assumptions about how it is used on a specific chip
3. Provides low-cost abstractions that are compassable into higher-order abstractions (note that embedded_hal is still a very low-level abstraction)
4. Stemming from the previous point: Offers sufficient freedom and capabilities to base device-independent drivers upon.

Note, that embedded_hal does not implement most of the functionality, but defines interfaces that are eventually implemented for a specific chip or family of devices.

Drivers

Apart from accessing mere hardware, the most important aspect of embedded development is, as in non-embedded scenarios, processing data, and providing functionality. In the context of IoT data is typically produced by the periphery, and communicated over some network channel, it is still the internet of things. Enabling this, one finds themselves at a gap. So far, the discussed abstractions merely provide hardware access. Yet, communication, in particular, requires conformity to often complex protocols (i.e. IEEE802.11/WLAN[38]). Implementations for these protocols readily exist in C, less so in Rust, often because the modems are using more niche platforms, to begin with. Instead of reimplementing the existing C implementation for those modems, Rust focuses on offloading this functionality. Offloading means to employ a second chip running a firmware that drives a communication module and exposing the data access through a firmware specific (serial) interface. Drivers have been implemented for all sorts of such devices and often make use of the aforementioned embedded_hal to be usable from any host device.

While serial protocols such as USB, RS232 or i2C can be part of a HAL, data protocols like AT[39] are implemented separately. Crates like atat[40] transparently offer access to these protocols. Building on that, driver crates for popular networking modems are already available. With IoT in mind, we can find drivers for cellular access[41] or short-range networks[42] that connect to u-blox[43] devices. The drogue IoT project [44] not only brings support for common network standards like WiFi or LoRaWAN but also abstracts these to a transparent network interface, such that from Rust each of these network gateways can be used the same way providing TCP/UDP sockets. Building on this network abstraction the project also implements an MQTT and HTTP client.

Notable Mentions

LoRaWAN[15] is known for its application as the basis for IoT. Especially The Things Network[45] plays a major role in pushing LoRa by providing a shared infrastructure that is energy efficient, yet reliant, open and secure. Incidentally, crates to create clients to this network already exist.

Leaving Microcontrollers

At this point, Rust’s support for microcontrollers was comprehensively presented… Although a lot of the IoT is implemented on the smallest of processors, often one has more resources to spare. Devices that could be described as “raspberry pi sized,” can run a supported operating system (e.g. GNU/Linux) on a higher architecture, such as aarch64. Consequently, they profit from full Rust-Support. These offer more versatile tools and capabilities to connect to complex technologies such as Bluetooth, or processing greater amounts of data, such as camera feeds.

Rust on the Edge

In recent years WebAssembly [46] (WASM) has been growing as an OS-independent platform, meant to run programs in web-browsers at near-native speed in secure sandboxes. Rust as a language has been pushing this development forward, by language support and tooling. Not only have the three biggest freestanding WASM runtimes adopted Rust as their implementing language[49], but also there have evolved standard tools to integrate WASM into your JavaScript Codebase [50] and library support to narrow the gap between the Rust/WASM world and the JavaScript runtime[51].

Companies like fastly and Cloudflare have developed services that facilitate this platform to offer easy entrance to Edge Computing. Cloudflare Workers [52] offers the infrastructure for reliable functions on the web that can act as an ingress point for IoT devices. Workers run code compiled to WebAssembly which makes them a ready target for Rust.

Rust as a platform

In the previous section, a multitude of applications and capabilities of the Rust Language has been presented. While the ecosystem is large, it is important to also analyze it with an eye on qualitative factors to come up with a convincing conclusion about the usability of Rust as a Platform for IoT today. As rust is still evolving, many things will still improve. This essay aims to summarize the current developments and make an educated guess where Rust is heading.

Today

Rust is known for assessing its performance in many areas publicly in the form of “are we X yet” websites[54]. Unfortunately, for the domain of embedded/IoT, the community has not yet started such a project. Yet, as that form has proven very informative, this essay will adopt a similar approach.

++

Rust has stable and mature support. You can use Rust for this

+

Rust offers some support/development. Think twice.

±

Ideas are there but little has evolved from it.

-

Close to nothing has been developed. You are on your own, not recommended to use Rust here.

The good

In some regards, rust can already shine, although it might need some polish in some places. Especially the strong features of Rust, tooling and performance, can shine too in the IoT context.

Performance and Ergonomics (++)

One of the strongest points to make about Rust is probably its performance. This does not mean solely its runtime performance but also its development process.

Rust is often hailed for the high-level elements that make it look and act like a general-purpose language in many regards. At the same time, it embraces the concept of “Zero-Cost Abstractions” that let it produce highly optimized code without accepting drawbacks on its high-level features.

Yet in areas where every byte counts, with rust one, has to trust on the optimizer to produce sufficiently small binaries. Which becomes harder given how easy it is to add dependencies to a project. Projects exist to help monitor the size of binaries but the main problem remains.

Looking at it in another way, given how high-level rust can be, memory accesses are not as obvious as they are with C. Clearly, this can cause unexpected problems in performance, especially on microcontrollers with limited memory speeds.

Tools (++)

Rust is known for its great tooling. This doesn’t stop in the world of embedded systems and IoT. For one, Rust is pioneering the world of WebAssembly. Additionally, the entry into Embedded systems is made greatly easier given the rust tooling.

As we saw with the abstractions above, embedded Rust does not exclude the possibility to use cargo and its package management. In fact, it even provides a measure to tell whether a library can run on embedded devices or not, precisely as long as it does not use the standard library that is built on top of specific operating systems. Such crates are marked as #[no_std]. It doesn’t stop there. We saw projects automatizing the whole process of downloading binaries and running/debugging them through standard cargo invocations that integrate well with IDEs.

Of course, not everything is perfect in this regard area yet. As a lot of Rust’s tooling is automatized, as a user one faces a rather high-level view of the process. While this can be desirable, it reduces the account one has to tell what parts are causing errors. Also, due to abstraction over multiple interfaces, the functionality provided might be less than what would theoretically be possible using those tools directly, at which point using Rust might become more of a burden as configuring these tools to fit Rust might not be trivial in every case.

A similar thing can be said about Rust for Edge Computing using WebAssembly. We have seen it as the driving language for modern runtimes and offering rich library support[55]. As such given its tools, it is easy to integrate them into existing platforms or important in the domain of Edge Computing write high performant workers in it.

Frameworks and Libraries (+)

The presentation shed light on some of the most influential projects. While on the WASM front Rust has already developed a mature environment. In the space of embedded devices, libraries while bringing support for many devices, do not enjoy the same kinds of maturity and maintenance.

The embedded-hal is a great leap toward a unified API on microcontrollers and is already heavily used. Yet, many projects have implemented drivers independently or base on incompatible versions. Drivers are generally added more as an implementation to tick the boxes for a specific use case and therefore do usually not cover the available functionality.

drogue-wifi for instance implements a driver for the WiFi breakout board ESP8266. Yet, while the target chip is capable of a whole range of functionality, the driver only implements a limited subset of that such that it fits the drogue project. More extensive support would be desirable to use rust more ergonomically and ease the development.

The bad

As a comparatively young language, especially compared to its contenders in the embedded world, naturally, Rust has a set of drawbacks mainly connected to its development pace and lack of maturity and experience in the industry.

Documentation (+/±)

Generally, Rust is known for its great documentation. There is even specialized tooling around it. Rustdoc[56] is the standard tool to generate documentation from rust source code. Docs.rs[57] adds to this hosting documentation for the whole crates.io index of packages. The fact that all documentation is entangled this tightly is a major win to the whole community and aids development dramatically.

Rust has also evolved the mdBook[58] tool. It is used throughout the rust community to assemble more high-level documentation and tutorials. For embedded the rust Community has published multiple of these books that aim to introduce programming Rust for low power hardware. Unfortunately, while covering the basics, once diverting from these or using newer technologies the amount of documentation is very sparse. At that point, it is mostly up to the developer of certain libraries to provide decent documentation. Yet, often development is faster than documentation, leading to outdated tutorials and examples. Additionally, many parts of the rust ecosystem are still mostly documented through blog posts. Although this is certainly not a bad thing in itself, as it shows that the interest is big, it is difficult to tell the recency or relevance of such articles. Likewise, there is little being written about how different projects are supposed to work together.

Stability (±)

As mentioned earlier, incompatibility especially at the abstraction level can be a problem. Indeed, embedded-hal is still considered unstable and while sticking to a semantic versioning protocol, must still be considered to break existing code without previous warning. Consequently, this means that the whole ecosystem around it is at most as stable as the abstraction layer they built upon. In reality, most libraries are yet to be considered incidental, meaning they were built to fit a need of someone at some time and are therefore not actively maintained or feature complete.

A few projects trying to build a stable foundation for certain functional units, such as the IoT network/cloud project drogue. Still, the interfaces that come to existence due to these projects aim foremost for internal compatibility.

Framework Interoperability (±)

Extending on the internal compatibility, it would be desirable to have projects interoperate cleanly. The aforementioned stability issues and fragmented ecosystem can often not be equalized by the overarching abstraction layer.

Moreover, documentation to integrate one library within the other is often hard to find as well. RTIC[59] for example promises to provide a framework that works concurrently by managing interrupts and resources. Yet, it brings in a very different way to set up projects that complicate how to get started with the whole system especially as examples are scarce here too.

The ugly

Lastly, some tasks are so far virtually impossible to achieve with Rust or require a lot of work to be invested by the user.

ARM (-)

We have seen that on ARM Rust offers great support. Indeed the stm32-rs group, for example, tries to provide embedded-has implementations for all chips manufactured by STM. Other ARM vendors have similar good support.

It is once one asks to write software for other architectures. First, they are limited by support for these architectures of the LLVM backend. The popular IoT platform Arduino for instance runs on AVR based chips which require a fork of LLVM to be programmed for In Rust. Extensa, the architecture employed by the ESP8266/ESP32 chips also requires additional care. These reasons drive the current disinterest in writing software for these chips (as it is not possible or difficult) which in turn affects the motivation to bring support for the platforms to Rust in the first place.

In the future

Today, Rust’s ecosystem is not all roses. A lot of things are missing still. Yet, observing the community fundamental steps are being taken. Looking at the future there are a few clear indicators that Rust will become a growing influence for not and ubiquitous computing.

Rust Foundation

More generically, the advent of the Rust Foundation [60] will eventually also benefit the embedded section of the language. With greater structural organization and backing from industry-leading companies such as Google and Amazon, Microsoft, Huawei and Facebook, Rust manifests itself as a credible choice.

Embedded-WG

The embedded working group has been mentioned throughout this essay. They have proven themselves as the originators of remarkable work that has brought rust a long way. Its projects will for sure continue to improve the experience of embedded development in Rust.

Ferrous

Ferrous Systems, the leading force behind the excellent knurling project, is committed to further invest in Rusts embedded future. Ferrous already contributes to the Rust Open Source community in many ways, and with knurling still being a young project, its influence in the embedded world is still to be expanded.

Libraries/Tools

Finally, with more companies and individuals committing to Rust as their language of choice, albeit its current pitfalls will eventually populate the language with more helpful libraries and tools. Especially, once fundamental libraries such as the embedded-hal a point of stability, libraries building on these are expected to follow.

Taking into account WebAssembly, we see that with its adoption in all major browsers and the stabilization of its specification, WASM is there to stay. Rust has been influencing its development a lot until today

Conclusion

Throughout this essay, the domain of Ubiquitous Computing and IoT has been described in detail. Building on that the programming language Rust has been closely examined on its capabilities to fulfil the needs of this field. We have seen which infrastructure drives Rust’s support of embedded programming, how WebAssembly enables Rust to play a leading role as a language to implement functions on the Edge. Apart from this descriptive part, we put the available ecosystem into perspective, pointing out its strengths and current weaknesses.

Finally, we can conclude that Rust is perfectly capable of doing specific tasks, in the area of Embedded Computing and more so on higher levels of the Internet of Things, such as lightweight computing on the edge and the implementation of backend services. Yet, it shows that Rust is a fairly recent language. As such parts of its ecosystem, relevant to IoT, are still evolving, lack even some significant foundations and are far from stable. For early adopters and the generally curious Rust still offers the foundations on which one can build their own solutions, albeit without providing the maturity of decades of development. On the bright side, we see several companies like Drogue, Ferrous Systems and other independent groups, doing exactly that. Crucial foundations are in active development and promise a brighter future for Rust.

Condensing this essay into one sentence

Rust shows the potential to become the IoT platform of choice in the future, providing speed, ergonomics and safety, but does not show the maturity to be readily used as such without thorough consideration.

… Wow! You made it until here. Thank you a lot for your time and interest. If you have any thoughts about this post after all, please feel free to reach out to me.

References

Git hooks are very useful in theory e.g. to enforce style guidelines of code being pushed or doing arbitrary cleanup/analysis in response to various git events. Yet, if you are using nix, the way git hooks are set up and managed goes against the ideas of Nix.

The presented approach does not solve the underlying issue of mutability but makes git-hooks more trackable and easily appliable.

The big picture

Let me present a full-fledged example first, the functions are individually posted down in the appendix.

let
    installGitHooks = hookTypes:
        let mkHook = type: hooks: {
        hook = pkgs.writeShellScript type
        ''
            for hook in ${pkgs.symlinkJoin { name = "${type}-git-hooks"; paths = hooks; }}/bin/*; do
            $hook
            RESULT=$?
            if [ $RESULT != 0 ]; then
                echo "$hook returned non-zero: $RESULT, abort operation"
            exit $RESULT
            fi
            done
            echo "$INSTALLED_GIT_HOOKS $type"
            exit 0
        '';
        inherit type;
        };

        installHookScript = { type, hook }: ''
            if [[ -e .git/hooks/${type} ]]; then
                echo "Warn: ${type} hook already present, skipping"
            else
                ln -s ${hook} $PWD/.git/hooks/${type}
                INSTALLED_GIT_HOOKS+=(${type})
            fi
        '';
        in

        pkgs.writeShellScriptBin "install-git-hooks" 
        ''
            if [[ ! -d .git ]] || [[ ! -f flake.nix ]]; then
                echo "Invocate \`nix develop\` from the project root directory."
                exit 1
            fi

            if [[ -e .git/hooks/nix-installed-hooks ]]; then
                echo "Hooks already installed, reinstalling"
                ${uninstallGitHooks}/bin/${uninstallGitHooks.name}
            fi

            mkdir -p ./.git/hooks
            
            ${pkgs.lib.concatStringsSep "\n" (pkgs.lib.mapAttrsToList (type: hooks: installHookScript (mkHook type hooks)) hookTypes )}

            echo "Installed git hooks: $INSTALLED_GIT_HOOKS"
            printf "%s\n" "''${INSTALLED_GIT_HOOKS[@]}" > .git/hooks/nix-installed-hooks
        '';

    uninstallGitHooks = pkgs.writeShellScriptBin "uninstall-git-hooks" 
        ''
            if [[ ! -e "$PWD/.git/hooks/nix-installed-hooks" ]]; then
            echo "Error: could find list of installed hooks."
            exit 1
            fi

            while read -r hook
            do
            echo "Uninstalling $hook"
            rm "$PWD/.git/hooks/$hook"
            done < "$PWD/.git/hooks/nix-installed-hooks"

            rm "$PWD/.git/hooks/nix-installed-hooks"
        '';
    
    rustFormatHook = pkgs.writeShellScriptBin "check-rust-format-hook"
        ''
            ${pkgs.rustfmt}/bin/rustfmt --check
            RESULT=$?
            [ $RESULT != 0 ] && echo "Please run \`cargo fmt\` before"
            exit $RESULT
        '';

    hookInstaller =  installGitHooks { pre-commit = [rustFormatHook]; } 
in 
    pkgs.mkShell {
        packages = [ (installGitHooks { pre-commit = [rustFormatHook];) } uninstallGitHooks ];
        inputsFrom = [ ];

        shellHook = ''
            echo "=== Development shell ==="
            echo "Info: Git hooks can be installed using \`install-git-hooks\`"
            # or run `install-git-hooks` automatically
        '';
    };

Explanation

The above code defines functions to build installation and uninstallation commands for git hooks.

To run a hook, create a derivation with the hook (or hooks) for one event located in the bin/ folder, i.e. using. writeShellScriptBin :

rustFormatHook = pkgs.writeShellScriptBin "check-rust-format-hook" ''
    ${pkgs.rustfmt}/bin/rustfmt --check
'';

Then create a hookInstaller by adding the derivation to a list of hooks for a specific event type:

hookInstaller =  installGitHooks { pre-commit = [rustFormatHook]; } 

This will run all commands under ${rustFormatHook}/bin/* for pre-commit events.

The event types can be arbitrary but have to comply with actual git hooks to be run.

Finally, add the installer (and optionally the uninstall-command) to your dev shell as input programs. You can choose to automatically run the installer at entrance to the shell as a shellHook or manually by the user.

pkgs.mkShell {
    packages = [ hookInstaller uninstallGitHooks ];
    inputsFrom = [ ];

    shellHook = ''
        echo "=== Development shell ==="
        echo "Info: Git hooks can be installed using \`install-git-hooks\`"
        # or run `install-git-hooks` automatically
    '';
};

Clarification

I am aware that there is much more advanced tools available with more advanced configuration systems etc. Yet, this approach is nix-native and sufficiently flexible for simple hook setups. It does still require you to write all hooks yourself but this way they can be tracked with nix and in theory make use of programs not even populated to the final environment.

Nonetheless, this might be of interest for someone and if so thanks for reading.

Appendix

Copy the respective functions here to include them into your project or checkout the repository on GitHub avoid copy-pasting and receive upstream bugfixes. ## Installer

installGitHooks = hookTypes:
    let mkHook = type: hooks: {
    hook = pkgs.writeShellScript type
    ''
        for hook in ${pkgs.symlinkJoin { name = "${type}-git-hooks"; paths = hooks; }}/bin/*; do
        $hook
        RESULT=$?
        if [ $RESULT != 0 ]; then
            echo "$hook returned non-zero: $RESULT, abort operation"
        exit $RESULT
        fi
        done
        echo "$INSTALLED_GIT_HOOKS $type"
        exit 0
    '';
    inherit type;
    };

    installHookScript = { type, hook }: ''
        if [[ -e .git/hooks/${type} ]]; then
            echo "Warn: ${type} hook already present, skipping"
        else
            ln -s ${hook} $PWD/.git/hooks/${type}
            INSTALLED_GIT_HOOKS+=(${type})
        fi
    '';
    in

    pkgs.writeShellScriptBin "install-git-hooks" 
    ''
        if [[ ! -d .git ]] || [[ ! -f flake.nix ]]; then
            echo "Invocate \`nix develop\` from the project root directory."
            exit 1
        fi

        if [[ -e .git/hooks/nix-installed-hooks ]]; then
            echo "Hooks already installed, reinstalling"
            ${uninstallGitHooks}/bin/${uninstallGitHooks.name}
        fi

        mkdir -p ./.git/hooks
        
        ${pkgs.lib.concatStringsSep "\n" (pkgs.lib.mapAttrsToList (type: hooks: installHookScript (mkHook type hooks)) hookTypes )}

        echo "Installed git hooks: $INSTALLED_GIT_HOOKS"
        printf "%s\n" "''${INSTALLED_GIT_HOOKS[@]}" > .git/hooks/nix-installed-hooks
    '';

Uninstaller

uninstallGitHooks = pkgs.writeShellScriptBin "uninstall-git-hooks" 
    ''
        if [[ ! -e "$PWD/.git/hooks/nix-installed-hooks" ]]; then
        echo "Error: could find list of installed hooks."
        exit 1
        fi

        while read -r hook
        do
        echo "Uninstalling $hook"
        rm "$PWD/.git/hooks/$hook"
        done < "$PWD/.git/hooks/nix-installed-hooks"

        rm "$PWD/.git/hooks/nix-installed-hooks"
    '';

This post follows the release of Nix 2.4, which among many other things introduced an all new (experimental) nix command along with the previously discussed flake feature.

The nix command aims to collect most common commands such as nix-build, nix-copy-closure, nix-env, … as subcommands of one common program. Unsurprisingly, this does not spare nix-shell.

Yet, unlike some of the other commands which received a more or less one-to-one replacement it is not so easy with nix-shell. This command was actually broken up into multiple commands with different semantics: nix shell, nix develop and nix run. Yet, depending on what you used nix-shell for in the past the new commands may not exactly do what you would expect. Indeed, they happen to cause quite some confusion already. Most notably, while nix-shell invoked without any other flags previously set up the build environment of a derivation (which could be somewhat abused to define general development environments) that is not what nix shell will do…

What the shell…¹

Yes, but from the start…

If you are here just for the commands you can read the tl;dr for nix develop, nix shell, or nix run directly or jump to the notes section.

Development shells (nix-shell [DERIVATION])

Development shells using the new nix command are now created with nix develop.

That command is a more focused version with additional tools for development. It is meant to be used with flakes but supports standalone derivations using the -f flag. How to use it then and what will it do?

A development aid

The intended use of nix develop is to recreate build environments for single packages. That is, you call nix develop nixpkgs#hello and are dropped into a shell that is as close as possible to the nix builder environment.

“yes,.. and?”

This allows you to build a package step by step, or better phase by phase, meaning that in particular the build instructions used by nix can be reused for development purposes. Thus, we can use nix without needing to repeatedly go through full build processes, including wasteful copying, configuring, etc.

To make this process even easier, nix develop now comes with special arguments to run those phases directly.

In essence, this is exactly what nix-shell was intended for!

Practically, nix-shell was also used for another purpose; reproducible development environments.

Development environments

Particularly useful combined with tools like direnv, one can leverage the fact that the resulting shell of nix-shell nix develop includes all declared buildInputs and environment variables to put together an environment with all sorts of dependencies and development tools available. Importantly, nix will also ensure all setupHooks are run when the shell is opened allowing for some impure setup to happen.

pkgs.mkShell = {
  # a list of packages to add to the shell environment
  packages ? [ ]
, # propagate all the inputs from the given derivations
  inputsFrom ? [ ]
, buildInputs ? [ ]
, nativeBuildInputs ? [ ]
, propagatedBuildInputs ? [ ]
, propagatedNativeBuildInputs ? [ ]
, ...
}: ...

Being closely connected to flakes, nix develop supports loading a flake’s development shell directly if a devShell output is defined.

{
  description = "Flake utils demo";

  inputs.nixpkgs.url = "github:nixos/nixpkgs";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils }:
    let
    in
      flake-utils.lib.eachDefaultSystem (
        system:
          let
            pkgs' = import nixpkgs { inherit system; };
          in
            rec {
              packages = { myPackage = ... }; # packages defined here
              devShell  = pkgs'.mkShell = {
                # a list of packages to add to the shell environment
                packages ? [ jq ]
                , # propagate all the inputs from the given derivations
                  # this adds all the tools that can build myPackage to the environment
                inputsFrom ? [ pacakges.myPackage ]
              };
            }
      );
}

Temporary Programs

The fact that nix is built on the idea of the nix store from which user environments are created by cherry-picking the desired packages may raise the question, whether we may be able to amend our current environment imperatively². And in fact, we can. It is possible to add software to a user’s profile by imperatively by the means of nix-env -iA or nowadays nix profile install.

Yet, we also know that installing software this way is not the Nix way of doing things.

For the times when we do want to have some program at our disposal, either to try it out, use a different version or just needing it only temporarily, there should be a way to get this without going through the effort of adding it to your configuration.nix, home.nix, project default.nix, etc. and rebuilding your environment. In these cases, traditional distributions reach back to installing software or relying on containerization (i.e. docker). In Nix, while you could install the piece of software, the aforementioned usage of the nix store allows making software available temporarily without installing it.

This is what nix-shell -p <package+> is used for. nix-shell will retrieve the desired packages and open a shell with these packages mixed in.

$ asciidoc -b html5 manual.adoc
zsh: command not found: asciidoc

$ nix-shell -p asciidoc
[nix-shell] $ asciidoc -b html5 manual.adoc

with import <nixpkgs> { }; (pkgs.runCommandCC or pkgs.runCommand) "shell" { buildInputs = [ (asciidoc) ]; } ""

$ nix-shell -p "import ./some.nix {}" 

But this is not about nix-shell…

Nix 2.4 allows the same thing using nix shell now focussing on flakes.

With nix shell any output of a flake can be added to the environment by running

$ nix shell nixpkgs#asciidoc

Here, nix strictly expects a flake URL.

Like nix-shell this command supports multiple arguments.

Run scripts

Apart from dropping into development shells, nix-shell can also be used to run commands and programs from derivation not currently installed to the user’s profile. This is it can build a shell as before and run a command inside transparently.

We discussed the use of nix-shell --command COMMAND ARGS above, where we would run a command from within the build environment of a derivation. Similarly, we may want to just run a program provided by a derivation. For this nix-shell provided the --run argument

$ asciidoc 
zsh: command not found: asciidoc

$ nix-shell -p asciidoc --command asciidoc
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

$ asciidoc # still available as were in the build shell with asciidoc present
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

$ asciidoc 
zsh: command not found: asciidoc

$ nix-shell -p asciidoc --run asciidoc
Man page:     asciidoc --help manpage
Syntax:       asciidoc --help syntax

$ asciidoc # not available anymore
zsh: command not found: asciidoc

nix shell -c

As the functions of nix-shell DERIVATION and nix-shell -p DERIVATION were separated, the new tools come with new clearer semantics.

The generic nix-shell --run function is now nix shell -c. Given an installable, nix allows to run any command in an environment where the installable is present. Note that this command is run in a non-interactive shell. The shell is dropped as the command ends.

$ nix shell nixpkgs#asciidoc -c asciidoc

nix run

Yet, nix shell -c will still require to type the name of the executed program. As for most programs this command is the same as the derivation name e.g. nix shell nixpkgs#asciidoc -c asciidoc another command was introduced named nix run. With nix run the previous command can be run as nix run nixpkgs#asciidoc.

Naturally, the functionality of nix run goes further and as is the case for many other new commands mainly concerns flakes. Next to packages, nixosModules and others, flakes can now also define apps. Written as records with two fields -type (currently, necessarily app) and program (an executable path) - these apps can be run directly using nix run.

This app definition

...
outputs = {self}: {
  ...
  apps.x86_64-linux.watch = {
    type = "app";
    program = "${generator-watch}";
  };
}

… could be used to watch and build this blogs source directly using nix run .#watch, given generator-watch is a script in the nix store. Note that program only accepts paths in the store and no default arguments.

If an attribute to nix run is not found as an app, nix will look up a program using this key instead and execute programs.${<program>}/bin/<program> instead.

Shell interpreter #!/usr/bin/env nix-shell

Lastly, a useful feature of nix-shell is its usage as a shell interpreter. What that means is that nix-shell can be used to dynamically fetch dependencies for a script file and execute the file in that context. Shell interpreters are defined using a special syntax at the start of script files.

#!/usr/bin/env python

# file.py

print("hello")

The nix-shell command can be used to use nix to provide the actual interpreter using

#! /usr/bin/env nix-shell
#! nix-shell -i real-interpreter -p packages

Therefore, the above example would therefore continue to work even without python installed, if defined as

#! /usr/bin/env nix-shell
#! nix-shell -i python -p python3
print("hello")

Notes and Resources

To flake or not to flake

Most of the new nix commands are designed in a flake first way. Most notably nix {shell,develop,run} expect flake URLs as argument. Traditional *.nix files can be used with the --expr argument all commands support. As flake mode imposes greater purity strictness, imports have to happen with the --impure flag given:

$ nix shell --impure --expr "import my.nix {}"

A word about shellHooks

While shell hooks were previously introduced as a means to run certain commands to set up a development shell with nix develop and nix-shell, they find use in other places. Concretely, neither nix run nor nix shell are running these hooks. This is even though their nix-shell equivalent does so.

What that means is that nix shell cannot be used to load e.g. a Python environment with packages directly as python modules use the shellHook to set up the PYTHONPATH. The same goes for Perl and other ecosystems with similar principles. This means

nix shell nixpkgs#{python,python3Packages.numpy}

and

nix-shell -p python python3Packages.numpy

Do not evaluate to the same thing.

Instead, one needs to work around with custom expressions or mkShells passed to nix develop.

Workarounds

For Python in particular we can use python3.withPackages to build an ad-hoc derivation with the paths set up correctly:

nix shell --impure --expr "with import <nixpkgs> {}; python.withPackages (pkgs: with pkgs; [ prettytable ])" 

This might work as well with other language ecosystems

Alternatively, we can (ab)use nix develop by passing it a devShell:

nix develop --impure --expr "with import <nixpkgs> {}; pkgs.mkShell { packages = [python3 python3Packages.numpy ];}" 

Both approaches work to some degree but are clunky (i.e. not improving UX as promised) and rely on the supposed-to-be-superseded channels.

The second-hardest problem

Picking up the confusion mentioned in the beginning, there is another problem with nix shell… naming. Being so closely named to its semantically different predecessor, it is impossible to query google for meaningful, targeted results. This is a pain for newcomers and more experienced nix’ers alike. And indeed, there is a heated discussion on renaming the shell command. Yet, until that is resolved, I hope this guide helps to understand the differences a bit better.

newnixshellguide

Appendix

Summary table

This essay was part of the DD2482 DevOps Course at KTH (open on github).

The aim was to present a topic of our interest in 2000 ± 5% words. We chose the rust build system out of interest in the language and to learn about the difficulties and implications of managing a reasonably big open source project such as the Rust Programming language.

While researching about and writing this essay, the general lack of definite and up-to-date information about the topic drove our motivation to assemble this snapshot which is supposed to cover all important steps given the word restrictions.

This essay is the product of a pair project by:

• Aron Hansen Berggren (aronber@kth.se)
• and Yannik Sander (yannik@kth.se / contact@ysndr.de)

If you want to use this article, please consult either of us. A PDF version is available here

Introduction

There is a lot of talk about how DevOps supposedly solves all your problems when developing software, from the features planned to how they are used in production and everything between. Even open-source projects are expected to follow DevOps principles these days. The Rust programming language is no exception.
But how does DevOps Scale for projects as massive as rust, an advanced programming language which compiles to every mainstream platform with tooling, documentation and hundreds of contributors? This is what will be explored and explained in the upcoming sections.

We start by explaining what we are looking at, before diving into the tools that ensure this project’s integrity.

What is Rust

According to the official Rust language website[1], Rust is a low-level programming language that focuses on performance, reliability and productivity. It extends these in the following ways:

• Performance:
  Memory efficient without any garbage collector or run-time.

• Reliability:
  Extensive type system and ownership models to guarantee memory and thread safety at compile time.

• Productivity:
  The tooling for working with rust such as documentation, build tools and compiler errors are feature-complete and reliable.

These points together with a strong type-system and its expansive ecosystem are pointed out as the reasons for Rust being voted as the most loved programming language 4 years in a row in the yearly surveys by Stack Overflow[2].

What is a “CI”

A CI, or Continuous Integration, takes some input and performs a set of steps to output some product at the end. This often includes code reviews, automated tests, code compilation, packaging of compiled code and further validation of the packages before publishing.

Throughout this process, one of the most important aspects is to monitor these results and act on that feedback.[3], [4]

The Rabbit Hole Rust Build CI

This section presents the specifics of the Rust project’s build process. We strive to present a comprehensive picture of the most important pieces, as the overall process would not get a fair representation in this limited scope. We will focus on how Rust facilitates the bors bot and its development pattern while also introducing the social interactions bot triagebot.

Triagebot

The Triagebot can be summarised as the people and issues managing bot of the Rust development process. Its documentation[5] explains how it achieves this:
It is the instantiator of tags on issues by being invoked by its GitHub name, “rustbot?” It also makes it easy to ping teams within the Rust organisation, for example, the documentation maintainers by tracking more than just people and issues. This makes it easier for the teams to assist without those in need of help knowing whom directly to contact, as well as for the teams to find issues that need their attention.

Developing with bors

Inspecting the rust-lang/rust¹ repository two things become apparent:

First, none of the main branches shows any trace of being a development branch. On the contrary, their naming (stable, beta, master) rather suggests a connection with the release. Secondly, supporting the first insight, the commit history appears very automated as well. Many commits are structured as follows:

Auto merge of #84353 - estebank:as-ref-mir, r=davidtwco

Under closer inspection these commits become apparent as merge-commits, merging a set of changes into the master branch. All of these commits are merged by bors and refer to PR.

We see: bors automates the merging of PR. Indeed none of the commits made directly onto the master branch are by any other committer than bors.

Yet, where do these commits come from and more importantly are they even verified?

The Pull Request

When opening a Pull Request (PR) on the rust repository, if you did not ask for any specific reviewer, the rust-highfive bot assigns a rust team member to it. This affects the assignee on GitHub as well as leaving a comment picked up by triagebot.

A GitHub Action pr will test the code on three major platforms (Windows and two Linux systems with different toolchains).

If if the action succeeds and a maintainer reviewed your code they might comment in one of the following ways:

• r+ In bors terms this is equivalent to pressing the green “Merge” button and comes with the same responsibility

• Mark the PR as rollup target. Note; We will cover rollups in depth after going through bors’ workflow below.

• Instruct the author to do changes

• r? username? Ask for another review

In case of the first action (r+) we see bors commenting with something like:

⌛ Testing commit f43ee8e with merge e888a57…

This, given the tests run successfully and the change is approved, will cause bors to merge the PR.

A bot’s playground

The whole CI of the rust project is managed in a fork of the main repository. That fork lives in rust-lang-ci/rust². It ensures that the main repository, especially its master branch, always builds without failure. Interactions with this repository are indirect and managed by bors. The two main branches that are of interest are auto and try both of which trigger GitHub Actions workflows with the same name as defined in the repository.[6]

When a maintainer approves a change (i.e. by commenting with r+) bors takes the changes and force-pushes them to the auto branch. Changes on this branch subsequently trigger the GitHub Actions system.

The auto Action

The auto action is the heart of rust’s CI build process. It builds and tests the rust compiler on all 57 supported platforms[7]. Most of the action sets up the CI environment including connecting to a compiler cache for intermediate built dependencies ³sccache and initializing the repository by pulling all related submodules.

Once this is done the actual build will be executed. On available platforms this is run natively. All other platforms are cross-compiled using a custom docker image.

Finally, assuming the build succeeded the produced artefacts are pushed to an S3 object storage⁴.

Unsurprisingly, as the build and tests need to succeed for such a large number of targets, a full CI run on auto takes two-and-a-half to three hours[8]. The whole CI runs on custom machines owned by the rust projects[9]. This means they are always available for bors, but only in a limited capacity. The extension of this is that it is not possible to run more than eight auto builds per day and thus at most eight PRs are merged every day!

The very curious reader might ask themselves now how this system is supposed to facilitate a CI workflow incorporating regular automated testing. In part, this is where the second action comes into play.

The try Action

The try action, like auto is triggered by pushes made by bors targeting the try branch.

This action is structurally equivalent to the one described before. The major difference shows when looking at the build targets. Instead of 57 targets it only runs for x86_64 Linux, for it is the most widely available target. It allows a maintainer to get an idea of whether the change introduced by a PR will build as in contrast to the pr action this one build the complete rust toolchain including all tests.

With try and auto in mind one might wonder:

“Will all changes trigger a full build and test on 50+ platforms? Is that not pretty wasteful for simple changes such as correcting missspellings?”

It is and that brings us back to the PR process from before.

Rollups

In Section 2.2.2.1 a maximum number of 8 PR’s per day limit was exposed. However, documentation changes and other small fixes that do not impose any notable risk to the pipeline can be bundled. These bundled PR’s are known as a rollup[10].

A rollup is initiated by a maintainer through bors’ web interface ⁵. It shows mergeability and the associated risk for open PRs and lets the maintainer select a group of PRs that will be queued for bors to push them onto the CI’s auto branch. This causes them to be tested collectively thus saving resources.

The Green Master

As stated previously, successfully running/testing on causes bors to commit these changes back to master. As the auto branch is always the latest master this ensures that the main repository is always green.
Yet, not everything on the rust repository is the rust compiler! There are some custom tools that might fail to build but do not influence the compiler. These breakages should not fail the tests. Therefore when the CI merges changes into master it runs its final action master which collects information about which tools succeeded to build and publishes them to the toolstate repository⁶.

Action! Or lack there off

The Rust CI is running on the GitHub Actions platform[7], but what is it? One engineer, Jonas Hecht describes GitHub Actions in a blog post[11] as the next generation CI/CD. He goes further to explain why: CI/CD has been available as file configurations for a good while now, but GitHub Actions takes it one step further with the easily accessible pre-built actions available to everyone on GitHub. He describes them as the missing Lego™ blocks for building a complete CI/CD pipeline, with close to single click interactions with code analysis and deployment tools as examples.
Yet, analysing the actions file[12] we see that there are not any GitHub Actions ran by the Rust CI. The Rust project does not need more features for their build pipeline as they have built custom tooling for the necessary parts instead, sacrificing DevOps velocity offered by this available platform to push for higher build-pipe throughput, reliability and security.

Wrap Up

Having escaped the wonderland of rust, what did we learn?

The most unique feature of the rust project might be that most of its CI, although running through GitHub PRs, is mostly independent. First of all, it’s run in its own repository, whose master is a bot. Secondly, all the interactions with the CI happen through commands sent to a bot either in form of comments or its web interface. This extends into this bot bors performing all the heavy lifting merging PRs. And finally, although running on GitHub’s infrastructure, the actual actions are entirely implemented as part of the project.

These features constitute a unique integration of custom tooling and GitHub and make rust one of the most extraordinary open-source projects.

I am happy to get back posting on this blog again after a while of silence. I had been kept busy lately with finishing my Bachelor’s and moving to Stockholm to start my Master’s at the Royal Institute of Technology. In the meantime I also started working for NGI-ZERO, an initiative supporting open source projects led by the European Commission. As part of my work for them I package supported projects for the Nix ecosystem. The goal of this effort is to provide a reproducible and easy way to install and use this software. Due to the fact that those projects come in various shapes and sizes, need to be reviewed and should not clutter nixpkgs unnecessarily, nig-nix from builds on top of the latest feature to be added to Nix, Flakes.

Although still not available in stable Nix, Flakes can already be used using wrappers and the unstable nix package. Inspired by my work and eager to try them for some private project i migrated my blog’s tooling to use Flakes while also keeping up with the latest additions to the Haskell/Hakyll world.

Was it worth it? We will see but first, what am I even talking about?

Flake (noun, /fleɪk/)

a person you cannot trust to remember things or to do what they promise, or someone who behaves in a strange way

Cambridge Dictionary¹

Well, this was not too helpful and does not sound too promising either. So which other definition do we find there?

a small, thin piece of something, especially if it has come from a surface covered with a layer of something

This will definitely sound relatable in a moment.

Why?

Nix derivations are not completely sand boxed! Well the build is, meaning during the phases no network access, or access to to the hosts file system is possible. What is not so hermetic is the nix expression itself. Have you ever seen this line?

{pkgs ? import <nixpkgs> {}}:

Yes you have, certainly! One of the reasons why a derivation might fail on your system is due to this. The problem is that it relies on the nixpkgs channel currently active on your system, which is a major impurity, because people tend to not have the same version active or might even set it to different releases (release-xx.xx, unstable, …).
The common way around this is to pin the nixpkgs used by the developer. This is only a convention though. There is no definite way how to do this. You can see an example in my previous aticle about how I use nix on this blog. The main idea is to state the current version of nixpkgs in a file, fetch the tarball of that version and build against that. This is fine but requires us to update the version manually, and is not consistent among projects.
Alternatively, the tool that emerged from this is niv. Niv provides an easy interface to add external sources and import them from the projects Nix file. Niv in the background writes a lockfile and streamlines updating.

So there might be a way to pin packages and with some discipline we can get around this impurity. How about this then?

{ develop ? true }:
...
if develop then ...
           else ...

So easily your derivation becomes dependent on some command line argument. In this case it’s a simple flag (how do you know?). In other cases it could expect a string even, like setting the ghc version used. This poses multiple problems. First you create an impurity, you depend on some command line flag to be given. More importantly, these inputs are not discoverable without inspecting the nix file, for string inputs, it might even be hard to figure out which values it takes, in which format, etc for this you need to dig through the definition. Also, even with defaults set, importing the project from another place expects the importer to know which arguments they can/must give and require them to inspect the definition again.

Another issue you might come across:

{...}:
let
...
in {
  inherit shell app some-other-app;
}

How do you know what to import, what does it do? There are simply no inherent semantics and also no conventions.

Additionally to this Eelco also lists some more issues with this. Firstly, due to missing strong conventions and semantics composing packages is difficult, or not nice of an experience for the user. It also means that discovering packages can not be automated easily.

Creating a flake for this blog

To understand how flakes aim to solve those problems, lets - once again - have a look on the new definition of this blog.

{
  description = "Flake utils demo";

  inputs.nixpkgs.url = "github:nixos/nixpkgs";
  inputs.nur.url = "github:nix-community/NUR";
  inputs.flake-utils.url = "github:numtide/flake-utils";


  inputs.uikit-src = {
    url = "https://github.com/uikit/uikit/archive/v3.5.8.tar.gz";
    flake = false;
  };


  outputs = { self, nixpkgs, nur, flake-utils, uikit-src }:
    let
    in
      flake-utils.lib.eachDefaultSystem (
        system:
          let
            pkgs = import nixpkgs { inherit system; overlays = [ nur.overlay ]; };
            blog = pkgs.callPackage ./blog.nix {
              inherit pkgs;
              nur = pkgs.nur;
              thirdparty = [
                {
                  name = "uikit";
                  path = "${uikit-src}/src";
                }
              ];
            };
          in
            rec {
              packages = { inherit (blog) generator generator-with-thirdparty ci shell; htop = pkgs.htop; };
              defaultPackage = blog.generator-with-thirdparty;
              apps.compile =
                flake-utils.lib.mkApp { drv = blog.ci.compile; exePath = ""; };
              defaultApp = apps.compile;
              devShell = blog.shell;
            }
      );
}

So that’s a lot! And notice, its even simplified through the use of flake-utils. But it does come with some structure, which we will inspect more closely now.

So what do we see here?

Pure imports

In flake mode nix expects the flake files to only contain the top level attributes description::string, inputs::set , outputs::{inputs..}->set. Notice, flakes are not a function at the top level anymore.
What does that imply? First, there are no arguments to be given by command line (via –arg/–argstr). This means no nixpkgs attribute to be passed from a higher level. Instead, your dependencies are expected to be included in inputs you give. inputs is a set where each attribute is the identifier of one input.

The corresponding part in the above flake.nix is

  description = "Flake utils demo";

  inputs.nixpkgs.url = "github:nixos/nixpkgs";
  inputs.nur.url = "github:nix-community/NUR";
  inputs.flake-utils.url = "github:numtide/flake-utils";
  inputs.uikit-src = {
    url = "https://github.com/uikit/uikit/archive/v3.5.8.tar.gz";
    flake = false;
  };

Each of these input follows this structure:

inputs.<inputname> = {
  url = git+<url-to-git-repo>[?ref=****[&rev=****]]
      | (github|gitlab):<user>/<repo>[/<ref> | ?ref=****[&rev=****]];
      | <known-flake>[/<ref> | ?ref=****[&rev=****]]
      | /absolute/path
  flake = true | false; # default = true
}

Input url

use git+<url> to checkout a git repository at <url>

use /local/absolute/path to load a local source

use gitlab:<user>/<repo>/github:<user>/<repo> to shortcut gitlab or github repositories

use <known-flake> to shortcut to a globally defined alias

known-flake

nix will manage a registry of named flakes that can be included as is and are a shortcut to another repo

by default following global registry items are defined in this repo and available in every nix installation:

global flake:blender-bin github:edolstra/nix-warez
global flake:dwarffs github:edolstra/dwarffs
global flake:hydra github:NixOS/hydra
global flake:nimble github:nix-community/flake-nimble
global flake:nix github:NixOS/nix
global flake:nixops github:NixOS/nixops
global flake:nixos-hardware github:NixOS/nixos-hardware
global flake:nixos-homepage github:NixOS/nixos-homepage/flake
global flake:nur github:nix-community/NUR
global flake:nixpkgs/release-19.09 github:edolstra/nixpkgs/release-19.09
global flake:nixpkgs github:NixOS/nixpkgs
global flake:templates github:NixOS/templates
global flake:patchelf github:NixOS/patchelf

read more about registry on the wiki

Specifying a commit or branch

by default, the main branch of the repository pointed to will be used as to provide the dependency flake.

Nix will also update dependencies to the latest commit on nix flake update calls

to pin repositories to a certain commit or override the default branch append a ?ref=commit-ref or ?<branch> respectively

Non flake inputs

nix will try to interpret inputs as flakes, to suppress that, set <inputname>.flake = false

Overriding dependency inputs

traditionally, if one wanted a dependency to use a certain nixpkgs version (i.e. to make use of binary caches) one would specify a certain nixpkgs argument while import’ing. Flakes allow the same thing by setting inputs.<inputname>.inputs.<inputname>.follows = <local inputsname>

See also the reference below for more information and examples.

So far so good, but we have only discussed about 10 lines of the flake s far…

Pure outputs

As mentioned above, nix flakes do not serve as top level function anymore. That does not mean there is no function involved. For flakes the output is the result of a function as well.

outputs = { self, nixpkgs, nur, flake-utils, uikit-src }:
  ...

This third and last attribute takes as arguments all the inputs defined earlier. But unlike classical nix approaches, only a certain set of attributes are allowed and expected as the result of this function.
An extensive list of output attributes is provided below.

Generally the format for these is:

<kind>.<arch>.<output> = ...;

or concretely

packages."x86_64-darwin".generator = blog.generator;

Notice how the architecture and OS of the target platform is explicitly encoded, allowing for specializations or exclusive packaging for specific platforms. This also means that the used nixpkgs needs to be instantiated for all desired platforms. In practice one will want to use a shortcut for that and while this is used for all our projects packaged for NGI so far, for my blog this shortcut is provided by flake-utils.

Using these, defining the outputs for this blog just becomes:

  let
    in
      flake-utils.lib.eachDefaultSystem (
        system:
          let
            pkgs' = import nixpkgs { inherit system; overlays = [ nur.overlay ]; };
            blog = pkgs'.callPackage ./blog.nix {
              pkgs = pkgs';
              nur = pkgs'.nur;
              thirdparty = [
                {
                  name = "uikit";
                  path = "${uikit-src}/src";
                }
              ];
            };
          in
            rec {
              packages = {
                inherit (blog) generator generator-with-thirdparty ci shell;
              };
              defaultPackage = blog.generator-with-thirdparty;
              apps.compile =
                flake-utils.lib.mkApp { drv = blog.ci.compile; exePath = ""; };
              defaultApp = apps.compile;
              devShell = blog.shell;
            }
      );

To take this apart again, we face three different sections here.

1. flake-utils.lib.eachDefaultSystem, like its name suggests will provide the attribute set in its scope for all default systems (basically linux and darwin on x86_64 hardware). Hence, we don’t need to worry about these tags.
2. The actual output still needs to conform to flakes’ expected attributes. In this case we set packages, defaultPackage, apps, defaultApp and devShell. There are even more most importantly overlay which are referenced below
3. third one is the middle part and shows an important aspect of nix flakes. Because there is no access to impure variables (--arg, environment variables, the current system, etc.), nixpkgs must be imported explicitly for each system. Additionally, this gives the opportunity to apply any possible overlay. In the case of this blog everything is defined in an external blog.nix file and imported from there.

Output attributes

$ nix-shell -p nixUnstable

Coming back to the outputs. packages is the actual output that is used with nix shell and nix build commands in particular (yes there are/will be a lot more).

To open a shell with specific packages available (alike nix-shell -p) run:

$ nix shell ./#generator-with-thirdparty

or from an external repo:

$ nix shell "gitlab:pi-lar/neuropil#generator-with-thirdparty"

$ nix shell nixpkgs#{htop,nodePackages.act,tealdeer}

apps is used together with nix run, similarly to npm run. You can define binaries that can then be run directly without explicitly building them first. This allowed me to change the somewhat cryptic $(nix-build -A packages.x86_64-linux.ci.compile --no-out-link) to nix run ./#compile in the CI script. Actually given that defaultApp point to the same compile attribute, the argument ./#compile could be elided paying with decreased clarity.

devShell is another special one used in combination with nix develop which opens a bash shell with the given derivation prepared.

Of course, project shells using for example mkShell do also still work (and are used here as well). Either by using nix develop with such a derivation or nix shell in combination with buildEnv.

Composing flakes

In the last two sections I presented solutions for six of the seven problems that were identified in the beginning. That leaves us with:

7. poor composability

Actually we’re almost there.. Remember how I mentioned the possibility to inject overlays when importing nixpkgs? Composing multiple sources of packages is easy if one can just drop them in there.
Moreover, there are conventional output arguments overlay and overlays that make discovering the fakes; overlay(s) a bit easier.

{
  description = "Flake utils demo";

  inputs.flake-utils.url = "github:numtide/flake-utils";
  # assuming that this flake exports an overlay attribute
  inputs.someflake.url = "foo.com/bar";

  outputs = { self, nixpkgs, flake-utils, someflake }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        artifacts = final: prev: {
          my_hello = prev.hello;
        };
        pkgs = import nixpkgs {
          overlays = [ someflake.overlay artifacts ];
        };
      in
      # and here is the core of the consumer flake
      {
        defaultPackage = pkgs.my_hello;
        overlay = artifacts;
      }
    );
}

Adapted from this answer on the forum this example shows how someflake’s overlay can be imported, and how to define your own overlay in the same manner. Here it is used rather uselessly, but one can imagine to define outputs in a separate file as overlay, that can then be plugged for use and export in the flake.

Retrospective

So I changed the nix setup of this blog to flakes… Was it worth it?

Of course it depends. For the project that is this blog the benefits of committing to flakes are rather unconvincing. No one uses this as a dependency or wants to discover it as such. I am the only one developing this, so I might have my own conventions to import things, pin my dependencies and define my interface. Also, maybe I do want to use CLI arguments to nix.

Personally though it did convince me (obviously). I am a great supporter of the Nix philosophy and the Flakes do make a great addition to the ecosystem. I first got into contact with them when starting to work on the NGI0 project at which time Eelco had just written the first public post about flakes and the documentation was sparse to say the least… But as a new thing it still caught my interest. Looking for projects outside NGI this blog seemed like a great candidate to apply this new technology to as it posed literally no risk whatsoever if it didn’t work out. It gave me a chance to make myself comfortable with the technology and dig deeper to write this article.

Now I really like how this turned out, especially the nix run part comes in handy here and the native caching might come in handy for quicker direnv shells.

One downturn is still that nix develop is not able to open zsh shells so the usual mkShell technique does not work with flakes and one needs to fall back to nix-shell or buildEnv alternatives. Also there are still flags and a special nix binary required to make use of the flake related nix features.

References

Flake reference conventions

Nix commands will eventually reference outputs. There are multiple ways how these can be referenced depending on where they come from:

A flake reference is actually structured similar to an URL:

<flake url>[?<git ref/rev>][#<output>]
    │
    ├── git+<url>
    ├── gitlab:<user>/<repo> or github:<user>/<repo>
    ├── /local/absolute/path
    ├──./relative/path (only as an argument to `nix` commands)
    └── a known source in the registry such as `nixpkgs`

If git repositories are pointed to directly query parameters can be added to select a specific branch and/or commit.

As nix has not yet adapted to some git providers renaming their master branches to main you might occasionally be confronted with an error similar to

fetching Git repository 'https://gitlab.com/pi-lar/neuropil'fatal: couldn't find remote ref refs/heads/master

To resolve this the actual branch main or any tag might be explicated using the ref argument:

git+https://gitlab.com/pi-lar/neuropil.git?ref=main

If using the abbreviated GitLab/GitHub urls this can be expressed like this:

gitlab:pi-lar/neuropil/main

Similarly, if an exact commit is targeted, this can be pointed to using the rev argument

   gitlab:pi-lar/neuropil?rev=1043b9c1736c2a9b92938f0304ddb55ce4ab68a6
or gitlab:pi-lar/neuropil/1043b9c1736c2a9b92938f0304ddb55ce4ab68a6

The output is finally selected in the format known from the -A argument of old nix commands such as nix-shell:

   nixpkgs#haskellPackages.pandoc
or .#generator-with-thirdparty      (used to build this blog's generator)

Output attributes

Following is a quick reference over all flake output attributes.

{
    # Executed by `nix flake check`
    checks."<system>"."<attr>" = derivation;

    # Executed by `nix build .#<name>`
    packages."<system>"."<attr>" = derivation;

    # Executed by `nix build .`
    defaultPackage."<system>" = derivation;

    # the derivation that is prepared when running `nix develop`
    # defaults to defaultPackage."<system>"
    devShell."<system>" = derivation;

    # Executed by `nix run .#<name>
    apps."<system>"."<attr>" = {
      type = "app";
      program = "<store-path>";
    };
    defaultApp."<system>" = { type = "app"; program = "..."; };

    # TODO: Only seen used in connection with nixpkgs
    legacyPackages = TODO;

    # overlay to easily override/packages in the package set of the importer
    overlay = final: prev: { };

    # TODO: Same idea as overlay but several.
    overlays."<attr>" = final: prev: { };

    # Additionally to packages also modules for Nixos can be provided (Flakes can be used with `nixos-rebuild`)
    nixosModule = Module definition;

    # TODO: Same idea as nixosModule but several
    nixosModules."<attr>" = Module definition;

    # Attrset of nixos configurations by hostname.
    nixosConfigurations."<hostname>" = {};

    # Hydra checks
    hydraJobs."<attr>"."<system>" = derivation;

    # Used by `nix flake init -t <flake>`
    defaultTemplate = {
      path = "<store-path>";
      description = "template description goes here?";
    };
    # Used by `nix flake init -t <flake>#<attr>`
    templates."<attr>" = { path = "<store-path>"; description = ""; );
  }

As always, thank you for reading this far I hope you enjoyed this article!

It helped me a lot to structure my knowledge about nix flakes and I am glad to share this to a wider audience.

If you have comments drop me a mail, or comment publicly on the media you where found this. For additions or corrections and critique feel free to use GitHub issue tracker.

This blog is hosted by GitHub Pages. GitHub Pages, aside static HTML content, only supports building Jekyll pages natively. At some point a website like this one has to be generated, i.e. turned into static HTML content. This means Markdown has to be converted to HTML static pages (like the homepage), an archive has to be created, and RSS and ATOM feeds have to be produced. In this blog’s case, this is done by a Haskell DSL called Hakyll. More details about that can be found in prior articles.

In order to keep everything in the same place, I use GitHub Actions as a mean to automatically build and publish this blog whenever I push updates or merge pull requests.

Was this not an article about Nix?

Yes indeed it is. In order to create reproducible builds of this blog I employ the Nix package manager. But in order…

What is Nix?

Nix is sort of a hybrid tool between a package manager and a build system. The Nix homepage states:

Nix is a powerful package manager for Linux and other Unix systems that makes package management reliable and reproducible. It provides atomic upgrades and rollbacks, side-by-side installation of multiple versions of a package, multi-user package management and easy setup of build environments.

Sounds… good? What does all of that mean though?

Nix (the package manager) is built around the functional programming language nix (what a coincidence, ha?). A program, library, etc. is built by Nix as the output of a pure function. Runtime/buildtime dependencies go in mix with the sources of the package, are built in an isolated space, and output an executable or whatever you want to build. Such a function is called a Derivation.

Everything that should be built with Nix has to be described as such a function. In fact, the whole set of packages that is there is just a giant collection of functions, each defined in their own files.

The isolated manner in which the packages are built also shows in the way they are managed. All packages are installed as such:

/nix/store/6fgjp1wsv1w44f890f6kvwywjnc32svr-zoxide-0.2.1/bin/zoxide

Lets take this apart:

/nix/store/

This is the nix store all packages are contained in this folder

6fgjp1wsv1w44f890f6kvwywjnc32svr

This is a hash of the producing function, it takes into account the function itself, and its inputs

It also easens binary caching.

zoxide

the package’s name

0.2.1

the packaged version

/bin/zoxide

the actual executable

actually the file structure of packages is similar to what you would normally find in places like /usr or ~/.local/

Yet, while that package might exists in the store, this does not mean it is installed or usable. — Huh?

You see, the hash we have seen right now is there for a reason. It allows to have multiple versions of one and the same program at the same time, even with the same version. Normally all packages are built according to one specific state of the nixpkgs repo. .nix files get evaluated by writing a build instruction file (.drv). This resolves all dependencies (building them if they do not exist yet), prepares the package’s source and combines those into a hash. If one instead builds it with a newer version where some dependency was updated or changed, not only this derivation hash changes, but it also happens to change the target derivation’s hash, forcing it to be rebuilt using the new dependency. The same happens if sources change. As a result, many different versions of a package can coexist in the store.

How to install/use packages?

Global installation

There are different ways to achive this. Using nix locally, the easiest but least idiomatic would be:

$ nix-env -iA zoxide

This installs zoxide¹) globally for the current user, by linking all of zoxides contents to a special directory (~/.nix-profile/). The bin contents of this directory are on the $PATH</code> making the package’s binary available to the user.</p> <p>As mentioned this is not idiomatic Nix, which focuses on declerativity. If you want something installed, consider using <a href="https://github.com/rycee/home-manager"><code>home-manager</code></a> or add packages to your <code>configuration.nix</code> if you are on <a href="https://nixos.org/nixos">NixOS</a>.</p> <h4 class="y-header" id="local-usage"><span>Local usage</span><a href="#local-usage" class="anchor las la-anchor" title="local-usage"></a></h4> <p>If you need a program but do not want to install it because you don’t need it often and you don’t want to clutter your <code>$PATH, or it’s a project specific tool you can make use of either nix run or nix-shell.

$ nix run nixpkgs.git

This will drop you into a Bash shell with git available.

$ nix-shell -p git

This will essentially do the same but can do much more.

• nix-shell accepts multiple packages

• can be used as a shebang (see)

• can drop you into development environments of a specific derivation (see)

• can be used to make development environments for a project.

Let’s focus on the last point for this article.

Nix on this blog

This blog uses a shell environment in which I have all the tools I need at hand. Let’s look at the structure file by file.

default.nix

By default, Nix commands such as nix-build read the default.nix file in the directory they are executed from. In my projects, I use them to define and export everything that might be useful. This might include build tools, language environments, scripts or shells.

Generally my default.nix file looks like this:

{pkgs ? import (if pin == false then <nixpkgs> else pin) {},
 pin ? ./nixpkgs.nix, ... }:
with pkgs;
let

<some packages and configuration>

shell = mkShell {
​    name = "<name>";
​    buildInputs = [ <packages I want in my shell> ];
​    shellHook = '' # shell command to be executed when I enter the shell
​    '';
}
in
{
​    inherit shell;
​    inherit packageA, packageB, ...;
}

This exposes a shell environment for nix-shell and programs to be executed directy using nix-build.

So what about this blog’s default.nix?

let
  all-hies = import (builtins.fetchTarball "https://github.com/infinisil/all-hies/tarball/master") {};
in
{pkgs ? import (if pin == false then <nixpkgs> else pin) {},
 pin ? ./nixpkgs.nix, ... }:
with pkgs;

let

Aside from the default, I only priorly import all-hies, which is a project that maintains the Haskell Ide Engine on Nix.

# -------------- Utils -------------

nur = import (builtins.fetchTarball "https://github.com/nix-community/NUR/archive/master.tar.gz") {
​    pkgs=pkgs;
};

script = {...} @ args: nur.repos.ysndr.lib.wrap ({
  shell = true;
} // args);

This are utilities I might use. NUR is the nix version of arch’s AUR, albeit not nearly as active sadly. From my own collection, I use the shell script which helps me putting together runnable scripts.

# ------------- Haskell ------------

hie = all-hies.selection { selector = p: { inherit (p) ghc865; }; };
myHaskellPackages = haskell.packages.ghc865.extend( self: super: { });

These lines prepare the Haskell environment I use here. I do not override anything from GHC so this is a bit superflous.

# ------------ dist ---------------

thirdparty = linkFarm "thirdparty" [
  {
​    name = "uikit";
​    path = (fetchTarball "https://github.com/uikit/uikit/archive/v3.2.4.tar.gz") + "/src";
  }
];

I import all third party tools (only uikit currently) into the nix store.

# ------------- generator -----------

generator = myHaskellPackages.callCabal2nix "Site" ./generator {};

generator-with-thirdparty = generator.overrideAttrs(old: {
  nativeBuildInputs = old.nativeBuildInputs or [] ++ [makeWrapper];
  installPhase = old.installPhase + "\n" + ''
​    wrapProgram $out/bin/generator --set THIRDPARTY ${thirdparty}
  '';
});

Then, I define the generator as a derivation from its Cabal file and the corresponding “Site” executable key defined in ./generator/. generator-with-thirdparty makes what I imported as thirdparty content available under the $THIRDPARTY environment variable.

# --------------- Commands ----------------

generate-website = script {

  name = "generate-website";
  paths = [generator-with-thirdparty git];
  script = ''
​    generator rebuild
  '';

};

generate-website is supposed to be a build command which, at this state, only runs generator rebuild.

# ---------------- Shell ------------------

haskell-env = myHaskellPackages.ghcWithHoogle (
  hp: with hp; [ cabal-install ] ++ generator.buildInputs );

shell = { enable-hie ? false }: mkShell {
  name = "blog-env";
  buildInputs = [
​    # put packages here.

​    # generator
​    haskell-env

​    (lib.optional (enable-hie) hie) # optionally setup hie
  ];
  shellHook = ''

​    export THIRDPARTY="${thirdparty}"

​    export HIE_HOOGLE_DATABASE="${haskell-env}/share/doc/hoogle/default.hoo"
​    export NIX_GHC="${haskell-env}/bin/ghc"
​    export NIX_GHCPKG="${haskell-env}/bin/ghc-pkg"
​    export NIX_GHC_DOCDIR="${haskell-env}/share/doc/ghc/html"
​    export NIX_GHC_LIBDIR=$( $NIX_GHC --print-libdir )

  '';
};

Finally, I bundle everything together to build an environment with ghc and a Hoogle index containing the generators dependencies as well as cabal-install for reasons.

The shell has this Haskell environment and optionally the hie executable exposed. It also sets some exports that hie needs to properly function as well as the $THIRDPARTY variable to build the blog:

in {

  inherit shell generator generate-website ;
  ci = {
​    compile = generate-website;
  };

}

All important parts are exported.

shell.nix and nixpkgs.nix

The first file is used by nix-shell by default. All it does is call the shell attribute of the default.nix and controlling if hie is added, making use of the lazy nature of nix: hie will not get evaluated unless enabled.

{ pin ? null, enable-hie ? false } @ args:

(import ./default.nix args).shell { inherit enable-hie; }

The nixpkgs.nix file defines a common snapshot of the nixpkgs repo, and is therefore important to ensure everything works on other machines.

{}: import (builtins.fetchTarball {
  url = "https://github.com/NixOS/nixpkgs/archive/88d9f776091.tar.gz";
  sha256 = "sha256:0z8a0g69fmbbzi77jhvhwafv73dn5fg3gsr0q828lss6j5qpx995";
}) {}

Nix and GitHub Actions

Nix has some integration with GitHub Actions through install-nix, an action that installs nix •_•

With nix installed, I run $(nix-build -A ci.compile --no-out-link) to make Nix build the blog generator and rebuild the blog’s content into build/site. This works because nix-build --no-out-link will just print the path of the resulting package to stdout, which in this case is only an executable script produced by the script function above. The next step is to take that content and push it to the deployment branch.

See more…

Epilog

I see this has become more of a roundup about Nix. Nix is huge though.. and this article does not try to capture everything (obviously). From reading this, I hope you have a rough idea of what nix does and how it was applied here. If you knew Nix already, maybe you found something new or interesting among this pile of words and snippets. If you did not know Nix before, I hope this article was still of interest to you.

Thank you for getting here! I hope you enjoyed it and have learned something. If you have questions, improvements, or any other comment, do not hesitate to get in touch on GitHub.

• Part 01: Hakyll basics and context
• Part 02: A collection of Custom Fields this article

In the previous article I introduced the basic mechanisms of Hakyll. Part of this introduction was that every content lives in Hakyll as an Item. An Item is created by loading some content from a file (or a wildcarded list of files). In most cases these are Markdown files. For example, this is what loading an Item looks like:

match "posts/**.md" $ do
    route $ setExtension "html"
    compile
        $   pandocCompiler
        >>= loadAndApplyTemplate "templates/post.html" postCtx

For each *.md file inside posts/ an .html file is generated by compiling the document using pandoc (which turnsthe markup it into html) and including this content in a template, namely the templates/post.html template, by loading the template and applying it with a given Context.

What is a Context? Contexts deliver meta information to the templating engine. The information is constructed for each document individually and by default contains (in that order)

1. A body field: current content of the document (usually after conversion using pandoc)
2. Metadata fields defined in the document’s frontmatter
3. A url urlField: url of the final document
4. A path pathField: path of the original file
5. A title titleField: title defined in frontmatter

These variables can in turn be used in the template and will be substituted by Hakyll. You can read more about templates, again, on the official Hakyll Website

Custom fields

Generally Contexts are the only way to include dynamic content. Anything that is not plain text from the template or the document comes through a context Field. Fields hold data that is individually derived from each Item i.e. document. They are composed to form a full context which is fed into the templating engine. As Fields are implemented as Monoids composing gets very easy:

context :: Context String
context = mconcat
    [ defaultContext
    , pathField "sourcefile"
    ]
-- or
context = defaultContext
       <> pathField "sourcefile"
       <> constField "field" "value"

Hakyll comes with a range of default fields:

• boolFields
• constField
• ’modificationTimeField
• and some more that you can find here

While setting up the structure of this blog I found the “need” of having some more fields available though. This is how I started to adapt some implementations I found on the internet, and developed my own. Find those in generator/Fields.hs.

Most fields are created by giving an implementation of (Item a -> Compiler String) to the following field function.

field :: String -> (Item a -> Compiler String) -> Context a

peekField

I showed this field already in the previous article, it is one of the first fields I made to understand the logic behind them. It simply takes the fist length words and makes them availlable to the template engine under key. As templates are applied after pandoc converted the document to html I needed to take the original content from an earlier created snapshot of the document.

peekField
    :: String           -- ^ Key to use
    -> Int              -- ^ length to peak
    -> Snapshot         -- ^ Snapshot to load
    -> Context String   -- ^ Resulting context
peekField key length snapshot = field key $ \item -> do
    body <- itemBody <$> loadSnapshot (itemIdentifier item) snapshot
    return (peak body)

    where peak = T.unpack . T.unwords . take length . T.words . T.pack

Problem with this is undeniably that it does not that markup into account at all. That way Code blocks are included without any styling and look very bad.

Git Fields

I wanted to allow readers of my blog to follow the history of an article. As the sourceof this blog is hosted on GitHub using GitHub’s history view would be an easy way to achieve this, I thought.

-- Git related fields
--------------------------------------------------------------------------------
data GitVersionContent = Hash | Commit | Full
     deriving (Eq, Read)

instance Show GitVersionContent where
    show content = case content of
        Hash -> "%h"
        Commit -> "%h: %s"
        Full -> "%h: %s (%ai)"

-- Query information of a given file tracked with git
getGitVersion :: GitVersionContent -- Kind of information
              -> FilePath          -- File to query information of
              -> IO String         --
getGitVersion content path = do
    (status, stdout, _) <- readProcessWithExitCode "git" [
        "log",
        "-1",
        "--format=" ++ (show content),
        "--",
        "src/"++path] ""

    return $ case status  of
        ExitSuccess -> trim stdout
        _           -> ""

    where trim = dropWhileEnd isSpace

-- Field that contains the latest commit hash that hash touched the current item.
versionField :: String -> GitVersionContent -> Context String
versionField name content = field name $ \item -> unsafeCompiler $ do
    let path = toFilePath $ itemIdentifier item
    getGitVersion content  path

-- Field that contains the commit hash of HEAD.
headVersionField :: String -> GitVersionContent -> Context String
headVersionField name content  = field name $ \_ -> unsafeCompiler $ getGitVersion content  "."

With the current implementation of getGitVersion I am able to get the latest commit that changed any given document. It spawns a git process and reads its output afterwards.

I can even choose from predefined formats:

• Hash gives the Commit’s hash,
• Commit the Hash+Message
• Full Hash+Message+Time

Although I think for many pages the number of git invocations might lead to significantly longer build times, for the time being it works rather well.

readTimeField

Another really simple but usefull function that naïvely computes the reading time of a document. Essentially it counts the words of the docutment snapshot and devides it by the average reading velocoty of about 200 words/min.

readTimeField :: String -> Snapshot -> Context String
readTimeField name snapshot = field name $ \item -> do
    body <- itemBody <$> loadSnapshot (itemIdentifier item) snapshot
    let words = length (T.words . T.pack $ body)
    return $ show $ div words 200

publishedGroupField

Adapted from biosphere.cc

This field is actually a listField. It is used on the archive page to group posts by year. It is also a quite intimidating one at first sight, but it works perfectly, so 🤷🏼‍♂️.

It works by first extracting the year out of the date of every post using extractYear. It then groups the resulting tuples by the year item and merges the groups.

Have I already mentioned that working with elements contained in a Compiler Monad is incredibly weird? - It is!

Anyway, in the end the template can use the list referenced by name that exposes a field year containing the actual year of a list of posts that is accessable as posts and that have a given postContext applied.

publishedGroupField :: String           -- name
                    -> [Item String]    -- posts
                    -> Context String   -- Post context
                    -> Context String   -- output context
publishedGroupField name posts postContext = listField name groupCtx $ do
    tuples <- traverse extractYear posts
    let grouped = groupByYear tuples
    let merged = fmap merge $ grouped
    let itemized = fmap makeItem $ merged

    sequence itemized

    where groupCtx = field "year" (return . show . fst . itemBody)
                  <> listFieldWith "posts" postContext (return . snd . itemBody)

          merge :: [(Integer, [Item String])]  -> (Integer, [Item String])
          merge gs = let conv (year, acc) (_, toAcc) = (year, toAcc ++ acc)
                      in  foldr conv (head gs) (tail gs)


          groupByYear = groupBy (\(y, _) (y', _) -> y == y')

          extractYear :: Item a -> Compiler (Integer,  [Item a])
          extractYear item = do
             time <- getItemUTC defaultTimeLocale (itemIdentifier item)
             let    (year, _, _) = (toGregorian . utctDay) time
             return (year, [item])

concatField

An actual use of functionFields – yey.

It is used to dynamically apply a different header in the base template. There I construct a path to a partial template using another constField (item-type) which, for posts will display a customized header

$partial(concat("templates/includes/",item-type, "-header.html"))

The implementation is simply:

concatField :: String -> Context String
concatField name = functionField name (\args item -> return $ concat args)

FunctionFields

For the unintroduced, function fields are defined as:

functionField :: String                                  -- name
              -> ([String] -> Item a -> Compiler String) -- actual function
              -> Context a

When used, like in the above example (concat(“hello,” " “,”world")) the function

(\args item -> return $ concat args)

is evaluated where args contains exactly the arguments given (args == [“hello,”" “,”world"]). as well as the Item it is used on i.e. the document. You can now do all you want with the document’s body and arguments given. Apparently though calling the functionField with the same field as argument is not possible.

For another explanation see also Beerend Lauwens’ post

plainTocField

Although I have written another implementation of this one that allows additional classes applied on certain elements I would like to show this anyway. To generate a simple table of contents pandocs builtin toc gereator is leveraged. I load the documents body, give it to panodc to parse it into a Pandoc _ [Block] and use that to write html with the template table − of − contents. The output of that is only the table of contents and nothing else.

plainTocField :: String -> Int -> String -> Context String
plainTocField name depth snapshot = field name $ \item -> do

    body <- loadSnapshot (itemIdentifier item) snapshot

    let writerOptions = def
            {
              writerTableOfContents = True
            , writerTOCDepth = depth
            , writerTemplate = Just "$table-of-contents$"
            }
        toc = case runPure (readHtml defaultHakyllReaderOptions
                                     (T.pack $ itemBody body))
               >>= \pandoc -> runPure ( writeHtml5String writerOptions pandoc) of
                   Left err    -> fail $ ""
                   Right item' -> T.unpack item'

    return toc

Final Words

As I have already mentioned above, I am not the expert in Haskell that I’d like to be. But writing this blog’s engine has taught me much and was a great excuse to dive Haskell and understand its ideas.

I hope you enjoyed this post anyway. For ideas critic and about this use the GitHub issue tracker

Hakyll is tool to build static webpages and written in Haskell borrowing part of its name from it and the other from the renown static site engine Jekyll. The fundamental difference is that instead of being a standalone program, Hakyll is implemented as a library offering a rich DSL to define the contents of a website. The DSL can be used to include static assets, compile pages using templates and even generate whole pages by itself i.e. RSS and ATOM feeds. As Hakyll is written in Haskell combining it with the great text conversion tool pandoc is a low hanging fruit. The combination of both allows compiling from several different file formats including (an extended version of) Markdown, RST and Emacs Org-Mode.

• Part 01: Hakyll basics and context (this article)
• Part 02: Custom Fields

Setup

Setting up Hakyll in general using stack or cabal is already documented extensively. My take on this project involves using nix as dependency/package manager. How I set up this blog using nix will also be discussed another time.

Basic Concepts

In this article I would like to introduce the basic concepts of Hakyll in an applied way.

The heart of this page is generator/Main.hs.

At it’s bare minimum a basic Hakyll generator looks like this:

config :: Configuration
config = defaultConfiguration {
    -- config overrides such as the path to the page content source
    -- and compiled item's destination
}

-- A context extending the default context by a date field
postCtx :: Tags -> Tags -> Context String
postCtx tags category = dateField "date" "%B %e, %Y"
                     <> defaultContext

main :: IO ()
main = hakyllWith config $ do
    match "posts/**.md" $ do
        route $ setExtension "html"
        compile
            $   pandocCompiler
            -- applies the post template
            >>= loadAndApplyTemplate "templates/post.html" postCtx
            -- embeds the rendered post into the page template
            >>= loadAndApplyTemplate "templates/default.html" postCtx
            >>= relativizeUrls

    create ["archive.html"] $ do
            route idRoute
            compile $ do
                posts <- recentFirst =<< loadAll "posts/**"
                let postCtx = postCtx tags categories
                let archiveCtx = listField "posts" postCtx (return posts)
                            <> constField "title" "Archive"
                            <> customBaseContext

                makeItem ""
                    >>= loadAndApplyTemplate "templates/archive.html" archiveCtx
                    >>= loadAndApplyTemplate "templates/default.html" archiveCtx
                    >>= relativizeUrls

This will render all markdown files under posts/ to html files under our build route and setup an archive site listing all posts.

In this snipped we see several things lets go through them individually:

1. config
2. matching routes
3. compiling with pandoc
4. templates
5. contexts

config

config :: Configuration
config = defaultConfiguration { }

This sets up the runtime configuration of hakyll itself. With it we can override among others the folder in which Hakyll searches for its content and where the result should be stored. All available options can be found inside the documentation

matching routes

match "posts/**.md" $ do
    route $ setExtension "html"
    compile
        $   pandocCompiler
        -- applies the post template
        >>= loadAndApplyTemplate "templates/post.html" postCtx
        -- embeds the rendered post into the page template
        >>= loadAndApplyTemplate "templates/default.html" postCtx
        >>= relativizeUrls

In order to create a page from an existing source we use the match function. It takes a FilePath and a function telling it what to do with the file. route will then map the matched items to the paths created by setExtension which unsurprisingly only changes the resulting files extension to html. There are a few other functions that can be used here (basically all functions that return Routes). Those can be used to do more advanced route editing such as creating fancy paths using index.html¹.

But what does this function actually do in particular? 1. It searches for all markdown files under the posts/ directory. 2. Defines their final route 3. Compiles them to html using pandoc 4. Embedds them into a post template 5. Embedds raw posts into the default page template 6. And finally relativizes urls, which is a fancy function that keeps track of the resources referenced locally and keeps their links up to date.

Compiling with Pandoc

In the snippet above the default pandocCompiler function is used to read the content of the file and transform it into HTML using Hakyll’s default options for pandoc. Aside pandocCompiler there are a few more low level functions available that allow deeper customization in the regards of which pandoc templates are used, which extensions activated and so forth. There are also pandocCompilerWithTransform and pandocCompilerWithTransformM that allow editing the intermediate parsed content pandoc uses internally. At this point rich postprocessing can be applied, just alike the usual pandoc filters. The only grain is that existing pandoc filters (i.e. pandocfilters or panflute) cannot be easily applied with Hakyll.

Creating Routes

Additionally to matching existing files and compiling them, one can also generate fully independent files using the create function.

create ["archive.html"] $ do
    route idRoute
    compile $ do
        posts <- recentFirst =<< loadAll "posts/**"
        let postCtx = postCtx tags categories
        let archiveCtx = listField "posts" postCtx (return posts)
                    <> constField "title" "Archive"
                    <> customBaseContext

        makeItem ""
            >>= loadAndApplyTemplate "templates/archive.html" archiveCtx
            >>= loadAndApplyTemplate "templates/default.html" archiveCtx
            >>= relativizeUrls

This creates a files archive.html which is built using the compile function that basically wraps an Item a in the Compiler monad. The corresponding item is created using makeItem that itroduces an empty String Item that is enriched first using the archive template and subsequently the default page template.

Notice the use of loadAll that makes the set of all posts availlable inside he compile scope. Most importantly though are the both contexts, especially the archiveCtx that makes the posts available to the template engine as a list of postCtxs.

Contexts

Contexts contain the meta information that is available to the templating engine when building an Item. Thus allowing the usage of the computed value in the template files. A context holds a number of fields which are contexts as well. Each field’s value is created for every compilation item (this is every item the context will be applied on, usually individual posts).

Take for example the following field definition:

peekField
    :: String           -- ^ Key to use
    -> Int              -- ^ length to peak
    -> Snapshot         -- ^ Snapshot to load
    -> Context String   -- ^ Resulting context
peekField key length snapshot = field key $ \item -> do
    body <- itemBody <$> loadSnapshot (itemIdentifier item) snapshot
    return (peak body)

    where peak = T.unpack . T.unwords . take length . T.words . T.pack

This is a very simple field once created to serve as my own version of a teaser field. As you can see a field is created from a key and a function mapping an item to a string. In this case the item’s body is extracted from a snapshot of the item’s content. Then length words are taken and returned. (If you are in the actual need of a teaser field I would advice you though to use the teaserField).

Yet what this example demonstate is the integral importance of Items in Hakyll.

Items

Items are a simple datatypes that wrap a body of some type a (usually String) and an identifier.

Most of Hakyll’s functions that deal with content are working with Items. As seen above Items can also be stored in snapshots and retrieved later on. Especially in fields Items contain the content from which the desired field can be derived from.

I don’t mean to write an extensive documentation of all the concepts in Hakyll, an complete introductory tutorial as well as links to other peoples takes can be found at Jasper’s (the founder of Hakyll) website.

In a follow up article I would like to share a compilation of custom fields that I created or adapted from other blogs.

Hej, and welcome!

You appear to have successfully stumbled upon my Blog, nice to have you here.

This blog will be home to all the ideas that I mind worthy to be yelled out in the wild. Expect random thoughts posts about Nix, rust, contents of my studies as well as the internals of this blog.