Bit Fields, Byte Order and Serialization

by Wu Yongwei

n order to store data most efficiently, the C language has supported bit fields since its early days. While saving a few bytes of memory isn’t as critical today, bit fields remain widely used in scenarios like network packets. Endianness adds complexity to bit field handling – especially since network packets are typically big-endian, while most modern architectures are little-endian. This article explores these problems and their solutions, including my reflection-based serialization project.

Memory layout of bit fields

The memory layout of bit fields is implementation-defined. In a typical little-endian environment, bit fields start from the lower bits of the lower byte and extend toward higher bits and bytes. In a typical big-endian environment, bit fields start from the higher bits of the lower byte and extend toward lower bits and higher bytes.

Let’s consider a practical scenario. Suppose we want to use a 32-bit integer to store a date. How should we achieve this? A simple approach is to store the number of days from a fixed point of time (e.g. 1 January 1900). We can calculate the number of years that can be expressed as follows:

However, with this approach, extracting specific year, month, and day information becomes very cumbersome. A simpler way is to store the year, month, and day as bit fields. We can define the following struct, using only 32 bits:

  struct Date {
    int      year  : 23;
    unsigned month : 4;
    unsigned day   : 5;
  };

Our intention is to use a 23-bit signed integer for the year (ranging from -4,194,304 to 4,194,303), a 4-bit unsigned integer for the month (0–15, covering legal values 1–12), and a 5-bit unsigned integer for the day (0–31, covering legal values 1–31). This representation is similarly compact, with a slightly narrower range, but it’s quite sufficient and much more convenient for many common usages (excepting interval calculation).

Creating a Generic Insertion Iterator, Part 2

by Raymond Chen

Last time, we tried to create a generic insertion iterator but ran into trouble because our iterator failed to satisfy the iterator requirements of default constructibility and assignability.

We ran into this problem because we stored the lambda as a member of the iterator.

So let’s not do that!

Instead of saving the lambda, we’ll just save a pointer to the lambda.

template<typename Lambda>
struct generic_output_iterator
{
    using iterator_category = std::output_iterator_tag;
    using value_type = void;
    using pointer = void;
    using reference = void;
    using difference_type = void;

    generic_output_iterator(Lambda&& lambda) noexcept :
        insert(std::addressof(lambda)) {}

    generic_output_iterator& operator*() noexcept
        { return *this; }
    generic_output_iterator& operator++() noexcept
        { return *this; }
    generic_output_iterator& operator++(int) noexcept
        { return *this; }

    template<typename Value>
    generic_output_iterator& operator=(
        Value&& value)
    {
        (*insert)(std::forward<Value>(value));
        return *this;
    }

protected:
    Lambda* insert;

};

template<typename Lambda>
generic_output_iterator<Lambda>
generic_output_inserter(Lambda&& lambda) noexcept {
    return generic_output_iterator<Lambda>(
        std::forward<Lambda>(lambda));
}

template<typename Lambda>
generic_output_iterator(Lambda&&) ->
    generic_output_iterator<Lambda>;

This requires that the lambda remain valid for the lifetime of the iterator, but that may not a significant burden. Other iterators also retain references that are expected to remain valid for the lifetime of the iterator. For example, std::back_inserter(v) requires that v remain valid for as long as you use the inserter. And if you use the iterator immediately, then the requirement will be satisfied:

Creating a Generic Insertion Iterator, Part 1

by Raymond Chen

Last time, we created an inserter iterator that does unhinted insertion. We noticed that most of the iterator is just boilerplate, so let’s generalize it into a version that is all-boilerplate.

// Do not use: See discussion
template<typename Lambda>
struct generic_output_iterator
{
    using iterator_category = std::output_iterator_tag;
    using value_type = void;
    using pointer = void;
    using reference = void;
    using difference_type = void;

    generic_output_iterator(Lambda&& lambda) :
        insert(std::forward<Lambda>(lambda)) {}

    generic_output_iterator& operator*() noexcept
        { return *this; }
    generic_output_iterator& operator++() noexcept
        { return *this; }
    generic_output_iterator& operator++(int) noexcept
        { return *this; }

    template<typename Value>
    generic_output_iterator& operator=(
        Value&& value)
    {
        insert(std::forward<Value>(value));
        return *this;
    }

protected:
    std::decay_t<Lambda> insert;

};

template<typename Lambda>
generic_output_iterator<Lambda>
generic_output_inserter(Lambda&& lambda) {
    return generic_output_iterator<Lambda>(
        std::forward<Lambda>(lambda));
}

template<typename Lambda>
generic_output_iterator(Lambda&&) ->
    generic_output_iterator<Lambda>;

For convenience, I provided both a deduction guide and a maker function, so you can use whichever version appeals to you. (The C++ standard library has a lot of maker functions because they predate class template argument deduction (CTAD) and deduction guides.)

Using Senders/Receivers

by Lucian Radu Teodorescu

This is a follow-up to the article in the previous issue of Overload, which introduced the upcoming C++26 senders/receivers framework [WG21Exec]. While the previous article focused on presenting the main concepts and outlining what will be standardized, this article demonstrates how to use the framework to build concurrent applications.

The goal is to showcase examples that are closer to real-world software rather than minimal examples. We address three problems that can benefit from multi-threaded execution: computing the Mandelbrot fractal, performing a concurrent sort, and applying a graphical transformation to a set of images.

All the code examples are available on GitHub [ExamplesCode]. We use stdexec [stdexec], the reference implementation for the senders/receivers proposal. Additionally, some features included in the examples are not yet accepted by the standard committee, though we hope they will be soon.

How do I create an inserter iterator that does unhinted insertion into an associative container like std::map? 

by Raymond Chen

The C++ standard library contains various types of inserters:

• back_inserter(c) which uses c.push_back(v).
• front_inserter(c) which uses c.push_front(v).
• inserter(c, it) which uses c.insert(it, v).

C++ standard library associative containers do not have push_back or push_front methods; your only option is to use the inserter. But we also learned that the hinted insertion can speed up the operation if the hint is correct, or slow it down if the hint is wrong. (Or it might not have any effect at all.)

What if you know that the items are arriving in an unpredictable order? You don’t want to provide a hint, because that’s a pessimization. The inserter requires you to pass a hint. What do you do if you don’t want to provide a hint?

C++26: Erroneous Behaviour

by Sandor Dargo

If you pick a random talk at a C++ conference these days, there is a fair chance that the speaker will mention safety at least a couple of times. It’s probably fine like that. The committee and the community must think about improving both the safety situation and the reputation of C++.

If you follow what’s going on in this space, you are probably aware that people have different perspectives on safety. I think almost everybody finds it important, but they would solve the problem in their own way.

A big source of issues is certain manifestations of undefined behaviour. It affects both the safety and the stability of software. I remember that a few years ago when I was working on some services which had to support a 10x growth, one of the important points was to eliminate undefined behaviour as much as possible. One main point for us was to remove uninitialized variables which often lead to crashing services.

Thanks to P2795R5 by Thomas Köppe, uninitialized reads won’t be undefined behaviour anymore - starting from C++26. Instead, they will get a new behaviour called “erroneous behaviour”.

The great advantage of erroneous behaviour is that it will work just by recompiling existing code. It will diagnose where you forgot to initialize variables. You don’t have to systematically go through your code and let’s say declare everything as auto to make sure that every variable has an initialized value. Which you probably wouldn’t do anyway.

But what is this new behaviour that on C++ Reference is even listed on the page of undefined behaviour? It’s well-defined, yet incorrect behaviour that compilers are recommended to diagnose. Is recommended enough?! Well, with the growing focus on safety, you can rest assured that an implementation that wouldn’t diagnose erroneous behaviour would be soon out of the game.

Contracts for C++ Explained in 5 Minutes

by Timur Doumler

With P2900, we propose to add contract assertions to the C++ language. This proposal is in the final stages of wording review before being included in the draft Standard for C++26.

contract_assert replaces assert

A Pattern for Obtaining a Single Value While Holding a Lock

by Raymond Chen

It is often the case that you have a mutex or other lockable object which protects access to a complex variable, and you want to read the variable’s value (possibly modifying it in the process) while holding the lock, but then operate on the value outside the lock.

The traditional way is to do something like this:

// Assume we have these member variables
std::mutex m_mutex;
Widget m_widget;

// Get a copy of the widget
Widget widget;
{
    auto guard = std::lock_guard(m_mutex);
    widget = m_widget;
}
⟦ use the variable "widget" ⟧

This does suffer from the problem of running the Widget constructor for an object that we’re going to overwrite anyway. The compiler will also have to deal with the possibility that the lock_guard constructor throws an exception, forcing the destruction of a freshly-constructed Widget.

Fun with C++26 reflection - Keyword Arguments

by Che

An example implementation of the technique presented in this blog post can be found on GitHub. It can be used with Bloomberg’s experimental P2996 clang fork. If you enjoy these shenanigans, feel free to leave a star.

Prior art

Named, labeled or keyword arguments have been proposed many times over the years, but as EWG issue 150 notes: all of these attempts have failed. Here is several past proposals on the topic:

• n4172 Named arguments
• p1229 Labelled Parameters
• p0671 Self-explanatory Function Arguments

Since none of these proposals were accepted, we have to be somewhat creative to get similar functionality in C++. Naturally, there are various approaches to this problem. Below is a short overview of what you can already do without reflection.

Designated initializers

Let’s start with the simplest way to achieve keyword argument-like syntax. C++20 introduced designated initializers for aggregate types, which gives us the initialization syntax Point{.x=42, .y=7}.

In a function call’s argument list the type can potentially be deduced, so we could write foo({.x=2, .y=2}). While this requires extra curly braces and .-prefixes for every member name, syntactically this is almost what we want.

A Brief and Incomplete Comparison of Memory Corruption Detection Tools

by Raymond Chen

I promised last time to do a comparison of memory diagnostic tools. We have runtime diagnostic tools Address Sanitizer (ASAN), Valgrind, and Application Verifier (AppVerifier, avrf), and we have recording tools rr, and Time Travel Debugging (TTD)

First, the runtime tools: