Paul E. Jones' Blog

I enabled photo syncing in the OneDrive mobile app. As it was doing that, I saw an option to allow photos to be arranged by year. That's nice, because having thousands and thousands of photos in one folder is way too many. I turned it on while it was syncing. What did the app do? It synced files it had not already synced by year, but those already uploaded were not moved. I waited for the sync to complete thinking it might fix that. Nope. I go to my PC’s OneDrive folder and move pictures into the right year folder myself. I just uploaded photos on my phone (three years), so it was manageable.

My PC got busy syncing to the cloud. That worked.

Then I waited a couple of days. I wanted to see if it would put new pictures into the right folder. It did not. I checked the settings. Somehow, that option to categorize by year got turned off. I deleted the one photo and took a new one. It worked this time.

But while I was in the OneDrive app, I noticed it said my pictures were not sorted into year folders. My PC says they are. OneDrive.com says they are. The mobile app was screwed up somehow. I cleared the data and cache, forced it to stop, and started again. Only then did it have an accurate view of my OneDrive.

I took a new picture to see if it still worked. It did. Picture was uploaded and stored in the folder 2026. Good!

Then I go to the Photos app on Windows to see the photos in there. It gives me an error saying it cannot connect to OneDrive. Something is wrong with OneDrive! I quit the Photos app, quit the OneDrive app, restart OneDrive, start Photos. OK, now I see the photos. But they're not sorted by years on the left-hand side of the Photos app! I see the year folders, though. The problem? The metadata to tell the Photos app the date (or location) of the photo is not available unless you download the photo. What is the point of using cloud storage for the photos if you need to keep a local copy to browse the thumbnails and have them sorted properly by year. I downloaded a few to verify my suspicion.

I ask Microsoft Copilot to confirm this madness, and it did.

I have been using Google Photos for years. I thought I’d give Microsoft a change, but their product isn’t even remotely in the same league as Google Photos.

Among the many things I do, I manage several Linux systems (including NAS) that need to be reliably backed up. Like many people, I used “tar” to create compressed archives of the data. The step after that was to encrypt it locally using AES Crypt. Once the archive file was encrypted, I would then store it in cloud storage or physical media I can store offsite. Encryption was (and still is) an important part of the process. It is vital that I hold the keys, because I cannot be sure who can access files after they are out of my direct control. (Anyone trusting a cloud provider to keep files secure should rethink that security strategy.)

While “tar” severed well for most things, it was a bad solution for the many thousands of files I managed. And while tar does allow for incremental backups, it can be painful to deal with as the number of increments grows. Retrieval of files across a set of incremental backups was torturous.

Archiving the entire set of files repeatedly was inefficient and expensive, and tar’s incremental feature did not really address my needs. I wanted a better solution with the following capabilities:

• Archive each distinct file only once, even if stored locally many times
• Allow multiple revisions of any given file to be archived
• Expire old revisions after some defined time
• Retain deleted files for a period of time to allow for recovery
• Don’t hastily back up new or changing files, as multiple midday edits are not important
• Compress files before archival
• Securely encrypt files before archival
• Enable retrieval of any specific file or revision
• Enable the recovery of all archived files
• Allow revision counts and retention rules to vary depending on the classification of files (e.g., directory or set of directories)

What I came up with is software designed for use by talented IT staff called Pug. Pug is published as free software, and the source code is also available. It can be run directly on a Linux machine, or it can be run inside a container. I run Pug inside a container on my own NAS, which allows me to keep all important data files securely backed up directly on the server hosting the data.

Pug checks off every single one of the requirements I had, including the ability to securely encrypt files before uploading them to cloud storage. Using AES Crypt for security was an important requirement, and I even made it possible to perform key rotation. That is also an important aspect of any secure solution.

Pug was written over a decade ago. So why am I writing a blog post about it now? The reason is because I was thinking about the fact that for over a decade now, my systems have been securely backed up with virtually no intervention on my part. I almost forget it’s running, except for when I periodically introduce a new encryption key for the software to use or when I replace the NAS server with a new one. Pug has been rock solid for years and, while it’s a bit too complex for most hobbyists to use – or even some inexperienced IT staff – it is an excellent solution for people who want a good backup solution and have a similar set of requirements that I had when I set out to write Pug.

In full disclosure, though, I still use “tar” (pipelined to AES Crypt for security) when backing up some files on Linux machines, system configuration, etc. Such backups are relatively small. I will use the right tool for the task. It’s just that “tar” was a terrible solution for the massive number of user files I had to keep securely backed up.

As a software engineer enthusiastic about creating cool technology, I am equally passionate about writing good code. By “good code,” I mean code that is bug-free, stable, reusable, and maintainable. Over the years, I have encountered both excellent code and horrendous code. The latter is a serious problem.

Bug-ridden software often results from carelessness, rushed development, or a lack of unit testing. Beyond the issues introduced during initial development, poorly written code creates long-term maintenance challenges. It significantly increases costs for companies.

In contrast, well-written and well-documented code enhances reusability, which can lower the cost of software development. Companies that invest in reusable libraries avoid redundant work, saving time on future projects. While open-source projects can be valuable, I have found tremendous benefits in maintaining an in-house repository of source code that follows consistent design, documentation, and style guidelines.

This belief led me to give a presentation to my team titled Writing Code for the Next Person. It was well-received, though not everyone agreed with every point. Nonetheless, the ideas have proven successful. Since then, I have created a short document summarizing the key takeaways from that presentation, which include:

• File descriptions
• Function descriptions
• Commenting
• System documentation
• Naming conventions
• Code style

I strongly believe that engineers should write code with future maintainers in mind. All successful code will eventually be maintained by someone other than the original author(s). I encourage you to review Writing Code for the Next Person and share any feedback you may have.

Windows 11 Pro users can disable automatic Windows updates using the following steps:

• Run "gpedit.msc"
• Go to "Computer Configuration" → "Administrative Templates" → "Windows Components" → "Windows Update" → "Manage end user experience"
• Select "Configure Automatic Updates"
• Select "enabled"
• Select "3 = Auto download and notify for install"
• Uncheck "Install during automatic maintenance"

As administrator via the command-line, run "gpupdate /force" to force changes to be effective.

I believe that "Home" versions may not have the ability to manipulate policies, so you just have to live with Microsoft forcing these surprise updates.

I have been busy working on the latest version of AES Crypt, one of the most popular file encryption tools on the Internet with well over a million downloads. That works on the latest release is now complete.

AES Crypt 4 is the fourth major version of the popular encryption software. AES Crypt 4 is available as a GUI and command-line application for Windows, Linux, and Mac, with both binaries and source code published.

Key Updates:

• Enhanced Security: The key derivation function now uses FIPS-recommended PBKDF2 using HMAC-SHA-512 with 300,000 iterations, configurable via the command-line tool. We’ve also switched to PKCS#7 padding.
• Performance Boost: Intel Intrinsics functions improve performance by approximately 30% on Intel processors.
• Apple Silicon Support: The Mac version is now compiled for Apple Silicon.
• Safety Measures: Mac and Linux versions now refuse to overwrite existing files, aligning with the Windows version.
• Command-Line Enhancements: More options, integrated key generation, and a progress meter for long files.
• Code Overhaul: Rewritten in C++20 with daily unit tests and improved integration tests for enhanced code safety.
• Backward Compatibility: AES Crypt 4 remains fully compatible with all previous versions, ensuring access to encrypted files created by older versions.

All packages are available at https://www.aescrypt.com.

I previously wrote a blog post about Webex Hologram, which uses any one of several 3D headsets to allow one to participate in holographic videoconferences. While that was really awesome, Cisco just released a video for Webex using the Apple Vision Pro. This is a very practical step toward both allowing one to utilize a 3D headset with existing videoconferencing, while also serving as a precursor to a full-blown holographic video experience.

In recent years, I’ve seen several methods for variable-length encoding of integers for transmission over a network. The most recent one I encountered was defined in section 16 of the QUIC protocol specification. In that document, 64-bit integers are encoded as 1, 2, 4, or 8 octets based on the value of the integer. Another format is specified in Appendix B of the CBOR specification, which specifies how to encode any one of the typical integer sizes commonly used on modern computers today. Using a rigid encoding like these offers a means of quickly serializing data, but at a cost of increasing the number of octets required. In the case of CBOR, it perhaps isn’t accurate to refer to it as a variable-length encoding, but rather an encoding for each possible signed and unsigned integer type. However, the effect is more-or-less similar, though larger integers are encoded with less space efficiency.

When considering space-efficiency, another encoding approach is to encode integers so that the most-significant bit (MSb) of each serialized octet indicates whether this is the final octet or whether there is another octet to consume. The following illustrates that idea:

10000011 11111111 01111111
^        ^        ^--- 0 == final octet

Here, if a 1 is present as the MSb, it means the next octet is a part of the integer. If the MSb is a 0, it indicates this octet is the last octet in the serialized sequence of octets. Following this method, a value between 0 and 127 can be serialized into a single octet, while incrementally larger integers consume additional octets. This method can be used for both unsigned and signed integers, where the signed integers are stored in twos-complement format.

Serializing and deserializing data can be efficient. Below, I present functions that will perform those operations on a buffer given an unsigned or signed 64-bit integer. This code uses C++ constexpr functions, though these could easily be transformed into C macros if one prefers those.

// Space-efficient encoding of variable length integers
// Copyright (C) 2023
// Paul E. Jones <paulej@packetizer.com>

#include <cstdint>
#include <cstddef>
#include <span>

// Find the most significant bit position for unsigned integers
constexpr std::size_t FindMSb(std::uint64_t v)
{
    std::size_t p = 0;

    if (v >= std::uint64_t(1) << 32) p += 32, v >>= 32;
    if (v >= std::uint64_t(1) << 16) p += 16, v >>= 16;
    if (v >= std::uint64_t(1) <<  8) p +=  8, v >>=  8;
    if (v >= std::uint64_t(1) <<  4) p +=  4, v >>=  4;
    if (v >= std::uint64_t(1) <<  2) p +=  2, v >>=  2;
    if (v >= std::uint64_t(1) <<  1) p +=  1, v >>=  1;

    return p;
}

// Find the most significant bit position for signed integers
constexpr std::size_t FindMSb(std::int64_t v)
{
    return ((v >= 0) ? FindMSb(static_cast<std::uint64_t>(v)) :
                       FindMSb(static_cast<std::uint64_t>(~v)));
}

// Return number of octets required to encode given std::uint64_t value
constexpr std::size_t VarUintSize(std::uint64_t value)
{
    return FindMSb(value) / 7 + 1;
}

// Return number of octets required to encode given std::int64_t value
constexpr std::size_t VarIntSize(std::int64_t value)
{
    return (FindMSb(value) + 1) / 7 + 1;
}

// Serialize the unsigned integer into the given buffer, returning 0 if the
// buffer is too short to hold the serialized integer
std::size_t Serialize(std::span<std::uint8_t> buffer, std::uint64_t value)
{
    // Determine space requirements for the variable-width integer
    const std::size_t octets_required = VarUintSize(value);

    // Ensure the buffer is of sufficient length
    if (buffer.size() < octets_required) return 0;

    // Write octets from right to left (reverse order)
    for (std::size_t i = octets_required; i > 0; i--)
    {
        // Get the group of 7 bits
        std::uint8_t octet = value & 0x7f;

        // Shift the data bits vector by 7 bits
        value >>= 7;

        // If this is not the last octet, set the MSb to 1
        if (i != octets_required) octet |= 0x80;

        // Write the value into the buffer
        buffer[i - 1] = octet;
    }

    return octets_required;
}

// Deserialize the unsigned integer from the given buffer, returning number of
// octets deserialized or zero if there was an error
std::size_t Deserialize(const std::span<std::uint8_t> buffer,
                        std::uint64_t &value)
{
    std::uint8_t octet{0x80};
    std::size_t total_octets{0};

    // Initialize the integer value
    value = 0;

    // Read octets until we find the last one having a 0 MSb
    while (octet & 0x80)
    {
        // A 64-bits value should never require more than 10 octets
        if (++total_octets == 11) return 0;

        // Ensure we do not read beyond the buffer
        if (total_octets > buffer.size()) return 0;

        // Get the target octet
        octet = buffer[total_octets - 1];

        // Add these bits to the returned value
        value = (value << 7) | (octet & 0x7f);
    }

    // If the total length is 10 octets, initial octet must be 0x81
    if ((total_octets == 10) && (buffer[0] != 0x81)) return 0;

    return total_octets;
}

// Serialize the signed integer into the given buffer, returning 0 if the
// buffer is too short to hold the serialized integer
std::size_t Serialize(std::span<std::uint8_t> buffer, std::int64_t value)
{
    // Determine space requirements for the variable-width integer
    std::size_t octets_required = VarIntSize(value);

    // Ensure there is sufficient space in the buffer
    if (octets_required > buffer.size()) return 0;

    // Write octets from right to left (reverse order)
    for (std::size_t i = octets_required; i > 0; i--)
    {
        // Get the group of 7 bits
        std::uint8_t octet = value & 0x7f;

        // Shift the data bits vector by 7 bits
        value >>= 7;

        // If this is not the last octet, set the MSb to 1
        if (i != octets_required) octet |= 0x80;

        // Write the value into the buffer
        buffer[i - 1] = octet;
    }

    return octets_required;
}

// Deserialize the signed integer from the given buffer, returning number of
// octets deserialized or zero if there was an error
std::size_t Deserialize(const std::span<std::uint8_t> buffer,
                        std::int64_t &value)
{
    std::uint8_t octet{0x80};
    std::size_t total_octets{0};

    // Ensure we do not read beyond the buffer
    if (buffer.empty()) return 0;

    // Determine the sign of the number by inspecting the leading sign bit
    value = (buffer[0] & 0x40) ? -1 : 0;

    // Read octets until we find the last one having a 0 MSb
    while (octet & 0x80)
    {
        // A 64-bits value should never require more than 10 octets
        if (++total_octets == 11) return 0;

        // Ensure we do not read beyond the buffer
        if ((total_octets) > buffer.size()) return 0;

        // Get the target octet
        octet = buffer[total_octets - 1];

        // Add these bits to the returned value
        value = (value << 7) | (octet & 0x7f);
    }

    // If the total length is 10 octets, ensure the initial octet is one
    // of the only two valid values
    if ((total_octets == 10) && (buffer[0] != 0x80) && (buffer[0] != 0xff))
    {
        return 0;
    }

    return total_octets;
}

Having recently finished work on a project involving holographic videoconferencing (which was totally awesome, by the way), my mind now drifts back to one of the things that is near and dear to my heart: the concept of distributed applications in the space of collaborative multimedia communications.

Those who know me know that I was championing this concept for several years just as the 2008 financial crisis took a grip on the world. Before that global recession, there was substantial momentum and interest. It’s a non-obvious concept, often confused and conflated with the concept of “disaggregated media.” It usually took me explaining the concept to hardware vendors, like TV and device makers, for them to understand it. However, the reaction was always positive once the lightbulb went off in their heads.

I’m back to thinking about it again. I put together a new presentation on the subject of “Revolutionizing Collaboration Technology using Distributed Applications.” It is hard for me to not think about this concept because the concept is powerful and has the potential to truly revolutionize the way we communicate. It opens the door for innovation unlike anything we have seen to date in this space.

As I take inventory of the current collaborate communications landscape, I see an unfortunate trends in the industry, including more isolation, disregard for standards or interoperability, and more centralized control. I am convinced that a more open collaborative communications platform can be created that not only creates more value for users but can simultaneously create more customer loyalty and profit growth. I have heard many users express a yearning for more interoperability, and it is frustrating to them (and me, quite frankly) that it isn’t just seamlessly possible to jump on a video call with anyone in his or her address book because different people use different vendors.

Building a distributed application platform is not trivial. I have been studying this for a long time. I understand the challenges and there are plenty. At the same time, if done properly it opens the door for substantially more user adoption and collaboration among people. This is especially true for those who need to communicate with industry partners, vendors, etc.

Perhaps most neglected of all in all of the existing collaborative communications offerings is the consumer space. I understand why. And while there are many non-interoperable solutions from which to choose, the abundance of non-interoperable options is one of the biggest problems. I, and I think most users, would prefer to select a preferred communication application provider and, from that application, be able to collaborate with anyone.

This is achievable.

Some programmers say pointers are dangerous. I think pointers are fine; bad programmers are dangerous.

I have been asked several times “What is the metaverse?”

The short answer is that it’s a flailing company’s effort to remain relevant as their user base and core business erodes.

Seriously, that’s it.

Young people do not care about Facebook anymore, and older adults are getting bored with it, too. This is reflected in decline in number of daily active users.

Teens and young adults are more interested in newer platforms like TikTok and Snapchat.

I appreciate that my definition of metaverse is entirely non-technical, but it’s important to put things into perspective. There is and will be a lot of hype around metaverse, with Facebook jockeying to put itself at the front of the pack. it’s unlikely to unfold that way, though.

The metaverse concept is substantially equated with virtual reality or augmented reality. To that end, Facebook has a good foot in the door with the acquisition of Oculus. There is no doubt that virtual and augmented reality will become a bigger part of our lives. However, merely creating a piece of hardware is not going to make one the dominant player in the VR/AR market. Facebook understand that, which is why they’re trying to position themselves as the platform for this technology. This platform being the metaverse.

The challenge that Facebook has is that VR/AR will be most successful in entertainment (especially gaming) and business, neither of which Facebook has any significant presence. By far, the largest business opportunity for VR/AR will be gaming, which will be dominated by Microsoft and Sony. Apple could be strong contender, too, if it produces a long-rumored headset. Likewise, Google could be a player in this space and has dipped its toes in the water a bit. Importantly, though, the company that will dominate in this space will be one with a large platform. Microsoft, Sony, Apple, and Google have viable platforms. Facebook does not and likely will not. Facebook phone, anyone? They are too far behind.