Linux

Detecting vulnerabilities and managing the associated patching is challenging even in a small-scale Linux environment. Scale things up and the challenge becomes almost unsurmountable. There are approaches that help, but these approaches are unevenly applied.

In our survey, State of Enterprise Vulnerability Detection and Patch Management, we set out to investigate how large organizations handle the dual, linked security concerns of vulnerability detection and patch management.

The results produced interesting insights into the tools that organizations depend on to effectively deal with vulnerability and patch management at scale, how these tools are used, and which restrictions organizations face in their battle against threat actors. Download the copy of the report here.

Vulnerability management is an enterprise responsibility

Before we dive into the results of our survey, let’s take a quick look at why vulnerability management operations matter so much in large organizations.

Vulnerabilities are widespread and a major cybersecurity headache. In fact, vulnerabilities are such a critical problem that laws and regulations are in place to ensure that covered organizations adequately perform vulnerability management tasks – because the failure to do so can hurt a company’s customers.

Each industry has different rules that apply to it – with organizations that handle personal data such as healthcare records and financial service firms operating under the strictest rules. It has an impact on day-to-day vulnerability management operations – some organizations must act much faster and more thoroughly than others.

This is one of the points we explored in the survey, trying to understand how different industry compliance requirements affect vulnerability operations on the ground.

The survey

Early in 2021, we kicked off a survey with the intention to study three key factors in vulnerability and patch management operations. We examined patch deployment practices, how maintenance windows are handled, and tried to get a view into the overall level of security awareness of the organizations that responded.

The survey was advertised publicly to IT professionals around the world and it continues to run, even though we have published the initial results.

We know that live patching has real benefits because it significantly reduces the downtime associated with frequent patching. But live patching is relatively difficult to achieve without causing other problems and for that reason live patching is not implemented as frequently as it could be. After all, the last thing sysadmins want is a live patch that crashes a system.

Reproducible builds are one of the tools that can help developers to implement live patching consistently and safely. In this article, I explain why reproducible builds matter for live patching, what exactly reproducible builds are, and how containers are coming to the rescue.

Live patching: a key threat management tool

Patching is a critical part of systems maintenance because patching fixes faulty and buggy code. More importantly, security teams rely on patching to plug security holes, and there is a real urgency to it. Waiting for a convenient maintenance window to patch is risky because it leaves an opportunity for hackers to take advantage of an exploit.

It creates a difficult conundrum: maintain high availability but run a security risk, or patch frequently but end up with frustrated stakeholders. Live patching bridges that gap. With live patching, the offending code is swapped out while a process is actively running, without restarting the application or service that depends on that process.

Implementing live patching isn’t easy

Live patching is not that straightforward to accomplish – the drop-in code must “fit” in a like-for-like manner, or all sorts of unwanted things can happen. Get it wrong, and the application – or entire server – will crash.

The code behind a running process usually comes from a binary executable file – a machine-readable block of code compiled from source code. A kernel, for example, has thousands of source files all compiled into a few binaries.

With live patching, the live patch code must fit in at an exact level. Yes, the binary file containing the patch code will be different from the binary file containing the bad code. Nonetheless, the new code must slot into place precisely and must depend on the same version of imported libraries. The live patch code must also be compiled using the same compiler options and flags. Bit endianness matters too – the binary file must be ordered in exactly the same way.

In principle, all this is achievable – but in practice, it is a challenge. For example, day-to-day system updates often impact libraries. These libraries could be slightly different, in turn producing binaries that are slightly different when compiling code.

Whether you are welcoming CentOS Stream or looking for alternatives, the recent decision from the CentOS community to focus on CentOS Stream has forced a lot of technical leaders to rethink their Enterprise Linux strategy.  Beneath that decision, the business landscape involving Linux has shifted and expanded since its enterprise debut in the late 90s, when IBM would invest $1 billion in its development.

Today, Linux comes in every shape and size imaginable — with the kernel running on tiny low power computers and IoT devices, mobile phones, tablets, laptops all the way up to midrange and high-power mainframe servers.

Cutting through that expansive selection to understand which Linux distributions truly align with the needs of a business can lead to more frictionless deployments and successful execution while minimizing waste in maintenance cycles and optimizing overall cost.

This abridged guide to the Enterprise Linux landscape can give businesses an overview of which flavor (or flavors) of Linux will most adequately match their use cases.

For those looking for a more comprehensive guide, be sure to check out the Decision Maker’s Guide to Enterprise Linux.

Finding the Right Linux Flavor

Committing to a flavor can introduce many concerns. Beyond managing the deployments host-by-host, administrators must also consider the ecosystem components available to support the implementation at scale.

What mechanisms will be available for automatic patching? Can you optimize bandwidth by mirroring the distributions repository? Is remote desktop a concern?  What about the kernel version requirements? Linux Kernel 4 contains optimizations that lead directly to dollars saved on cloud deployments, can you take advantage of that?

Are you looking at a container strategy, thinking of deploying your apps into Kubernetes, or other multi-cloud strategies? What about options for embedded Linux

Nowadays there’s a preferred flavor of Linux for each of these concerns. A single flavor of Linux is really the Linux kernel surrounded by a curated suite of other free software. That other free software is what makes one flavor of Linux distinct from another.

Introduction

In an age where hacker attacks are a daily occurrence, it is of fundamental importance to minimize the attack surface. Containerization is probably the best way to isolate a service provided for the public, but this is not always possible for several reasons. For example, think of a legacy system application developed on systemd. This could make the most of the capabilities provided by a systemd-based operative system and it could be managed via a systemd unit, or it could automatically pull updates using a systemd timer, and so on.

For this reason, we are going to explain how to improve the security of a systemd service. But first, we need to step back for a moment.  With the latest releases systemd has implemented some interesting features relating to security, especially sandboxing. In this article we are going to show step-by-step how to strengthen services using specific directives, and how to check them with the provided systemd suite.

Debugging

Systemd provided an interesting tool named systemd-analyze. This command analyzes the security and the sandboxing settings of one or more specified services. The command checks for various security-related service settings, assigning each a numeric "exposure level" value, depending on how important the setting is. It then calculates an overall exposure level for the whole unit through an estimation in the range 0.0…10.0, which tells us how exposed a service is security-wise.

This allows us to check the improvements applied to our systemd service step-by-step. As you can see, several services are now marked as UNSAFE, this is probably due to the fact that not all of the applications are applying the features provided by systemd.

Getting Started

Let's start from a basic example. We want to create a systemd unit to start the command python3 -m http.server as a service:

[Unit]
Description=Simple Http Server
Documentation=https://docs.python.org/3/library/http.server.html

[Service]
Type=simple
ExecStart=/usr/bin/python3 -m http.server
ExecStop=/bin/kill -9 $MAINPID

[Install]
WantedBy=multi-user.target

Save the file and place it under the specific systemd directory of yor distribution.

By checking the security exposure through systemd-analyze security we get the following result: