crowmoor.se

It is New Year´s Eve again. The champagne is cooling, the family is getting ready and the security architect is, as tradition demands, writing a blog post.

What a year. It started with me ranting about compliance and ended with half the world searching their servers for Log4j. In between there was more ransomware than I have ever seen. Pipelines, hospitals, service providers, small municipalities and large corporations. My calendar this spring was one long bridge call from six in the morning to nine in the evening.

Two vacations were cancelled. One in June for a service provider that lost more than fifty forests, and one in New York that turned into a week in a hotel room. My wife says she has forgiven me. The credit card statement says otherwise.

But it has also been a year of real progress. More companies than ever are protecting their admins, building immutable backups and giving their SOC the mandate to act. Zero Trust is no longer a buzzword in the meetings I attend. And the vaccines have given us a little bit of normal life back.

My wish for 2022 is simple: Fewer incidents, more vacations and that everyone patches Log4j before the holidays end.

Happy new year, and greetings from all my dachshunds!

Tags: Business, Methodology, Personal

This week the world found out about a vulnerability in Log4j, a logging library used in a huge number of Java applications. It can be exploited remotely by simply getting the application to log a specific text string. The attackers started scanning for it within hours.

The first question every customer asked was: Where do we have Log4j? And for most of them it was a very hard question to answer.

Why? Because nobody had the whole picture. The server team knew their servers, the application team knew some of the applications, the network team knew the firewalls and the client team knew the laptops. But Log4j can be inside a vendor product on a server, an appliance in the network or a tool on a developer´s laptop. Nobody owned the question ‘where is this component?’.

This is a pattern I see in almost every incident. The attacker doesn´t care about your organisation chart. He moves from a laptop to a server to a network device to the cloud, and every time he crosses a border between two teams the investigation slows down.

Technically, you need an inventory that covers everything: hardware, software and the components inside the software. You need logs from every layer in one place. And you need tools that let you search all of it with one query.

Know your whole environment, not just your part of it.

Tags: Log4Shell, Security Architecture, Visibility

Following my previous post, let me go a bit further on the technical side of seeing the whole environment.

Most IT organisations are split by technology: servers, clients, network, cloud, applications. Security has often followed the same split. One team for endpoint protection, one for the firewalls, one for identity, one for the SIEM. Each team buys its own tools and each tool has its own console.

The result is that an attack that touches all of those areas, and they all do, has to be pieced together manually across five consoles and five teams. In an incident that takes days we don´t have.

So what does it look like when it works? The detection tools share signals. An alert on a laptop is connected to the identity that was used, the servers it logged on to and the mail that started it. The investigation happens in one place, with one timeline. And the response can act on all layers at once: isolate the machine, disable the account, block the mail, all from the same incident.

You don´t need one vendor for everything to get there, but the tools must talk to each other, and somebody must own the whole chain.

Attacks don´t respect your silos. Your defence shouldn´t either.

Tags: Detection and response, Organisation, Security Architecture

A last business post for the year, and it ties together the two previous ones.

Many security organisations are built on old competence areas. Network security, server security, application security, compliance. Each with its own manager, budget and priorities. It made sense when the threats followed the same lines. They don´t anymore.

Look at the big events this year. Ransomware that moves from a laptop to the domain controllers in hours. Attacks through service providers. And now Log4j, a vulnerability that can be anywhere in your environment at once. None of them fits in one box.

A modern security organisation is built around the whole. A security architecture function that owns the big picture. A detection and response function that sees and acts across all layers. Identity at the centre, because that is where every attack eventually goes. And a governance function that connects the risks to the business and handles compliance, as a result of good security rather than a goal of its own.

It also needs the mandate to cut across the IT organisation. Security that has to ask five different teams for permission will always be too slow.

For management this is not just a reorganisation. It is deciding that security is something you manage as a whole.

The attackers think in systems. So should you.

Tags: Business, Organisation, Security strategy

Every Swede of a certain age knows Anders F. Rönnblom´s ‘Det är inte snön som faller’ from 1980, the punk anti-Christmas song with the cartoon video that was shown on TV on Christmas Eve for years. I can´t get it out of my head this time of year, so here is my own security version. Not his lyrics, just borrowing the idea.

It´s not the snow that´s falling
it´s servers going down
the backups were reachable
and the admins logged on all over town.

It´s not the snow that´s falling
it´s the password from last year
the SOC sent off a mail on Friday
and on Monday it´s all too clear.

Joking aside, Christmas is coming and with it the holidays. For the attackers it is high season. Fewer people at work, slower decisions, systems that nobody wants to touch until January. Every year I spend some part of the holidays on an incident that started on the 23rd or the 24th.

So before you light the candles: Patch what is exposed to internet. Check that the backups ran and that one copy is out of reach. Make sure someone is watching the alerts, and that they can act.

Then enjoy the snow. The real one.

Tags: Christmas, Holidays, Ransomware

In my previous post I wrote about the holidays being high season for attackers. This one is for management: prepare your readiness now, not on the 23rd of December.

First, decide who is on call. Not just in IT but in management. If you are hit on Christmas Eve, someone needs the authority to make decisions like shutting down the network, calling in external help or talking to the press. Make sure that person is reachable and knows it.

Second, check that your contacts work. Phone numbers to the SOC, the incident response retainer, the service providers, the insurance company. Write them down on paper too, because during an incident the mail and Teams may be down.

Third, look at your incident response plan. Is it up to date? Has it been tested this year? Does everyone know where it is?

Fourth, reduce the risk before the holidays. Ask IT to patch the critical things, check the backups and pause risky changes. Many companies have a change freeze during the holidays. Good, but make the security fixes before it starts.

And finally, give your people a real break where you can, by staggering the on-call so nobody works the whole holiday.

A plan made in November costs a few meetings. A plan made during the incident costs a lot more.

Tags: Business, Incident response, Preparedness

Time for an unusually technical post. Let´s talk about Rowhammer.

Your computer´s memory, DRAM, stores bits as tiny electrical charges in cells arranged in rows. As the chips have become denser, the cells have moved closer together. So close that if you read the same row over and over again, very fast, the electrical disturbance can make bits flip in the neighbouring rows. That is Rowhammer.

Why is that a security problem? Because a flipped bit in the wrong place can change a permission, a pointer or a page table entry. Researchers showed years ago that it could be used to escape a sandbox and take full control of a machine, without any software vulnerability at all.

The memory manufacturers added protections inside the chips that try to detect rows being hammered and refresh their neighbours. For a while it looked like the problem was solved. Then in May this year Google published a technique called Half-Double, which hammers a row two steps away instead of right next to the victim. The protections only looked at the closest neighbours, so they missed it.

What can you do? Honestly, not much yourself. Use ECC memory in servers, keep firmware updated and follow what the hardware vendors recommend.

The main lesson is that security goes all the way down to the physics of the silicon.

Tags: Hardware security, Memory, Rowhammer

A short personal one.

My wife and I finally got away for a week in New York. Our first real trip in a long time. The plan was simple. Walk around Manhattan, eat too much and see a show or two.

The plan lasted about a day. Then a customer incident escalated and I spent the rest of the week in a hotel room with my laptop, a headset and an impressive amount of coffee. Bridge calls in the morning with Europe, support calls in the afternoon and status reports late at night.

My wife, on the other hand, had a great week. She saw the shops on Fifth Avenue and SoHo, and she made sure to tell me about all of them over dinner.

Second vacation this year that turned into work. I´m starting to see a pattern.

Tags: Incident response, Personal, Work life balance

After my post on Rowhammer I got a worried mail from a manager. Should we be protecting against this?

My answer: Probably not. At least not before you have done everything else.

Rowhammer is fascinating, but it is hard to exploit in practice and there are very few, if any, known real world attacks using it. Meanwhile, the attackers who hit companies every day use stolen passwords, unpatched servers and admins logging on everywhere. They don´t need to flip bits in your memory. You are giving them the keys at the door.

This is one of the most important principles in security, and I have repeated it on this blog for years: Do the right things first. You don´t have to protect against everything. You have to protect against the easy things, because that is what most attackers use. Make the easy ways hard, and most attackers will go looking for an easier target.

So when someone brings you an exotic threat, ask three questions. Is it actually used against companies like ours? Have we already closed the easy ways in? And what would it cost compared to what we haven´t done yet?

If the easy ways are still open, spend the money there.

The advanced threats make the headlines. The easy ones make the incidents.

Tags: Business, Prioritisation, Risk management

Following my short post from New York, something more serious.

The last eighteen months have been one crisis after another for security people. First the move to working from home, then a constant wave of ransomware, supply chain attacks and critical vulnerabilities that had to be patched yesterday. Many of the people I work with have not had a real vacation since 2019.

That has a cost, and management needs to see it. Tired people make mistakes. They miss things in the logs, they skip a step in a recovery, they make the wrong call at three in the morning. Tired people also leave. And replacing an experienced incident responder or security architect takes months, if you can find one at all.

So what can management do? Accept that crisis mode can´t be the normal mode. Plan for it: build teams big enough that people can actually be off when they are off. Use partners and retainers for the peaks instead of expecting your own staff to absorb everything. Make sure that people who worked through their vacation get it back. And ask your security team how they are doing, and listen to the answer.

Your security is only as good as the people running it, and those people need rest.

As for me, I´m fine. I like what I do. But the credit card bill from New York was unusually high this month.

Tags: Burnout, Business, Staffing