SolarWinds

By now most should realize that SolarWinds is a compromise on an almost unimaginable scale. It is a crisis.  While there are "indicators of compromise" there are no indicators of all compromises.  While the attackers have concentrated on gathering intelligence on only a small number of target sites, all SolarWinds customers must assume that they are compromised and that there may be multiple backdoors into their systems for which there are no ICUs.  Only a small number of enterprises, perhaps none, have sufficient control over the content of their systems to be sure that they are resistant to such backdoors.

In https://us-cert.cisa.gov/ncas/alerts/aa20-352a DHS/CISA has suggested that some enterprises under some circumstances will have to "rebuild (from scratch) hosts monitored by the SolarWinds Orion monitoring software using trusted sources."  In fact, we may have to rebuild all enterprise systems.  

President Obama's chief of staff, Rahm Emanuel, famously said in 2008, “You never want a serious crisis to go to waste. I mean, it's an opportunity to do things that you think you could not do before.”  It would be tragic, if after rebuilding our systems, we should come away as vulnerable as when we started.  

We should take Rahm's "opportunity" to introduce "zero trust," indeed zero trust on steroids.  One might well start with a Software Defined Network.  One should include mutually suspicious processes, strong authentication at all levels, and "least privilege" access control.  

Rebuilding systems in month's that took decades to evolve is a daunting task.  I am reminded of what my father taught me when I was just starting out in IT almost sixty years ago.  "Son," he said, "all hard problems in information technology have one and the same answer: one application at a time."  We can do this.  We should use the crisis to overcome the inertia that has kept us from doing what we all know we should have done a while ago.  We know what to do: all we need is the leadership to do it.  

Do not worry about the cost.  Much of what we need to do, we can do with available resources.  For example, we can implement "least privilege" with available tools.  It only requires a change in intent.  In any case, there is always enough money to do that which must be done.  

On "Patching" III

One cannot patch to a secure system.

The rate of published "fixes" suggests that there is a reservoir of known and unknown vulnerabilities in these popular products (e.g., operating systems, browsers, readers, content managers). No matter how religiously one patches, the products are never whole.

They present an attack surface much larger than the applications for which they are used and cannot be relied upon to resist those attacks.  However, in part because they are standard across enterprises and applications, they are a favored target.  

They should not be exposed to the public networks. Hiding them behind firewalls and end-to-end application layer encryption moves from "good" practice to "essential."

Patching may be mandatory but it is expensive, a cost of using the product.  

Control of Privileged Insiders

On April 23, 2019 the FBI published a Private Industry Notification (20190423-001).  The document was distributed as a pdf only by e-mail.  While marked “TLP-White,” “may be distributed without restriction," I could not find it on the web.

The summary read:

The FBI continues to observe U.S. businesses’ reporting significant losses caused by cyber insider threat actors.  These cases often involve former or disgruntled employees exploiting their enhanced privileges—such as unfettered access to company networks and software, remote login credentials, and administrative permissions— to harm companies. Cyber insider threat actors most often are motivated by revenge, but they also conduct attacks to profit financially from stolen information, gain a competitive edge at a new company, engage in extortion, or commit fraud through unauthorized sales and purchases.

I recommend it to the reader.  (Since I cannot find it on the web, here is a link to a private copy.)

There are two kinds of insider risk, accidental and intentional, and three threat sources, benign, dishonest, and disgruntled employees.  Note that insider threat rate is much lower than the outsider threat but the consequences, and therefore risk, may be much greater.  Outsiders damage the brand while insiders may bring down the business.

Not only is employee error by otherwise well motivated and intended employees perhaps the biggest source of losses ("The dummies have it, hands down, now and forever."  --Robert H. Courtney)  but it contributes to the success of attacks by outsiders. (Think “phishing” and other forms of duping.)  Undetected errors may result in employee temptation and fraud.  The employee makes an error and no one notices.  She repeats and still no one notices.  She finally concludes that she could do it in her own favor and still no one would notice.   We distinguish between dishonest employees, who want to keep their activities secret, and disgruntled employees who want you to know that you have been injured.  

Management supervision is the most effective of all insider controls.  Effective supervision usually requires that the supervisor could do, or at least appreciate, the job being supervised.  This control often breaks down for privileged IT jobs.  The more sensitive or unique the task to be supervised, the more narrow should be the span of control.  While one might be able to supervise a dozen tellers or coders, one might supervise no more than five or six loan officers, system designers, or privileged administrators.  

The limitation of supervision is cost; while it is effective, it is also expensive.  Therefore, other more efficient and complimentary controls are often substituted for all or part supervision.  These might include background checks, training, division of responsibility and privileges (so-called multi-party controls), cross training, job rotation, measurement, mandatory vacations, audit trails and audits, recognition, compensation, and complete and timely separation.  

I had been writing and talking on this subject for a few years before I added “please and thank you” to my list of controls.  While equitable compensation is a powerful control, no amount of it can compensate for inadequate recognition.  Many dishonest and most disgruntled employees feel that their contribution to the enterprise has not been appreciated.  Please and thank you go a long way toward maintaining necessary morale.  

The FBI notification gives special attention to IT personnel and, especially, privileged users such as system administrators.  Management often focuses on lower level employees, like tellers or clerks, doing routine tasks.  Where these engage in fraud they get little and are caught early.  It is professionals, managers, and executives who bring down the business.

It is ironic that these highly privileged actors are often inadequately supervised, under paid, and unaccountable.  We caution against the sharing of user IDs and passwords, but it is privileged IDs and passwords that are most likely to be shared.  Many administrators have so much privilege that they cannot be held accountable, can escalate their privileges, and the privileges, once granted, cannot be effectively withdrawn.  Think about the privileges that Edward Snowden had to have accumulated to gain access to all the information that he exfiltrated.  

One should not grant privileges that one cannot withdraw.  Therefore, privileged users should be required to use hardware token based strong authentication.  One should not grant privileges without accountability for their use.  Therefore, when there is more than one privileged user, i.e., in most large enterprises, Privileged Access Management (PAM) controls should be in place.  These controls will be covered in a later post.

The Desktop, our Achilles Heel

During the last three or four years the number and rate of enterpirse wide breaches have increased dramatically.  Successful attacks have relied upon duping users into clicking on bait malicious objects in e-mail messages and web pages.  The malicious objects capture user credentials and then use them to attack peer systems in the enterprise network, spreading the compromise laterally.  These attacks exploit user gullibility, re-usable credentials, the default desktop access control rule of ”read/write,”  and flat enterprise networks.  Therefore, many security practitioners recommend user training, multi-factor user authentication, and structured networks.  Resistance to all three of these measures is high and their effectiveness limited.  Moreover, they do not address the vulnerability of the desktops to have their procedures modified by their data.   We are left with a high level of risk.

E-mail and browsing are the Achilles Heel of the desktop and the desktop is the Achilles Heel of the enterprise.  One of these two applications are involved in a large percentage of all breaches.  Note that while Achilles was vulnerable on only one heel, small attack surface, the enterprise may be vulnerable on many desktops.                                                            

One obvious defense would be to isolate these two applications from the system on which they run and those systems from the other applications and systems of the enterprise.  Neither of those applications should have the capability to make persistent changes to the procedures of the systems on which they run.  

In a world of cheap hardware, one way to do this would be to run these two applications on sacrifical hardware dedicated to these two applications.  In a world of reliable process-to-process isolation, another would be to use that isolation to protect the system on which the applications run from any changes originating in those porous applications.  The first solution is resisted because IT culture sees hardware as expensive, this in spite of the fact that its cost halves every two years.  The second is resisted because user culture prefers convenience, generality, flexiblity, and ”dancing pigs” to security.  As a consequence, most desktops are configured to offer read-write access to most objects and  few provide reliable protective isolation.  

It does not have to be this way.  Ten years ago Steve Jobs and Apple introduced us to iOS, with very limited capabilities but with very strong process-to-process isolation and strong protection from anything done at the user interface.  As it has matured its capabilities have increased.  Controlled application-to-application communication has been introduced while maintaining strong isolation and protection.  Some generality and flexibility have been sacrificed to usability and security but less than the defenders of the status quo predicted.  Nonetheless, resistance to iOS was so strong that it provoked Android, a more traditional system.  

However, iOS has been adopted by a large population of users that enjoy ”most, but not all, of the benefits offered by the traditional general purpose system.” (Fred Cohen)  At the user application interface, it appears as a single user single application-only machine.  While it can maintain application state, iOS is resistant to any persistent change to itself from the application or the user.  

Said another way, iOS protects itself from its data, its user, and its user’s data.  While the application may be vulnerable to a ”bait” attack, the system is not.  Therefore, it is a preferred environment in which to run vulnerable applications like e-mail and browsing, and sensitive applications like banking and healthcare.   

Personal computers can be configured with hypervisors to provide strong process to process isolation.  They can be configured with the ”least privilege” access control rule to resist contamination of procedures by their data.  Said another way, they can be configured such that simply clicking on a bait object is not sufficient to compromise the system.  Indeed they can even be configured in such a way that, as in iOS, nothing done with the keyboard and mouse is sufficient to compromise the system. 

This brings us to the ”flat enterprise network.”  Traditionally, enterprise networks have been configured for any-to-any connectivity; any node in the network could send a message to any other node.  The latency and bandwidth between any two nodes was roughly the same as the average across all nodes.  Often, and at least by default, they have been operated at a single level of trust.  That is to say, all nodes in the network were assumed to be benign, orderly, and well behaved. Nodes were not expected to have to protect themselves from traffic that originated on the network or question the origin address.  It is this configuration that leaves the enterprise vulnerable to lateral compromise with little more than one compromised system or user credentials.  

The alternative and safer network is referred to as ”zero trust.”  All nodes are assumed to be mutually hostile.  Traffic may flow only between specified pairs, e.g., user to application or client to server. Origin addresses are not trusted but must be authenticated.  Some cost in latency or bandwidth is tolerated for authorization of the connection and mutual authentication of the nodes. This kind of network is resistant to lateral compromise; a compromised node can attack only the nodes to which it is allowed to send traffic.  Even those nodes will treat it with suspicion and may require evidence as to its identity.  

There are a number of ways to restrict the flow of traffic to accord to this policy.  The first and most obvious is to provide only links between authhorized nodes; easy for two nodes, illustrative, but it does not scale even to a small enterprise.   However, the others simulate this illustration, usually through the use of encryption, e.g.,virtual local area networks, VLANs. virtual private networks, VPNs, and software defined networks, SDNs.  Note that in SDNs, users are included as ”nodes.”  Note also that to be most resistant to attack, connections should be at the application layer.  Applications are the nodes of interest and, contrasted to, for example, operating systems, have the smallest attack surface, i.e., the user interface.  

So, to summarize, the traditional use and configuration of desktops leave the enterprise vulnerable.  While awareness and strong authentication, remain essential practices they are limited in their effectiveness.  E-mail and browsing should be isolated from mission critical or otherwise sensitive applications.  The environment should be resistant to persistent changes to programs or procedures from application data; least privilege access control.  Network traffic should be encrypted end-to-end at the application layer; prefer software defined networks to VPNs to VLANs.  

 

Securability

In 2008 the ACM sponsored a Workshop on the Application of Engineering Principles to Information System Security.  Participants were asked to submit brief notes as seed material for the Workshop.  Far and away the most useful paper submitted to the workshop was by Amund Hunstad anJonas Hallberg of the Swedish Defence Research Agency entitled “Design for securability – Applying engineering principles to the design of security architectures.” This original paper points out “that no system can be designed to be secure, but can include the necessary prerequisites to be secured during operation; the aim is design for securability.” That is to say, it is the securability of the system, not its security, which is the requirement. We found this idea to be elegant, enlightening, and empowering. Like many elegant ideas, once identified it seems patently obvious and so useful as to be brillant.

One cannot design an airplane to be safe, such that it can never be unsafe, but one can, indeed aeronautical engineers do, design them such that they can be operated safely.  Neither IBM nor Microsoft can design a system that is safe for all applications and all environments.  They can design one that can be operated safely for some applications and some environments.  As the aeronautical engineer cannot design a plane that is proof against ”pilot error,” so IBM and Microsoft cannot design a system that is proof against the infamous ”user error.”  One cannot design a plane that is proof against terrorism or a computer that is proof against brute force attacks.

In the early days we talked about the properties of secure systems, Integrity, Auditability, and Controllability, and we told product managers that the properties, features, and functions of the product must be appropriate for the intended application and environment of the product. 

Integrity speaks to the wholeness, completeness, and appropriateness of the product.  One test of Integrity is predicability, that is the product does what, and only what, is expected.  Note that very few modern computer systems meet this test, in large part because they too complex. 

Auditability is that property that provides for relative ease in inspecting, examining, demonstrating, verifying, or proving the behavior and results of a system.  The tests for Auditability include accountability and visibility or transparency.  The test of accountability is that it must be possible to fix responsibility for every significant event to the level of a single individual.  The test of visibility is that a variance from the expected behavior, use, or content of the system must come to thattention of responsible management in such a way as to permit timely and appropriate corrective action. 

Controllability is that property of a system that enables mamnagemrnt to exercise a directing or restraining influence over the behavior, use, or content of the system.   The tests are Granularity and Specificity.  The test of granularity requires that the size of the resource to be controlled must be small enough to permit management to achieve the intended level of risk.  Specificity requires that management be able to predict the effect of granting any access to any resource, privilege, or capability from the meta-data, e.g., name, properties, of the resource, privilege or capability. 

Note that these properties compliment one another, indeed are really simply different ways of looking at the property of ”securability.”  However, they may be achieved at the expense of other desiderata of the system.  How to achieve the proper balance is the subject for another day. 

Internet Vulnerability

On March 24, 2017 Gregory Michaelidis wrote in Slate on "Why America’s Current Approach to Cybersecurity Is So Dangerous."

He cited an article by Bruce Schneier.

In response, I observed to a number of colleagues, proteges, and students that "One takeaway from this article and the Schneier article that it points to is that we need to reduce our attack surface.  Dramatically.  Perhaps ninety percent.  Think least privilege access at all layers to include application white-listing, safe dcfaults, end-to-end application layer encryption, and strong authentication."

One colleague responded "I think one reason the cyber attack surface is so large is that the global intel agencies hoard vulnerabilities and exploits..."  Since secret "vulnerabilities and exploits" account for so little of our attack surface, I fear that he missed my point.

While it is true that intelligence agencies enjoy the benefits of our vulnerable systems and are little motivated to reduce the attack surface, the "hoarded vulnerabilities and exploits" are not the attack surface and the intel agencies are not the cause.  

The cause is the IT culture. There is a broad market preference for open networks, systems, and applications. TCP/IP drove the more secure SNA/SDLC from the field. The market prefers Windows and Linux to OS X, Android to iOS, IBM 360 to System 38, MVS to FS, MS-DOS to OS/2, Z Systems to iSeries, Flash to HTML5, von Neumann architecture [Wintel systems] to almost anything else.  

One can get a degree in Computer Science, even in Cyber Security, without ever even hearing about a more secure alternative architecture to von Neumann's [e.g. IBM iSeries. Closed, finite state architecture (operations can take the system only from one valid state to another), limited set of strongly-typed (e.g., data can not be executed, programs cannot be modified) objects, single level store, symbolic only addressing, etc.)]

We prefer to try and stop leakage at the end user device or the perimeter rather than administer access control at the database or file system. We persist in using replayable passwords in preference to strong authentication, even though they are implicated in almost every breach. We terminate encryption on the OS, or even the perimeter, rather than the application. We deploy user programmable systems where application only systems would do.  We enable escape mechanisms and run scripts and macros by default.

We have too many overly privileged users with almost no multi-party controls. We discourage shared UIDs and passwords for end users but default to them for the most privileged users, where we most need accountability. We store our most sensitive information in the clear, as file system objects, on the desktop, rather than encryptied, in document management systems, on servers. We keep our most sensitive data and mission critical apps on the same systems where we run our most vulnerable applications, browsing and e-mail. We talk about defense in depth but operate our enterprise networks flat, any to any connectivity and trust, not structured, not architected. It takes us weeks to months just to detect breaches and more time to fix them.  

I can go on and I am sure you can add examples of your own. Not only is the intelligence community not responsible for this practice, they are guilty of it themselves. It was this practice, not secret vulnerabilities, that was exploited by Snowden. It is this culture, not "hoarded vulnerabilities and exploits," that is implicated in the breaches of the past few years. It defies reason that one person acting alone could collect the data that Snowden did without being detected.  

Nation states do what they do; their targets of choice will yield to their overwhelming force. However, we need not make it so easy. We might not be able to resist dragons but we are yielding to bears and brigands. I admit that the culture is defensive and resistant to change but it will not be changed by blaming the other guy. "We have seen the enemy and he is us."

The Role of Risk Assessment in Digital Security

The very idea of Risk Assessment has always been controversial.  I have been engaged in the controversy for fifty years. My ideas on the subject are well considered if otherwise no better than anyone else's.  I record them here.

I attribute the application of this idea to what was then called Computer Security to my mentors, later colleagues, Robert H. Courtney, Jr. and Robert V. Jacobson.  They did it in an attempt to rationalize decision making, more specifically the allocation of scarce security resources, to the then nascent field.  They did it in response to their observation that many, not to say most, security decisions were being made based upon the intuition of the decision maker and their belief, and a tenet of this blog, that security is a space in which intuition does not serve us well.  They wanted to bring a little reason to the process.

They could not possibly have known that in a mere fifty years that the resources applied to this effort would grow to the tens to hundreds of billions of dollars, that the safety and liberty of the individual, the health of public and private enterprise, the efficiency and resilience of our economy, and the security of the nations would turn on how effectively and efficiently we used those resources.  

So, at its core risk assessment is a decision making tool.  It is a tool that we use to answer the question "where to spend the next dollar of our limited resources?"  Courtney's Second Law says one should "Never spend more mitigating a risk than tolerating it will cost you." We will, do, make this decision, with or without tools.  We make it intuitively or we make it rationally but we do make it.  

At its most elaborate risk assessment is a very expensive tool requiring significant knowledge, skill, ability, and experience to use, more than most of us enjoy.  It should be used only for expensive decisions, decisions that are expensive to reverse if we get them wrong.  At its simplest, it protects us from making decisions based solely upon the threat, attack, vulnerability, or consequence de jour.  It protects us from intuition, from fear.

All that said, few of us are confronting expensive or difficult decisions, decisions requiring sophisticated decision making tools, risk assessment or otherwise..  We have yet to implement all those measures that we know to be so effective and efficient as to require no further justification.  They are what Peter Tippett calls essential practices.  Anyone can do them, with available resources, they are about 0.8 effective but work synergistically to achieve an arbitrary level of security. They fall in that category that we call "no brainers."  All we need is the will.  

A Leapfrog Enterprise Security Strategy

Recently I was quoted in an article on newly reported, but somewhat old, breaches.  In the report I was quoted as suggesting that these breaches suggest that security has fallen behind and that, just in order to catch up, we need a "leapfrog" strategy.  This post will suggest what such a strategy might contain.

Mine would start with strong authentication close to the users, i.e., at the end point. Strong authentication will start with privileged users and move to all employees. We have known about the limitations of passwords and what to do about them for thirty years. It is way past time to get on with it. Going forward, the end point of choice will be the mobile computer, colloquially referred to as a "smartphone."  This device already contains powerful sensors that can be used for authentication of claims to identity.  Apple Touch ID and Samsung Face Unlock are simply early examples of what can be done.  These are quick and easy to use and, in combination with possession of the device, constitute strong authentication.  

My strategy would include reducing the number of privileged users, the reduction of their privileges, and accountability for the use and exercise of privileges. It would include involving two or more people in the exercise of sensitive but rarely used privileges. We have too many privileged users and too little visibility into how those privileges are used.

It would include the automatic notification of the subjects of records and the owners and managers of accounts of all use, changes to, or transactions against those records or accounts. If we are to detect breaches on a timely basis, we must increase and improve transparency and accountability.

It will include isolating e-mail and browsing from mission critical and other sensitive systems and data. The intelligence is clear that many, not to say most, compromises begin by duping the users of these two applications.

It will include end-to-end end, end-point to application, not perimeter, not operating system, encryption. We cannot continue to operate large enterprise networks as flat spaces, as spaces in which any system may address any other system in the network.

It will include restrictive, i.e., "white list only," granular access control close to the applications and data. It will probably include access control at every layer, e.g. between the application and the database, between the database and the file system.

These measures are neither expensive nor disruptive. Google has demonstrated that even strong authentication can be flexible and convenient. They can be implemented in parallel. There are vendors recommending them and with products and services to implement them.

This is an "off-the-top of my head" list; I am sure I have omitted something important. However, it is informed by fifty years of thinking about this problem. I am sure that many of my colleagues have measures not on my list but which they would include in a leapfrog strategy.