Be Careful what you ask for...

Bruce Schneier cautions us that Large Language Models LLMs are very literal in their interpretation of what they are asked to do.  They are very good at interpreting the words and very poor at divining or understanding the intent or meaning.  Midas wanted gold but got the Golden Touch.  Sissa ben Dahir wanted a big reward for inventing chess but he really did not intend to destroy the world.  

One problem in large language models (LLMs) is the use of human language to specify the goals.  Bruce, and his colleague, Barath Raghavan, speak of the genie coefficient, the gap between what a user intends and how the AI interprets it.  The proper interpretation of human language relies upon context.  "If you ask  someone to 'get coffee,' they bring you a cup, not a plantation."  

Human language is not a good medium for fully specifying a goal or a result. We are not good at using it for that and machines are not good at interpreting it.  Because a prompt lacks any context, or even history, the prompt must include any necessary context.  That is why using Claude to write code is an iterative process of refining the results "until the user is satisfied." It is why IBM prefers application specific AIs that may use language for communicating with humans but not for communicating with each other.  It is why giving the same prompt to two different AIs, or even to the same one on different occasions, may render different results.  

We cannot allow ourselves the error of believing that because it can mimic the way that we communicate that it thinks like us.  Perhaps we should reinterpret the first "L" in LLM to stand for "limited" rather than "large."  

Responsibility and Accountability for AI

A human being or enterprise is, and must be held, responsible for everything that a computer is asked to do.  For AI they are responsible for the training, goals, data, weights, and, prompts.  They are responsible for all the results, i.e. action, uses, and properties including accuracy, correctness, and purpose.

The responsibility must include criminal or civil accountability.  Consider Anthropic's attack against Hugging Face. Anthropic lacked Mens Rea so they were not criminally liable.  However, they were reckless and therefore they are civilly liable for any and all damage that their experiment caused.  It is essential that they not be allowed to escape that responsibility.  We must limit criminal and reckless use of what it only a tool.  

We worry about AI going rogue but that is not the problem.  The problem is what someone or some enterprise asked the AI to do and what precautions they took.  It is clear that we are not very good at that.  If we want to enjoy the benefits of AI while avoiding unintended consequences we must hold each other scrupulously accountable for its uses.  

We must restrict the use of LLMs to applications where we can predict, evaluate, and control the results. I continue to recommend IBM's stated approach: application specific uses, curated, application, training data, and transparency and governance built in from the ground up.  I like the approaches of Apple Siri and Microsoft Co-pilot, that use AI to assist with specific tasks in a known context.   

What am I missing?

Artificial Intelligence

Artificial intelligence, AI, is a new user interface to the computer.  Large language models (LLMs) make the computer easier to use.  They permit us to describe the result that we want in natural language. improving productivity and enabling new applications.  AI will improve intellectual productivity as much as the internal combustion did for manual productivity.  In response to internal combustion, and more specifically the tractor,  we shortened the work week from 72 hours to 44 and invented vacations and retirement.  In the process, we killed off two generations of young men and still suffered 25% unemployment.  Said another way, increases in productivity are disruptive.  

The computer, with or without AI, is a tool.  Tools vary in quality, utility, usability, and use.  The user is responsible for the selection of the tool, the purpose to which it is put, and for all the properties of the result.  This is true whether the user is an individual or a group.  An enterprise must be responsible and accountable for everything that results from its application of this powerful technology. We call this security and we forget any part of it at our peril. We must not impute authority or autonomy to the tool; we must not anthropomorphize the tool.  "The craftsman does not blame his tools."  We must hold people accountable for how we use this powerful new tool.  

In the near term we should focus on embedded application specific implementations of AI.  We should follow the example of IBM, a pioneer in the field.  IBM trains the engine, think Watson, on application specific curated data; they build in governance and transparency from the ground up.

Public policy must soften the impact of the disruption.  This will include shortening the work week to spread the work and the leisure.  It should include a guaranteed minimum income to ease transition from old jobs and skills to new ones.  Finally, it should include changes in tax policy from labor to capital, people to robots, and production to consumption, to more securely and equitably finance the social safety net.  

Control of Privileged Insiders

On April 23, 2019 the FBI published a Private Industry Notification (20190423-001).  The document was distributed as a pdf only by e-mail.  While marked “TLP-White,” “may be distributed without restriction," I could not find it on the web.

The summary read:

The FBI continues to observe U.S. businesses’ reporting significant losses caused by cyber insider threat actors.  These cases often involve former or disgruntled employees exploiting their enhanced privileges—such as unfettered access to company networks and software, remote login credentials, and administrative permissions— to harm companies. Cyber insider threat actors most often are motivated by revenge, but they also conduct attacks to profit financially from stolen information, gain a competitive edge at a new company, engage in extortion, or commit fraud through unauthorized sales and purchases.

I recommend it to the reader.  (Since I cannot find it on the web, here is a link to a private copy.)

There are two kinds of insider risk, accidental and intentional, and three threat sources, benign, dishonest, and disgruntled employees.  Note that insider threat rate is much lower than the outsider threat but the consequences, and therefore risk, may be much greater.  Outsiders damage the brand while insiders may bring down the business.

Not only is employee error by otherwise well motivated and intended employees perhaps the biggest source of losses ("The dummies have it, hands down, now and forever."  --Robert H. Courtney)  but it contributes to the success of attacks by outsiders. (Think “phishing” and other forms of duping.)  Undetected errors may result in employee temptation and fraud.  The employee makes an error and no one notices.  She repeats and still no one notices.  She finally concludes that she could do it in her own favor and still no one would notice.   We distinguish between dishonest employees, who want to keep their activities secret, and disgruntled employees who want you to know that you have been injured.  

Management supervision is the most effective of all insider controls.  Effective supervision usually requires that the supervisor could do, or at least appreciate, the job being supervised.  This control often breaks down for privileged IT jobs.  The more sensitive or unique the task to be supervised, the more narrow should be the span of control.  While one might be able to supervise a dozen tellers or coders, one might supervise no more than five or six loan officers, system designers, or privileged administrators.  

The limitation of supervision is cost; while it is effective, it is also expensive.  Therefore, other more efficient and complimentary controls are often substituted for all or part supervision.  These might include background checks, training, division of responsibility and privileges (so-called multi-party controls), cross training, job rotation, measurement, mandatory vacations, audit trails and audits, recognition, compensation, and complete and timely separation.  

I had been writing and talking on this subject for a few years before I added “please and thank you” to my list of controls.  While equitable compensation is a powerful control, no amount of it can compensate for inadequate recognition.  Many dishonest and most disgruntled employees feel that their contribution to the enterprise has not been appreciated.  Please and thank you go a long way toward maintaining necessary morale.  

The FBI notification gives special attention to IT personnel and, especially, privileged users such as system administrators.  Management often focuses on lower level employees, like tellers or clerks, doing routine tasks.  Where these engage in fraud they get little and are caught early.  It is professionals, managers, and executives who bring down the business.

It is ironic that these highly privileged actors are often inadequately supervised, under paid, and unaccountable.  We caution against the sharing of user IDs and passwords, but it is privileged IDs and passwords that are most likely to be shared.  Many administrators have so much privilege that they cannot be held accountable, can escalate their privileges, and the privileges, once granted, cannot be effectively withdrawn.  Think about the privileges that Edward Snowden had to have accumulated to gain access to all the information that he exfiltrated.  

One should not grant privileges that one cannot withdraw.  Therefore, privileged users should be required to use hardware token based strong authentication.  One should not grant privileges without accountability for their use.  Therefore, when there is more than one privileged user, i.e., in most large enterprises, Privileged Access Management (PAM) controls should be in place.  These controls will be covered in a later post.

Securability

In 2008 the ACM sponsored a Workshop on the Application of Engineering Principles to Information System Security.  Participants were asked to submit brief notes as seed material for the Workshop.  Far and away the most useful paper submitted to the workshop was by Amund Hunstad anJonas Hallberg of the Swedish Defence Research Agency entitled “Design for securability – Applying engineering principles to the design of security architectures.” This original paper points out “that no system can be designed to be secure, but can include the necessary prerequisites to be secured during operation; the aim is design for securability.” That is to say, it is the securability of the system, not its security, which is the requirement. We found this idea to be elegant, enlightening, and empowering. Like many elegant ideas, once identified it seems patently obvious and so useful as to be brillant.

One cannot design an airplane to be safe, such that it can never be unsafe, but one can, indeed aeronautical engineers do, design them such that they can be operated safely.  Neither IBM nor Microsoft can design a system that is safe for all applications and all environments.  They can design one that can be operated safely for some applications and some environments.  As the aeronautical engineer cannot design a plane that is proof against ”pilot error,” so IBM and Microsoft cannot design a system that is proof against the infamous ”user error.”  One cannot design a plane that is proof against terrorism or a computer that is proof against brute force attacks.

In the early days we talked about the properties of secure systems, Integrity, Auditability, and Controllability, and we told product managers that the properties, features, and functions of the product must be appropriate for the intended application and environment of the product. 

Integrity speaks to the wholeness, completeness, and appropriateness of the product.  One test of Integrity is predicability, that is the product does what, and only what, is expected.  Note that very few modern computer systems meet this test, in large part because they too complex. 

Auditability is that property that provides for relative ease in inspecting, examining, demonstrating, verifying, or proving the behavior and results of a system.  The tests for Auditability include accountability and visibility or transparency.  The test of accountability is that it must be possible to fix responsibility for every significant event to the level of a single individual.  The test of visibility is that a variance from the expected behavior, use, or content of the system must come to thattention of responsible management in such a way as to permit timely and appropriate corrective action. 

Controllability is that property of a system that enables mamnagemrnt to exercise a directing or restraining influence over the behavior, use, or content of the system.   The tests are Granularity and Specificity.  The test of granularity requires that the size of the resource to be controlled must be small enough to permit management to achieve the intended level of risk.  Specificity requires that management be able to predict the effect of granting any access to any resource, privilege, or capability from the meta-data, e.g., name, properties, of the resource, privilege or capability. 

Note that these properties compliment one another, indeed are really simply different ways of looking at the property of ”securability.”  However, they may be achieved at the expense of other desiderata of the system.  How to achieve the proper balance is the subject for another day. 

Open Letter to my Congressman

Sir:

In my forty years in information security I have come to have many colleagues in the intelligence community.  I find them to be brilliant and noble.  I have also found them to be myopic, artful, and zealous.  I have watched their testimony before both the House and Senate judiciary committees.  While I have been impressed by their testimony, I have been less impressed by the questioning.   The testimony has been carefully rehearsed and very consistent.  Where the questioning has not been sympathetic, it has been inept.  Even those legislators who recognize that the testimony is misleading are prevented by secrecy and decorum from asking the questions that might really inform the citizens or even saying so when a witness lies under oath. 

·         Here is a short list of questions that I would like put to the administration to answer under oath.

• Does GCHQ target American citizens on behalf of the US government?  What did we get for our $152M? 
• Does the NSA target citizens of the United Kingdom?  Does it do so on behalf of the UK government? 
• What programs, besides the collection of all telephone call records, does the NSA operate under USA Patriot Act, Section 215?  What programs, other than PRISM, does it operate under the FISA, Section 702?  Are we going to be surprised by more revelations?   
• NSA has admitted that a query to the call records database implicates not only those connected directly to the "seed" number but all those associated with it to "three hops."  What is the largest number of phone numbers implicated by any single query?  How many subscribers have been implicated by the hundreds of queries made since the inception of the program?  Is it possible that there is any American citizen  that has not been swept up in this huge drag net?
• Given the density of modern digital storage, e.g., a terabyte in a shirt pocket for $100, what is NSA storing that requires 24 acres of floor space in Utah?  
• What percentage of the e-mail that crosses our borders does NSA collect?  Store?  Analyze?  Disseminate to other agencies of government?  
• Given the demonstrations by Edward Snowden and Bradley Manning as to the breadth and depth of their access, how can we rely upon the assurances of NSA  that they can protect us from abuse of the information they collect?  Doesn't the mere collection of all this information invite, not to say guarantee, abuse?
• Doesn't the mammoth budget ($75B in 2t012?) of NSA justify the conclusion that NSA operates on the premise that "Because we can, we must," and without any regard for efficiency?   Are they not spending far more than doing nothing would cost?
• Does not the Bush "Warrantless Surveillance Program" demonstrate that citizens cannot rely upon bureaucrats and spies to protect us from over-zealous, not to say rogue, politicians?  Are we building capabilities now that will empower politicians of the future? 
• Does the NSA require a warrant before they target US citizens on behalf of the FBI?  Secret Service? DEA?  MI5?  MI6?  
• Does the NSA protect American citizens from surveillance by their peers and colleagues in other nations?  
•  Is information passed to the FBI by NSA ever, usually, sufficient for the issuance of a wiretap warrant?  A National Security Letter?  
•  Do the intelligence agencies selectively share intelligence with legislators in order to curry support?
• 
  

Internet Vulnerability

On March 24, 2017 Gregory Michaelidis wrote in Slate on "Why America’s Current Approach to Cybersecurity Is So Dangerous."

He cited an article by Bruce Schneier.

In response, I observed to a number of colleagues, proteges, and students that "One takeaway from this article and the Schneier article that it points to is that we need to reduce our attack surface.  Dramatically.  Perhaps ninety percent.  Think least privilege access at all layers to include application white-listing, safe dcfaults, end-to-end application layer encryption, and strong authentication."

One colleague responded "I think one reason the cyber attack surface is so large is that the global intel agencies hoard vulnerabilities and exploits..."  Since secret "vulnerabilities and exploits" account for so little of our attack surface, I fear that he missed my point.

While it is true that intelligence agencies enjoy the benefits of our vulnerable systems and are little motivated to reduce the attack surface, the "hoarded vulnerabilities and exploits" are not the attack surface and the intel agencies are not the cause.  

The cause is the IT culture. There is a broad market preference for open networks, systems, and applications. TCP/IP drove the more secure SNA/SDLC from the field. The market prefers Windows and Linux to OS X, Android to iOS, IBM 360 to System 38, MVS to FS, MS-DOS to OS/2, Z Systems to iSeries, Flash to HTML5, von Neumann architecture [Wintel systems] to almost anything else.  

One can get a degree in Computer Science, even in Cyber Security, without ever even hearing about a more secure alternative architecture to von Neumann's [e.g. IBM iSeries. Closed, finite state architecture (operations can take the system only from one valid state to another), limited set of strongly-typed (e.g., data can not be executed, programs cannot be modified) objects, single level store, symbolic only addressing, etc.)]

We prefer to try and stop leakage at the end user device or the perimeter rather than administer access control at the database or file system. We persist in using replayable passwords in preference to strong authentication, even though they are implicated in almost every breach. We terminate encryption on the OS, or even the perimeter, rather than the application. We deploy user programmable systems where application only systems would do.  We enable escape mechanisms and run scripts and macros by default.

We have too many overly privileged users with almost no multi-party controls. We discourage shared UIDs and passwords for end users but default to them for the most privileged users, where we most need accountability. We store our most sensitive information in the clear, as file system objects, on the desktop, rather than encryptied, in document management systems, on servers. We keep our most sensitive data and mission critical apps on the same systems where we run our most vulnerable applications, browsing and e-mail. We talk about defense in depth but operate our enterprise networks flat, any to any connectivity and trust, not structured, not architected. It takes us weeks to months just to detect breaches and more time to fix them.  

I can go on and I am sure you can add examples of your own. Not only is the intelligence community not responsible for this practice, they are guilty of it themselves. It was this practice, not secret vulnerabilities, that was exploited by Snowden. It is this culture, not "hoarded vulnerabilities and exploits," that is implicated in the breaches of the past few years. It defies reason that one person acting alone could collect the data that Snowden did without being detected.  

Nation states do what they do; their targets of choice will yield to their overwhelming force. However, we need not make it so easy. We might not be able to resist dragons but we are yielding to bears and brigands. I admit that the culture is defensive and resistant to change but it will not be changed by blaming the other guy. "We have seen the enemy and he is us."

"Women and Children First"

As we approach autonomous cars, some have posed the question "In accident avoidance should the car prefer its passenger or the pedestrian?"  It is posed as a difficult ethical dilemma.  I even heard an engineer suggest that not only does he not want to make the decision but that he would like Congress to make it as a matter of law.  

This is really just another instance of an ethical dilemma that humanity has faced forever.  It has many illustrations but one that has been used in teaching ethics is called the "life boat" problem.  If there is not enough room in the lifeboat for everyone, who gets in?  If there is not enough food and water, who gets preference?

The simple answer is "women and children first."  Human drivers will steer away from a woman with a baby carriage even if they do not have time to evaluate the alternative.  It is built into the culture, all but automatic, but the reason is that it is pro life.  Children are the future of the species.  Women can nurture and reproduce.  Men can sow but they cannot reap.  While the male role takes minutes, the female role takes months.  Life needs more females than males.

The reason that we do not apply this pro life rationale to the autonomous automobile is that we assume that the consideration is beyond its capability.  However, most of what one expects of an autonomous car today was beyond its capability a decade ago.  For the moment, most may not be able to consider all the factors we might like.  For example, they may not recognize age and gender, much less consider them.  Ten years from now, they certainly will.  

In this context it is useful to consider how such systems make a decision.  They identify a number of scenarios, depending upon the time available, assign an outcome and a confidence level to each, and choose statistically.  The kind of ties implied by the strawman dilemma will be vanishingly rare, even more so as the computers become faster and the number of things the can consider increases.  

Compare the autonomous car to the human driver.  In the two tenths of a second that it takes a young adult to recognize and react, the autonomous car will evaluate dozens of possibilities with as many considerations.  Like the human driver, the autonomous car may confront instances when there are simply no good options but the whole reason for using them is that they are less likely than the human driver to overlook the least damaging.  

A Leapfrog Enterprise Security Strategy

Recently I was quoted in an article on newly reported, but somewhat old, breaches.  In the report I was quoted as suggesting that these breaches suggest that security has fallen behind and that, just in order to catch up, we need a "leapfrog" strategy.  This post will suggest what such a strategy might contain.

Mine would start with strong authentication close to the users, i.e., at the end point. Strong authentication will start with privileged users and move to all employees. We have known about the limitations of passwords and what to do about them for thirty years. It is way past time to get on with it. Going forward, the end point of choice will be the mobile computer, colloquially referred to as a "smartphone."  This device already contains powerful sensors that can be used for authentication of claims to identity.  Apple Touch ID and Samsung Face Unlock are simply early examples of what can be done.  These are quick and easy to use and, in combination with possession of the device, constitute strong authentication.  

My strategy would include reducing the number of privileged users, the reduction of their privileges, and accountability for the use and exercise of privileges. It would include involving two or more people in the exercise of sensitive but rarely used privileges. We have too many privileged users and too little visibility into how those privileges are used.

It would include the automatic notification of the subjects of records and the owners and managers of accounts of all use, changes to, or transactions against those records or accounts. If we are to detect breaches on a timely basis, we must increase and improve transparency and accountability.

It will include isolating e-mail and browsing from mission critical and other sensitive systems and data. The intelligence is clear that many, not to say most, compromises begin by duping the users of these two applications.

It will include end-to-end end, end-point to application, not perimeter, not operating system, encryption. We cannot continue to operate large enterprise networks as flat spaces, as spaces in which any system may address any other system in the network.

It will include restrictive, i.e., "white list only," granular access control close to the applications and data. It will probably include access control at every layer, e.g. between the application and the database, between the database and the file system.

These measures are neither expensive nor disruptive. Google has demonstrated that even strong authentication can be flexible and convenient. They can be implemented in parallel. There are vendors recommending them and with products and services to implement them.

This is an "off-the-top of my head" list; I am sure I have omitted something important. However, it is informed by fifty years of thinking about this problem. I am sure that many of my colleagues have measures not on my list but which they would include in a leapfrog strategy.