This blog post is a work in progress. Comments are solicited. It is an attempt to bring my seventy years of experience in using, governing, managing, and securing innovations in computing technology to the issue of "artificial" intelligence.
In 1956, I was part of the first generation of computer user/programmers. We stood on the shoulders of the giants that I call generation zero, Turing, Flowers, Shannon, Aiken, Eckert, Mauchly, von Neuman, Hopper, et. al.
We had to tell the computer exactly how to arrive at the result that we wanted. We expressed the computations that we wanted in terms of the operation codes of the hardware. Almost all computers had op codes for add, some could subtract. The powerful ones even had operations for multiply and divide. However, even those did multiplication by iterative addition and division by iterative subtraction. We called that programming. Those of us who could describe how to achieve a result by such tiny steps were called programmers.
In the mid fifties, one might create a program on a blackboard, flip-chart, or yellow pad. We would transcribe the program into punched paper cards or tape that could be eletro-mechanically "read" by and into the storage of the computer. The paper and the reader were our user input mechanism, our application programming interface, our API. We would enter the program first, followed by the data, also in punched paper. The process was so expensive that the results had to be very valuable in order for the process to be efficient.
While in the early days, I operated the computer, within a decade users and programmers had pretty much been excluded from the "computer room" and the role of operator had become specialized. Within another decade the computer terminal was the common user interface. In 1981 the IBM personal computer had a key-board and a CRT that could display 80 columns by 25 lines of green alpha-numeric characters. Like the terminals, it provided what we now think of as a command line interface. Three years later the Apple Macintosh had a graphical user interface (GUI) with a mouse. Two years later came Windows. "Point and click" was now part of how one used a computer.
As computers have become more powerful and cheaper, much of that improved power has gone into the user interface, into making them easier to use. Early examples include high level languages, like Fortran and Cobol, and interpreters like APL and Basic
Today's input and output devices include touch screens, cameras, microphones, ear phones, speakers, and environment and application sensors and controls. Modern computers can talk, listen, and recognize and process images. We have smart appliances, homes, vehicles, drones, agents, and robots. I can testify that in1956 some of these were almost beyond imagining.
Games have always been part of computing and a measure of computer program "intelligence." In 1956, my colleague, Dick Casey, and I wrote a program to teach the IBM 650, using console switches and lights for i/0, to play Tic-Tac-Toe. My mentor, Dr. Albert Samuels, wrote a program for the IBM 701 to "learn" to play checkers. It got really serious in 1997 when IBM's Deep Blue defeated chess champion, Gary Kasparov in a six game match. On February 16, 2011 IBM's Watson defeated champions Ken Jennings and Brad Rutter at the TV Show Game, Jeopardy.
In 2015, Google's Alpha defeated European champion Fan Hui 5-0. Its most famous victory was against world champion Lee Sedol in 2016, winning the series 4-1. Alphagozero "taught" itself the game.
In November 2022 OpenAI introduced ChatGPT and demonstrated its ability to converse in natural language. Using text-to-speech and speech recognition, it can even talk and listen. It was to be the first of many large language models (LLMs), the latest user interface to the computer. However, the ability of these chat bots to communicate in natural language hides the huge data sets, the enormous computing power, and the complexity involved. It caused many people to invest them with the personality and autonomy that they merely mimic.
Using these programs one no longer has to provide the computer with a program, the steps to create the result. One need merely describe the result, "prompt" the computer and it "figures out" how to arrive at it. If the prompt is ambiguous, the computer may, in natural language, ask the user for addtitional information. The "Model" instantiates both the capabilities and the limitations of the program.
We call the model "large" because it includes a huge amount of data, so much that it all but defies human comprehension. Indeed it can solve difficult problems in conversational time. However, it is correct to think of it as fast "table lookup" in a big table, using so much computing power and energy that it is just now becoming efficient.
The LLM organizes its data to optimize its use; we call the data "training" and the organization "learning," expressions that we analogize from human intelligence. Therefore, the "model" instantiates both the capabilities and limitations of the program and its data.
In part, because of the complexity, training, and the lack of precision of the natural language used in the prompts, what we want the computer to do, the results are sometimes not what the user intends and expects. Said another way, in spite of its capabilities, artificial intelligence may be no less error prone than natural intelligence. Sometimes the results are so far from the user's intention, again by analogy to the natural, the computer is said to "hallucinate." Therefore, it is best to restrict the use of AI to the set of problems where, while we might not be able to arrive at the results by ourselves we can easily check the result. For example, while one might not be able to compute the cube root of a large integer, we can recognize it when we see it.
Nothing useful can be said about the security of any mechanism except in the context of a specific application and environment. --Robert H. Courtney, his first law.
Some human being or enterprise is responsible for everything a computer is asked to do and for all the properties and uses of the result.
The responsibility of both individuals and enterprises for the application and use of a computer, specifically to include those applications that use natural language and mimic people, includes compliance with law, regulation, contract, morality standards, and prudence.
As with anything else, an individual exercises her responsibility by complying with the law, regulation, contract, ethics, use of appropriate tools, and due care. This requires knowledge, skill, ability, experience, and good judgement. One starts by ensuring that one meets the applicable requirements. One ends by doing nothing rather than doing the wrong thing.
An enterprise exercises its responsibility by policy and governance, i.e., what it authorizes its agents to do and tells them not to do. Management must use all available controls including, but not limited to:
- Direction
- Training
- Assignment of duties, roles, responsibility, limits, discretion, and authority
- Supervision
- Multi-party controls
- Automation
- Budget
- Recognition and respect
- Compensation
- Disciplinary action
to ensure that its intentions are met.
One of the things that policy must do is to express managements risk tolerance in such a way that all managers and professionals at all levels understand what that means that they should do. For example, the management of a mature enterprise might say such things as "Novel technology must be applied in a cautious, conservative manner," "line of business managers (not IT) must authorize and budget for the application of novel technology," "application of novel technology must be initiated, authorized, and budgeted for by two or more levels of line of business management," or "novel technology should be exploited and applied consistent with normal business risk."
Alternatively, the management of a high business risk start-up might say "novel technology should be aggressively applied and exploited in pursuit of business opportunity." Needless to say, that this author recommends the more conservative approach for most enterprises.
Recommendations
Things you Should be Doing Anyway
Use strong Authentication: Use at least two kinds of evidence, (shared secret, possession, biometric, or behavior). At least one form of which is resistant to replay; e.g. one-time password (OTP), biometric with liveness test. Internet facing applications, mission critical applications, on management and administrative controls, and everywhere else, in that order. This is the most efficient of all measures. Nothing else will give you as much protection for dollar expended. Consider Passkeys, one time passwords from hardware tokens, software tokens, e-mail, or SMS messages, in decreasing order of strength. Consider offering users a choice of methods for their convenience.
Structure your network and layer your defenses. Start by isolating high risk Internet facing applications, e.g., e-mail, browsing, and messaging from internal enterprise applications, from mission critical applications, from administrative and management controls, from servers, e.g. database, file, communications, from storage devices.
Employee "Least Privilege" access control. Individuals and processes should enjoy only those capabilities and privileges that are required to fulfill their assigned roles.
Restrict "write" access. In order to preserve accountability, "write" access to any object should be restricted to a single individual or process, e.g., application, database manager. If you employ the common risky practice of granting everyone "read/write," changing to this policy will involve some administrative effort and may encounter some user resistance. However, it will reduce insider risk.
Employ "zero trust." Zero Trust can be thought of a special case of least privilege in which every connection between processes is authenticated in both directions, both vertically and horizontally. For example, users and applications mutually authenticate one another. Applications authenticate the database manager, the database manager authenticates the file system and vice versa. This policy can often be implemented using existing controls.
Use multi-party controls. Assign duties and responsibilities in such a way that agents, simply by doing their jobs, act as checks upon one another and that two or more must act in concert for material acts.
Monitor and log all activity. Monitor and log both the traffic that passes a firewall and the traffic that is rejected. Changes in traffic patterns may be evidence of an attack.
These policies and controls can greatly increase the cost and time-to-success of attacks. They are efficient, that is their cost is small, usually justified by the reduction in the cost of the risk that they mitigate. They are both essential and efficient.
As we have noted, large language models (LLMs) increase both risk and opportunity. Several LLM builders have demonstrated the ability to efficiently discover large numbers of vulnerabilities in operating systems and browser. These models can demonstrate how to exploit the vulnerabilities that they discover. They should also be able to remediate them.
In the next sections we will make recommendations to reduce the risk and safely exploit the opportunities.
Use LLMs Conservatively. Prefer low-risk applications, applications where the quality of the results are obvious, applications that are tolerant of error. Choose vendors and products based upon experience and support; do not rely upon claims.
Train with Application Specific Data. Prefer to train LLMs with data that is curated and specific to the purpose.
Over Control New Applications. Err on the side of over controlling any new application, including all uses of LLMs. As the application matures, one can always relax controls. However, if an application gets out of control, merely tightening the screws will not bring it back into control.
Use AI in Development. Use LLMs for coding and testing, including detection and elimination of vulnerabilities. Apply AI to Network Monitoring. As attacks increase in rate, human monitoring will be less effective and efficient. Use AI to support incident analysis and remediation.
Use AI to vet New Products and Changes. The software supply chain is under attack and is becoming the preferred way to distribute malicious code. The response of suppliers has been Caveat Emptor. Use AI to "beware."
Restrict the use of Enterprise Data. Expose enterprise data only to enterprise AI, never to public or cloud applications.
Consider the use of AI to automate "patching." Expect automation to be necessary to maintain software at the rate at which AI will discover vulnerabilities.