Structured analysis of competing hypotheses

Intelligence Reasoning

Three Approaches to Reasoning

As we have addressed several times in this course, intelligence is a product manufactured from the exploitation of processed intelligence information, which in turn is a result of the collection of information by an intelligence collection platform. Intelligence analysts must often create intelligence products under conditions of incomplete or even inaccurate information.

To do so, we move back and forth between data points represented in intelligence information and models representing targets of interest to intelligence consumers; hence the reason why in this course we have placed so much emphasis on structured analytic techniques to manage information and target-centric approaches to building and analyze models. One way of looking at each side of this coin is by dividing it into two styles of reasoning, called inductive and deductive.

Inductive Reasoning

In inductive reasoning, several specific observations (data) lead to a generalized conclusion (theory) through a process (summarizing). So, for example, if we look around and see that every swan we see is white then we can use inductive reasoning to generate the theory that all swans are white. In fact, for centuries we in the Western world did think all swans were white until we observed black swans for the first time in Australia, which brings us to a key feature of inductive reasoning: probability.

Inductive reasoning is inherently probabilistic, meaning that it always allows for the possibility that a piece of data may come to light that contradicts a general principle that has served very well up until now to explain every other piece of data available. One of the issues with using inductive reasoning is that analysts sometimes forget that it is probabilistic. This means that the results of inductive reasoning are openly acknowledged to be the product of the data that have been included in the analysis, with the understanding that new data might mean a revision of the results.

A natural human bias, however, is to reject or discount contradictory evidence in part because changing one’s position on a subject incurs costs (remember our lecture on Appalachian Mountain problems in Week 5).

A well-known philosopher by the name of Karl Popper was reportedly in the company of a friend of his who was doctor. A nurse came to the door and told the doctor that he had a new patient waiting for an examination. The doctor asked what the patient’s symptoms were and after the nurse described them, the doctor declared the diagnosis and issued a prescription. After the nurse left, Popper supposedly asked the doctor how he knew what illness the patient had without even seeing him. The doctor responded that he had seen 1,000 people with those symptoms, to which Popper replied that the patient then must have been the 1,001st.

If analysts can keep in mind when they are (and are not) using inductive reasoning, then they can leave more room in their analysis for contradictory evidence, and therefore more room to change their analytic assessments.

Deductive Reasoning

Whereas inductive reasoning summarizes data to arrive at a theory, deductive reasoning tests a discreet fact to arrive at a single datum. To undertake deductive reasoning, we start with a theory and ask the question, “For this theory to be true, what else must be true?” The answer is typically a series of different elements, or aspects of the theory.

We identify which of these aspects we can test to see if they are true or false for a situation or problem to which we are applying the theory. In deductive reasoning, a generalization (theory) leads to a specific conclusion (datum) through a process (testing).

Remember during Week 1 when Chief Heaton gave us the theory that cases involving brothels in Tucson were actually part of some greater movement in organized crime? We handled that by asking ourselves what symptoms indicate the presence of organized crime involving human trafficking. We then ran tests for those symptoms to see if they were present or not.

The first ‘symptom’ of organization we found was that the variables of the age group and the nationality of the victims were correlated to a level of statistical significance. The ‘test’ we ran was a Chi-Square Test of Means, and the results supported the theory. After that, we ran other tests in that confirmed these results. We did something similar in Week 2 only that time instead of looking for the symptoms of organization in general, we looked for the symptoms of a specific organization in particular – its ‘fingerprints’. This next step required us to move from the theory that the brothels represented organized crime to the theory that a criminal organization behaves relatively consistent to modus operandi.

What if our tests did NOT demonstrate a pattern of organization or behavior? What would we do then? Well, we would probably keep looking. After all, we are accountable to a boss. But if we kept coming up dry, then at some point we have to admit to ourselves and Chief Heaton either that the brothel ‘industry’ was no industry at all, or that it was run by independent operators rather than a criminal organization. And that brings us to a key characteristic of deductive reasoning: determinism.

Deductive reasoning is inherently deterministic, which means that if our premise is false then our conclusion is false, as well. By ‘premise’ we mean the body of discrete facts against which we test our theory. Granted, we may run one test that comes back negative (false) but not throw out multitude of other tests that come back positive (true), but that is why we have to identify from the outset what our goals are and establish thresholds for when we are going to accept or abandon our theory (a.k.a. our alternative hypothesis).

In most statistical tests for criminal justice we will use a confidence threshold of 95%, meaning that if 95% of our results come back true, then we are willing take the 5% risk that we are wrong. Our criminal justice system for major crimes, however, requires a much higher threshold because in the United States we have built a system based on an openly acknowledged willingness to risk letting a guilty person go free rather than to convict an innocent person. If we were doing human subjects research on, say, fetuses, children, the elderly, or many other categories of vulnerable populations, then we would certainly use a confidence threshold at or above 99%.

Abductive Reasoning

It is very important for an analyst to be mindful of when he or she is working within the context of inductive or deductive reasoning. When the DEA told us in Week 4 that they believed El Movi was a drug-related organization because Daisy (our HUMINT source) had up until that point been involved in drugs, they were operating on the theory that informants who have a reporting history on one topic probably do not have access to information of intelligence value on another, unrelated topic.

The error in this way of thinking was that they were not sufficiently accounting for the probabilistic nature of inductive reasoning. In fact, they (probably) thought they were drawing a line of reasoning from one fact to another, rather than from a fact to a theory. On the flip side, the FBI were starting from the theory that terrorists have passed into the United States through Southeast Arizona from Mexico.

From this theory they surmised El Movi was probably a terrorism-related organization, but – as we now know – deductive reasoning demands that we construct a test for that and other facts based on the theory and then evaluate the results of that test to draw a conclusion rather than simply making an analytical assumption.

If we are not paying attention then our analysis is subject to the kinds of errors we addressed in Week 4, but that does not mean we cannot benefit from the advantages of operating within a well-reasoned framework of inquiry using a more fluid approach.

In reality, most of us are moving back and forth between inductive and deductive reasoning constantly even if for no other reason than that sometimes we start from a theory we are going to test and other times we start with data for whose characteristics we seek an underlying explanation simply by the nature of the problems we are attempting to solve. Analysis can begin at any point, and if we think of inductive and deductive reasoning as operating in a cycle, then we can see that the cycle does occur in only one direction (Figure 1).

Treating inductive and deductive reasoning as two, interrelated processes – as opposed to two, mutually contradictory ways of thinking – is called abductive reasoning. Through abductive reasoning we treat each side of the cycle as if its output served as an input for the other side.

We summarize data points to produce theories that we break into smaller parts that we test in order to arrive at data points we can use to produce theories – and the cycle goes on and on. For the sake of clarity, lets try an example starting with some facts that we will treat as a body of data. Lets say we, as analysts, are trying to solve a series of murders. We have information about multiple murders that seem to have similar characteristics.

We theorize that the best explanation for the murders is a serial killer. So we establish a profile for who this serial killer may be, what motivates him or her, where he or she lives, etc. When we identify someone that seems to meet these aspects of our theory, we can test them to determine whether they are, indeed, the person we are looking for or not.

Perhaps more importantly, abductive reasoning contains an auditing function, which an analyst can use to walk backwards through the cycle. If a theory is questionable then the analyst can locate the different observations that led to the creation of that theory, and interrogate the summarizing process to investigate whether, for example, a piece of information exerted an undue influence on the creation of the theory, or if the summarizing process included spurious information, or if information that should have been included in the summarizing process was ignored, for whatever reason.

If a specific piece of information needs to be examined, the analyst can interrogate the testing process to determine if, for example, the testing process itself failed to address the aspect of the theory that was relevant to the analyst’s purpose, or if the testing process used a standard that as so high it inadvertently eliminated useful results, or if the theory itself is composed of individual aspects that are inaccurate or imprecise that caused a fatal flaw in an otherwise valid test. In our serial killer example we may find we need to reassess the data or gather more data to refine our profile (i.e., our theory). Or we may find we are looking for the right indicators of our serial killer but not in the right way, so we need to reassess our testing methods.

Analysis of Competing Hypotheses

The reading for this week covers a structured analytic technique known as analysis of competing hypotheses (ACH). We have seen individual hypotheses at several points in our course, both generating them and testing them. More often than not, however, we deal with situations where we must consider the likelihood of more than one hypothesis at a time.

In fact, we should always be looking for opportunities to come up with multiple theories simply because otherwise we fall into the trap of deciding the truth rather than discovering the truth. When we expend resources to investigate a theory, we are simultaneously investing in that theory, which can cause us to resist responding effectively to new data or to new ways of looking at the data we have.

Biases are not always a bad thing. As humans we are constantly bombarded with infinite amounts of information (and especially so if you are an intelligence analyst, hence the TPEDs problem). Without a means to filter information we would be overwhelmed to the point of immobilization. As we grow up we learn rules of thumb to make it easier to survive as a limited being in a world of unlimited data.

· For example, if you are lost in the woods and need to find which direction to travel, you can use the old saying, “The sun rises in the east and sets in the west,” to orient yourself. Of course, the saying is untrue: the sun does not move! But in that situation, our rule of thumb – our bias – might be the difference between getting home and staying lost. We also usually work under conditions of incomplete information, requiring us to fill in gaps between data.

We have to use rules of thumbs about how we believe the world works in order to make analytical leaps that get us from one piece of data to another. We have to be mindful of when we are doing this, lest we lose track of the assumptions we have made, confusing them with the actual data themselves.

Lets say we are dealing with a case where we have a less than ideal amount of information – basically every case we deal with, actually.

We go over the data, back and forth, forward and backward, until over time we start to develop a story for what has taken place. Perhaps unthinkingly we start assigning attributes to the story based on other cases we may have worked.

We assume things about the people involved based on people we have met, and maybe not under the most amicable conditions. Information comes in that contradicts the story we have built, and we focus our energy on justifying all the reasons why we do not need to revise our story in light of new facts. ACH forces us to generate new theories in spite of our tendency to find one story and stick to it.

It also requires us to break up the pattern-filling function of our biases by looking at the contribution one piece of data makes for or against each hypothesis rather than allowing us to evaluate hypotheses as a complete storyline that can blind us to our own influence in its construction.

Briefing the Commissioner
Last week you applied techniques from social network analysis to interpret El Movi’s communication network based on SIGINT information acquired through a warrant authorizing the TPD to conduct wire tapping of the organization’s telephones. Your interpretation led you to produce a recommendation for Chief Heaton indicating which members of El Movi should be targeted to bring down the organization most efficiently.

Scenario

Guess what: Remember Mario and Julia from the brothel in Week 1 that started all this? Well, we found Julia – Julia Simone, actually – just outside of Flagstaff. She was pulled over by the Arizona Highway Patrol for speeding, found to be intoxicated, and was arrested. During her processing, she gave a different address to the officer at the station than the one on her driver’s license. When that officer ran the address, she found that it was the location of a brothel that had been taken down about a month ago. Oops!

After some further digging, the officer discovered Julia Simone matched the description of the “Julia” reported by the victims at the brothel as one of its managers. When Julia awoke the next morning an investigator interrogated her about the brothel, during which she could have invoked her right to remain silent. Instead, she agreed to testify against El Movi in exchange for a reduced sentence, apparently motivated by the fact that when she and Marco contacted Angél for assistance after the raid, he told them to get out of Tucson and never come back.

Later that day she was sent back to Tucson to face charges waiting for her there, at which point an investigating officer debriefed her about her knowledge of El Movi. Shortly after leaving town, Marco (whom we now know as Marcus “Marco” Wheatley) disappeared, taking most of their money with him.

Julia’s description of El Movi closely resembled that of Daisy’s, but with the details only an insider could provide. The man who started El Movi was named Jorgé Álvarez. He was the son of a mid-level member of the Beltrán Leyva Cartel who operated primarily along the coastline of the Mexican state of Sonora, which borders Arizona. Jorgé got himself into some kind of trouble sometime around the mid-2000s when the Beltrán Leyva Cartel became embroiled in a conflict with the Sinaloa Cartel.

Jorgé’s father sent him to the United States to get away from the cartel business, but Jorgé was ambitious. He was visiting a brothel in Tucson when it struck him as a good idea to transfer the coercive tactics he knew from his life in the cartel to building a business for himself in the city. While there were no other cartels in Tucson for him to annoy, he did get on the wrong side of a major transnational gang originally out of Los Angeles named Mara Salvatrucha, a.k.a. MS-13, when he attempted to cut into their drug activities in the area. MS-13 quickly “encouraged” Jorgé to return to Mexico, which he promptly did.

In his place, MS-13 found Angél Martínez. Angél was not a member of the gang, per se, but MS-13 did strike a deal with him: He could make all the money he wanted in the brothel business, but he had to keep human trafficking and all the attention MS-13 figured brothels were bound to attract away from MS-13’s drug business. MS-13 gave Angél the initial seed money he needed to begin organizing his business – a loan he has long since repaid, with interest – and El Movi was formed.

Julia did not know much about how the brothel’s victims were recruited or transported out of Mexico, but she did know that Michael Edwards and Jesse Hales arranged for the women to be transported to the brothels. The victims were usually distributed individually to the brothels in order to isolate them from anyone they may know outside the house. The women were informed that they would cook and clean in exchange for being sponsored for a green card to work in the United States. They were closely monitored for several months while they were made completely dependent on the brothel managers until they could be slowly introduced into the sex trade themselves. Most of the women by this time were so dispirited, it only took a little persuasion to push them into the business. After this point it was more matter of keeping them busy and under control, which was the job of the managers.

The managers generally divided their duties at the house between them. The men managers kept the peace when customers became aggressive or unruly. They were occasionally called into to threaten the women, but that was rarely necessary to keep them “working”. The men were also responsible for finding customers and introducing them to the brothels. The women managers made sure customers were happy and kept the brothels’ victims in line. The managers had almost complete control of how they handled financing their individual houses as long as they kept a low profile and gave enough of their profits to Christopher Thompson, Sidney Barrows, Adam Robles, and Garrett Davenport when they came around at the first of each month to collect the money.

Julia did not know where the other brothels were located, but she provided the names of the other brothel managers as the following:

Julia rarely interacted with Angél himself. She and Marco had been seeing each other for a while when Sidney – a mutual acquaintance at the time – introduced them to Angél. Angél offered them a ‘business proposition’, and considering that it paid well and offered free housing, food, and a car, it was an easy decision. Angél had a few specific rules the couple had to follow: First, they had to have a handgun in the house at all times just in case it was needed for Marco to maintain security, but only as a last resort.

Angél detested the use of violence because he thought it would only cause more trouble for his business than it would solve. The gun had to be registered in Julia’s name, although she did not know why Angél insisted on it. Second, for all intents and purposes, the couple had to live like a normal couple in a normal neighborhood. Julia was to play the role of housewife while Marco was to play the role of the faithful husband. Angél was very clear that if they did not look and act like someone who you would never suspect of running a brothel, then there would be ‘consequences’. Third, the couple were given a smart phone and forbidden to use any other forms of communication – no Internet, no landline telephone, nothing. Julia could never use the phone, but she could visit friends and family provided she kept her attention on the brothel. Lastly, if they were busted, they were on their own.

For a while things were going well until she and Marco got greedy. They knew they could never get away with stealing money from Angél, but they did convince him to let them move into a bigger house than the others in order to keep more women. They also started selling customers alcohol and keeping the profits for themselves – breaking another of Angél’s strict prohibitions against the use of drugs or alcohol in the house. One night, things got out of hand between two drunken customers. One shot the other in front of the house, and rest is history.

Armed with this new information, Chief Heaton thinks it might be time to prepare to bring down El Movi. Before he does this, however, he wants you to assess all the information we have so far and come up with a prioritized list of hypotheses about the organization. Specifically, he wants to know what will be the consequences of raiding the brothels. Is bringing down El Movi really only a matter of bringing a local human trafficking ring to an end? Will it spark a war between MS-13 and the police? Will it spark a war between a cartel and the police? So on and so forth.