Snort too many pipelined requests

Enhancing Byte-Level Network Intrusion Detection Signatures with Context

Robin Sommer TU München

Germany

sommer@in.tum.de

Vern Paxson International Computer Science Institute and

Lawrence Berkeley National Laboratory Berkeley, CA, USA

vern@icir.org

ABSTRACT Many network intrusion detection systems (NIDS) use byte sequen- ces as signatures to detect malicious activity. While being highly efficient, they tend to suffer from a high false-positive rate. We develop the concept of contextual signatures as an improvement of string-based signature-matching. Rather than matching fixed strings in isolation, we augment the matching process with additional con- text. When designing an efficient signature engine for the NIDS Bro, we provide low-level context by using regular expressions for matching, and high-level context by taking advantage of the se- mantic information made available by Bro’s protocol analysis and scripting language. Therewith, we greatly enhance the signature’s expressiveness and hence the ability to reduce false positives. We present several examples such as matching requests with replies, using knowledge of the environment, defining dependencies be- tween signatures to model step-wise attacks, and recognizing ex- ploit scans.

To leverage existing efforts, we convert the comprehensive sig- nature set of the popular freeware NIDS Snort into Bro’s language. While this does not provide us with improved signatures by itself, we reap an established base to build upon. Consequently, we evalu- ate our work by comparing to Snort, discussing in the process sev- eral general problems of comparing different NIDSs.

Categories and Subject Descriptors: C.2.0 [Computer-Communi- cation Networks]: General - Security and protection.

General Terms: Performance, Security.

Keywords: Bro, Network Intrusion Detection, Pattern Matching, Security, Signatures, Snort, Evaluation

1. INTRODUCTION Several different approaches are employed in attempting to detect

computer attacks. Anomaly-based systems derive (usually in an au- tomated fashion) a notion of “normal” system behavior, and report divergences from this profile, an approach premised on the notion that attacks tend to look different in some fashion from legitimate computer use. Misuse detection systems look for particular, explicit indications of attacks (Host-based IDSs inspect audit logs for this while network-based IDSs, or NIDSs, inspect the network traffic).

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CCS’03, October 27–31, 2003, Washington, DC, USA. Copyright 2003 ACM 1-58113-738-9/03/0010 ...$5.00.

In this paper, we concentrate on one popular form of misuse de- tection, network-based signature matching in which the system in- spects network traffic for matches against exact, precisely-described patterns. While NIDSs use different abstractions for defining such patterns, most of the time the term signature refers to raw byte se- quences. Typically, a site deploys a NIDS where it can see network traffic between the trusted hosts it protects and the untrusted exterior world, and the signature-matching NIDS inspects the passing pack- ets for these sequences. It generates an alert as soon as it encounters one. Most commercial NIDSs follow this approach [19], and also the most well-known freeware NIDS, Snort [29]. As an example, to detect the buffer overflow described in CAN-2002-0392 [9], Snort’s signature #1808 looks for the byte pattern 0xC0505289- E150515250B83B000000CD80 [2] in Web requests. Keeping in mind that there are more general forms of signatures used in in- trusion detection as well—some of which we briefly discuss in §2— in this paper we adopt this common use of the term signature.

Signature-matching in this sense has several appealing proper- ties. First, the underlying conceptual notion is simple: it is easy to explain what the matcher is looking for and why, and what sort of total coverage it provides. Second, because of this simplicity, signatures can be easy to share, and to accumulate into large “at- tack libraries.” Third, for some signatures, the matching can be quite tight: a match indicates with high confidence that an attack occurred.

On the other hand, signature-matching also has significant lim- itations. In general, especially when using tight signatures, the matcher has no capability to detect attacks other than those for which it has explicit signatures; the matcher will in general com- pletely miss novel attacks, which, unfortunately, continue to be de- veloped at a brisk pace. In addition, often signatures are not in fact “tight.” For example, the Snort signature #1042 to detect an exploit of CVE-2000-0778 [9] searches for “Translate: F” in Web requests; but it turns out that this header is regularly used by certain applications. Loose signatures immediately raise the major problem of false positives: alerts that in fact do not reflect an actual attack. A second form of false positive, which signature matchers likewise often fail to address, is that of failed attacks. Since at many sites attacks occur at nearly-continuous rates, failed attacks are often of little interest. At a minimum, it is important to distinguish between them and successful attacks.

A key point here is that the problem of false positives can po- tentially be greatly reduced if the matcher has additional context at its disposal: either additional particulars regarding the exact activ- ity and its semantics, in order to weed out false positives due to overly general “loose” signatures; or the additional information of how the attacked system responded to the attack, which often indi- cates whether the attack succeeded.

262

In this paper, we develop the concept of contextual signatures, in which the traditional form of string-based signature matching is augmented by incorporating additional context on different levels when evaluating the signatures. First of all, we design and imple- ment an efficient pattern matcher similar in spirit to traditional sig- nature engines used in other NIDS. But already on this low-level we enable the use of additional context by (i) providing full regu- lar expressions instead of fixed strings, and (ii) giving the signature engine a notion of full connection state, which allows it to corre- late multiple interdependent matches in both directions of a user session. Then, if the signature engine reports the match of a sig- nature, we use this event as the start of a decision process, instead of an alert by itself as is done by most signature-matching NIDSs. Again, we use additional context to judge whether something alert- worthy has indeed occurred. This time the context is located on a higher-level, containing our knowledge about the network that we have either explicitly defined or already learned during operation.

In §3.5, we will show several examples to demonstrate how the concept of contextual signatures can help to eliminate most of the limitations of traditional signatures discussed above. We will see that regular expressions, interdependent signatures, and knowledge about the particular environment have significant potential to reduce the false positive rate and to identify failed attack attempts. For example, we can consider the server’s response to an attack and the set of software it is actually running—its vulnerability profile— to decide whether an attack has succeeded. In addition, treating signature matches as events rather than alerts enables us to analyze them on a meta-level as well, which we demonstrate by identifying exploit scans (scanning multiple hosts for a known vulnerability).

Instrumenting signatures to consider additional context has to be performed manually. For each signature, we need to determine what context might actually help to increase its performance. While this is tedious for large sets of already-existing signatures, it is not an extra problem when developing new ones, as such signatures have to be similarly adjusted to the specifics of particular attacks anyway. Contextual signatures serve as a building block for increasing the expressivess of signatures; not as a stand-alone solution.

We implemented the concept of contextual signatures in the framework already provided by the freeware NIDS Bro [25]. In contrast to most NIDSs, Bro is fundamentally neither an anomaly- based system nor a signature-based system. It is instead partitioned into a protocol analysis component and a policy script component. The former feeds the latter via generating a stream of events that reflect different types of activity detected by the protocol analy- sis; consequently, the analyzer is also referred to as the event en- gine. For example, when the analyzer sees the establishment of a TCP connection, it generates a connection established event; when it sees an HTTP request it generates http request and for the corresponding reply http reply; and when the event engine’s heuristics determine that a user has successfully authenti- cated during a Telnet or Rlogin session, it generates login suc- cess (likewise, each failed attempt results in a login failure event).

Bro’s event engine is policy-neutral: it does not consider any particular events as reflecting trouble. It simply makes the events available to the policy script interpreter. The interpreter then ex- ecutes scripts written in Bro’s custom scripting language in order to define the response to the stream of events. Because the lan- guage includes rich data types, persistent state, and access to timers and external programs, the response can incorporate a great deal of context in addition to the event itself. The script’s reaction to a par- ticular event can range from updating arbitrary state (for example, tracking types of activity by address or address pair, or grouping re-

lated connections into higher-level “sessions”) to generating alerts (e.g., via syslog) or invoking programs for a reactive response.

More generally, a Bro policy script can implement signature-style matching—for example, inspecting the URIs in Web requests, the MIME-encoded contents of email (which the event engine will first unpack), the user names and keystrokes in login sessions, or the filenames in FTP sessions—but at a higher semantic level than as just individual packets or generic TCP byte streams.

Bro’s layered approach is very powerful as it allows a wide range of different applications. But it has a significant shortcoming: while, as discussed above, the policy script is capable of perform- ing traditional signature-matching, doing so can be cumbersome for large sets of signatures, because each signature has to be coded as part of a script function. This is in contrast to the concise, low-level languages used by most traditional signature-based systems. In ad- dition, if the signatures are matched sequentially, then the overhead of the matching can become prohibitive. Finally, a great deal of community effort is already expended on developing and dissemi- nating packet-based and byte-stream-based signatures. For exam- ple, the 1.9.0 release of Snort comes with a library of 1,715 signa- tures [2]. It would be a major advantage if we could leverage these efforts by incorporating such libraries.

Therefore, one motivation for this work is to combine Bro’s flexi- bility with the capabilities of other NIDSs by implementing a signa- ture engine. But in contrast to traditional systems, which use their signature matcher more or less on its own, we tightly integrate it into Bro’s architecture in order to provide contextual signatures. As discussed above, there are two main levels on which we use addi- tional context for signature matching. First, at a detailed level, we extend the expressiveness of signatures. Although byte-level pattern matching is a central part of NIDSs, most only allow signatures to be expressed in terms of fixed strings. Bro, on the other hand, al- ready provides regular expressions for use in policy scripts, and we use them for signatures as well. The expressiveness of such patterns provides us with an immediate way to express syntactic context. For example, with regular expressions it is easy to express the no- tion “string XYZ but only if preceded at some point earlier by string ABC”. An important point to keep in mind regarding regular expres- sion matching is that, once we have fully constructed the matcher, which is expressed as a Deterministic Finite Automaton (DFA), the matching can be done in O(n) time for n characters in the input, and also Ω(n) time. (That is, the matching always takes time linear in the size of the input, regardless of the specifics of the input.) The “parallel Boyer-Moore” approaches that have been explored in the literature for fast matching of multiple fixed strings for Snort [12, 8] have a wide range of running times—potentially sublinear in n, but also potentially superlinear in n. So, depending on the particulars of the strings we want to match and the input against which we do the matching, regular expressions might prove fundamentally more efficient, or might not; we need empirical evaluations to determine the relative performance in practice. In addition, the construction of a regular expression matcher requires time potentially exponential in the length of the expression, clearly prohibitive, a point to which we return in §3.1.

Second, on a higher level, we use Bro’s rich contextual state to implement our improvements to plain matching described above. Making use of Bro’s architecture, our engine sends events to the policy layer. There, the policy script can use all of Bro’s already existing mechanisms to decide how to react. We show several such examples in §3.5.

Due to Snort’s large user base, it enjoys a comprehensive and up-to-date set of signatures. Therefore, although for flexibility we have designed a custom signature language for Bro, we make use

263

of the Snort libraries via a conversion program. This program takes an unmodified Snort configuration and creates a corresponding Bro signature set. Of course, by just using the same signatures in Bro as in Snort, we are not able to improve the resulting alerts in terms of quality. But even if we do not accompany them with additional context, they immediately give us a baseline of already widely- deployed signatures. Consequently, Snort serves us as a reference. Throughout the paper we compare with Snort both in terms of qual- ity and performance. But while doing so, we encountered several general problems for evaluating and comparing NIDSs. We be- lieve these arise independently of our work with Bro and Snort, and therefore describe them in some detail. Keeping these limitations in mind, we then evaluate the performance of our signature engine and find that it performs well.

§2 briefly summarizes related work. In §3 we present the main design ideas behind implementing contextual signatures: regular expressions, integration into Bro’s architecture, some difficulties with using Snort signatures, and examples of the power of the Bro signature language. In §4 we discuss general problems of evaluating NIDSs, and then compare Bro’s signature matching with Snort’s. §5 summarizes our conclusions.

2. RELATED WORK [4] gives an introduction to intrusion detection in general, defin-

ing basic concepts and terminology. In the context of signature-based network intrusion detection,

previous work has focussed on efficiently matching hundreds of fixed strings in parallel: [12] and [8] both present implementations of set-wise pattern matching for Snort [29]. For Bro’s signature en- gine, we make use of regular expressions [18]. They give us both flexibility and efficiency. [17] presents a method to incrementally build the underlying DFA, which we can use to avoid the potentially enormous memory and computation required to generate the com- plete DFA for thousands of signatures. An extended form of regular expressions has been used in intrusion detection for defining se- quences of events [30], but to our knowledge no NIDS uses them for actually matching multiple byte patterns against the payload of packets.

In this paper, we concentrate on signature-based NIDS. Snort is one of the most-widely deployed systems and relies heavily on its signature set. Also, most of the commercial NIDSs are signature- based [19], although there are systems that use more powerful con- cepts to express signatures than just specifying byte patterns. NFR [28], for example, uses a flexible language called N-Code to declare its signatures. In this sense, Bro already provides sophisti- cated signatures by means of its policy language. But the goal of our work is to combine the advantages of a traditional dedicated pattern matcher with the power of an additional layer abstracting from the raw network traffic. IDS like STAT [35] or Emerald [26] are more general in scope than purely network-based systems. They con- tain misuse-detection components as well, but their signatures are defined at a higher level. The STAT framework abstracts from low- level details by using transitions on a set of states as signatures. A component called NetSTAT [36] defines such state transitions based on observed network-traffic. Emerald, on the other hand, utilizes P-BEST [20], a production-based expert system to define attacks based on a set of facts and rules. Due to their general scope, both systems use a great deal of context to detect intrusions. On the other hand, our aim is to complement the most common form of signa- ture matching—low-level string matching—with context, while still keeping its efficiency.

The huge number of generated alerts is one of the most impor- tant problems of NIDS (see, for example, [23]). [3] discusses some

statistical limits, arguing in particular that the false-alarm rate is the limiting factor for the performance of an IDS.

Most string-based NIDSs use their own signature language, and are therefore incompatible. But since most languages cover a com- mon subset, it is generally possible to convert the signatures of one system into the syntax of another. ArachNIDS [1], for example, generates signatures dynamically for different systems based on a common database, and [32] presents a conversion of Snort signa- tures into STAT’s language, although it does not compare the two systems in terms of performance. We take a similar approach, and convert Snort’s set into Bro’s new signature language.

For evaluation of the new signature engine, we take Snort as a reference. But while comparing Bro and Snort, we have encoun- tered several difficulties which we discuss in §4. They are part of the general question of how to evaluate NIDSs. One of the most comprehensive evaluations is presented in [21, 22], while [24] of- fers a critique of the methodology used in these studies. [14] further extends the evaluation method by providing a user-friendly environ- ment on the one hand, and new characterizations of attack traffic on the other hand. More recently, [10] evaluates several commer- cial systems, emphasizing the view of an analyst who receives the alerts, finding that these systems ignore relevant information about the context of the alerts. [15] discusses developing a benchmark for NIDSs, measuring their capacity with a representative traffic mix. (Note, in §4.2 we discuss our experiences with the difficulty of find- ing “representative” traces.)

3. CONTEXTUAL SIGNATURES The heart of Bro’s contextual signatures is a signature engine de-

signed with three main goals in mind: (i) expressive power, (ii) the ability to improve alert quality by utilizing Bro’s contextual state, and (iii) enabling the reuse of existing signature sets. We discuss each in turn. Afterwards, we present our experiences with Snort’s signature set, and finally show examples which demonstrate appli- cations for the described concepts.

3.1 Regular Expressions A traditional signature usually contains a sequence of bytes that

are representative of a specific attack. If this sequence is found in the payload of a packet, this is an indicator of a possible at- tack. Therefore, the matcher is a central part of any signature-based NIDS. While many NIDSs only allow fixed strings as search pat- terns, we argue for the utility of using regular expressions. Regular expressions provide several significant advantages: first, they are far more flexible than fixed strings. Their expressiveness has made them a well-known tool in many applications, and their power arises in part from providing additional syntactic context with which to sharpen textual searches. In particular, character classes, union, optional elements, and closures prove very useful for speci- fying attack signatures, as we see in §3.5.1.

Surprisingly, given their power, regular expressions can be matched very efficiently. This is done by compiling the expres- sions into DFAs whose terminating states indicate whether a match is found. A sequence of n bytes can therefore be matched with O(n) operations, and each operation is simply an array lookup— highly efficient.

The total number of patterns contained in the signature set of a NIDSs can be quite large. Snort’s set, for example, contains 1,715 distinct signatures, of which 1,273 are enabled by default. Matching these individually is very expensive. However, for fixed strings, there are algorithms for matching sets of strings simulta- neously. Consequently, while Snort’s default engine still works it- eratively, there has been recent work to replace it with a “set-wise”

264

matcher [8, 12].1 On the other hand, regular expressions give us set- wise matching for free: by using the union operator on the individ- ual patterns, we get a new regular expression which effectively com- bines all of them. The result is a single DFA that again needs O(n) operations to match against an n byte sequence. Only slight modifi- cations have been necessary to extend the interface of Bro’s already- existing regular expression matcher to explicitly allow grouping of expressions.

Given the expressiveness and efficiency of regular expressions, there is still a reason why a NIDS might avoid using them: the underlying DFA can grow very large. Fully compiling a regular ex- pression into a DFA leads potentially to an exponential number of DFA states, depending on the particulars of the patterns [18]. Con- sidering the very complex regular expression built by combining all individual patterns, this straight-forward approach could easily be intractable. Our experience with building DFAs for regular ex- pressions matching many hundreds of signatures shows that this is indeed the case. However, it turns out that in practice it is possible to avoid the state/time explosion, as follows.

Instead of pre-computing the DFA, we build the DFA “on-the- fly” during the actual matching [17]. Each time the DFA needs to transit into a state that is not already constructed, we compute the new state and record it for future reuse. This way, we only store DFA states that are actually needed. An important observation is that for n new input characters, we will build at most n new states. Furthermore, we find in practice (§4.3) that for normal traffic the growth is much less than linear.

However, there is still a concern that given inauspicious traffic— which may actually be artificially crafted by an attacker—the state construction may eventually consume more memory than we have available. Therefore, we also implemented a memory-bounded DFA state cache. Configured with a maximum number of DFA states, it expires old states on a least-recently-used basis. In the sequel, when we mention “Bro with a limited state cache,” we are referring to such a bounded set of states (which is a configuration option for our version of Bro), using the default bound of 10,000 states.

Another important point is that it’s not necessary to combine all patterns contained in the signature set into a single regular expres- sion. Most signatures contain additional constraints like IP address ranges or port numbers that restrict their applicability to a subset of the whole traffic. Based on these constraints, we can build groups of signatures that match the same kind of traffic. By collecting only those patterns into a common regular expression for matching the group, we are able to reduce the size of the resulting DFA dras- tically. As we show in §4, this gives us a very powerful pattern matcher still efficient enough to cope with high-volume traffic.

3.2 Improving Alert Quality by Using Context Though pattern matching is a central part of any signature-based

NIDSs, as we discussed above there is potentially great utility in incorporating more context in the system’s analysis prior to gener- ating an alert, to ensure that there is indeed something alert-worthy occurring. We can considerably increase the quality of alerts, while simultaneously reducing their quantity, by utilizing knowledge about the current state of the network. Bro is an excellent tool for this as it already keeps a lot of easily accessible state.

The new signature engine is designed to fit nicely into Bro’s lay- ered architecture as an adjunct to the protocol analysis event en- gine (see Figure 1). We have implemented a custom language for defining signatures. It is mostly a superset of other, similar lan-

1The code of [12] is already contained in the Snort distribution, but not compiled-in by default. This is perhaps due to some subtle bugs, some of which we encountered during our testing as well.

Figure 1: Integrating the signature engine (adapted from [25])

Event Control

Event Engine

Event stream

Real−time notification

Signature Control

Packet filter

Policy script

Filtered packet stream

Packet stream

Signature Engine

Signatures

Network

Policy Layer

Packet capture

guages, and we describe it in more detail in §3.3. A new component placed within Bro’s middle layer matches these signatures against the packet stream. Whenever it finds a match, it inserts a new event into the event stream. The policy layer can then decide how to re- act. Additionally, we can pass information from the policy layer back into the signature engine to control its operation. A signature can specify a script function to call whenever a particular signature matches. This function can then consult additional context and in- dicate whether the corresponding event should indeed be generated. We show an example of this later in §3.5.4.

In general, Bro’s analyzers follow the communication between two endpoints and extract protocol-specific information. For exam- ple, the HTTP analyzer is able to extract URIs requested by Web clients (which includes performing general preprocessing such as expanding hex escapes) and the status code and items sent back by servers in reply, whereas the FTP analyzer follows the applica- tion dialog, matching FTP commands and arguments (such as the names of accessed files) with their corresponding replies. Clearly, this protocol-specific analysis provides significantly more context than does a simple view of the total payload as an undifferentiated byte stream.

The signature engine can take advantage of this additional in- formation by incorporating semantic-level signature matching. For example, the signatures can include the notion of matching against HTTP URIs; the URIs to be matched are provided by Bro’s HTTP analyzer. Having developed this mechanism for interfacing the sig- nature engine with the HTTP analyzer, it is now straight forward to extend it to other analyzers and semantic elements (indeed, we timed how long it took to add and debug interfaces for FTP and Finger, and the two totalled only 20 minutes).

Central to Bro’s architecture is its connection management. Each network packet is associated with exactly one connection. This no- tion of connections allows several powerful extensions to traditional signatures. First of all, Bro reassembles the payload stream of TCP connections. Therefore, we can perform all pattern matching on the actual stream (in contrast to individual packets). While Snort has a preprocessor for TCP session reassembling, it does so by combin- ing several packets into a larger “virtual” packet. This packet is then passed on to the pattern matcher. Because the resulting analysis remains packet-based, it still suffers from discretization problems introduced by focusing on packets, such as missing byte sequences that cross packet boundaries. (See a related discussion in [25] of the problem of matching strings in TCP traffic in the face of possible intruder evasion [27].)

In Bro, a signature match does not necessarily correspond to an alert; as with other events, that decision is left to the policy script. Hence, it makes sense to remember which signatures have matched for a particular connection so far. Given this information, it is then possible to specify dependencies between signatures like “signature

265

A only matches if signature B has already matched,” or “if a host matches more than N signatures of type C, then generate an alert.” This way, we can for example describe multiple steps of an attack. In addition, Bro notes in which direction of a connection a particular signature has matched, which gives us the notion of request/reply signatures: we can associate a client request with the corresponding server reply. A typical use is to differentiate between successful and unsuccessful attacks. We show an example in §3.5.3.

More generally, the policy script layer can associate arbitrary kinds of data with a connection or with one of its endpoints. This means that any information we can deduce from any of Bro’s other components can be used to improve the quality of alerts. We demon- strate the power of this approach in §3.5.2.

Keeping per-connection state for signature matching naturally raises the question of state management: at some point in time we have to reclaim state from older connections to prevent the system from exhausting the available memory. But again we can leverage the work already being done by Bro. Independently of our signa- tures, it already performs a sophisticated connection-tracking using various timeouts to expire connections. By attaching the matching state to the already-existing per-connection state, we assure that the signature engine works economically even with large numbers of connections.

3.3 Signature Language Any signature-based NIDS needs a language for actually defining

signatures. For Bro, we had to choose between using an already existing language and implementing a new one. We have decided to create a new language for two reasons. First, it gives us more flexibility. We can more easily integrate the new concepts described in §3.1 and §3.2. Second, for making use of existing signature sets, it is easier to write a converter in some high-level scripting language than to implement it within Bro itself.

Snort’s signatures are comprehensive, free and frequently up- dated. Therefore, we are particularly interested in converting them into our signature language. We have written a corresponding Py- thon script that takes an arbitrary Snort configuration and outputs signatures in Bro’s syntax. Figure 2 shows an example of such a conversion.

Figure 2: Example of signature conversion alert tcp any any -> [a.b.0.0/16,c.d.e.0/24] 80

( msg:"WEB-ATTACKS conf/httpd.conf attempt"; nocase; sid:1373; flow:to_server,established; content:"conf/httpd.conf"; [...] )

(a) Snort

signature sid-1373 { ip-proto == tcp dst-ip == a.b.0.0/16,c.d.e.0/24 dst-port == 80 # The payload below is actually generated in a # case-insensitive format, which we omit here # for clarity. payload /.*conf\/httpd\.conf/ tcp-state established,originator event "WEB-ATTACKS conf/httpd.conf attempt"

}%

(b) Bro

It turns out to be rather difficult to implement a complete parser for Snort’s language. As far as we have been able to determine, its syntax and semantics are not fully documented, and in fact often only defined by the source code. In addition, due to different inter- nals of Bro and Snort, it is sometimes not possible to keep the exact semantics of the signatures. We return to this point in §4.2.

As the example in Figure 2 shows, our signatures are defined by means of an identifier and a set of attributes. There are two main

types of attributes: (i) conditions and (ii) actions. The conditions define when the signature matches, while the actions declare what to do in the case of a match. Conditions can be further divided into four types: header, content, dependency, and context.

Header conditions limit the applicability of the signature to a sub- set of traffic that contains matching packet headers. For TCP, this match is performed only for the first packet of a connection. For other protocols, it is done on each individual packet. In general, header conditions are defined by using a tcpdump-like [33] syntax (for example, tcp[2:2] == 80 matches TCP traffic with desti- nation port 80). While this is very flexible, for convenience there are also some short-cuts (e.g., dst-port == 80).

Content conditions are defined by regular expressions. Again, we differentiate two kinds of conditions here: first, the expression may be declared with the payload statement, in which case it is matched against the raw packet payload (reassembled where appli- cable). Alternatively, it may be prefixed with an analyzer-specific label, in which case the expression is matched against the data as extracted by the corresponding analyzer. For example, the HTTP analyzer decodes requested URIs. So, http /(etc\/(passwd |shadow)/ matches any request containing either etc/passwd or etc/shadow.

Signature conditions define dependencies between signatures. We have implemented requires-signature, which specifies another signature that has to match on the same connection first, and requires-reverse-signature, which additionally re- quires the match to happen for the other direction of the connection. Both conditions can be negated to match only if another signature does not match.

Finally, context conditions allow us to pass the match decision on to various components of Bro. They are only evaluated if all other conditions have already matched. For example, we have im- plemented a tcp-state condition that poses restrictions on the current state of the TCP connection, and eval, which calls an ar- bitrary script policy function.

If all conditions are met, the actions associated with a signature are executed: event inserts a signature match event into the event stream, with the value of the event including the signature identifier, corresponding connection, and other context. The policy layer can then analyze the signature match.

3.4 Snort’s Signature Set Snort comes with a large set of signatures, with 1,273 enabled

by default [2]. Unfortunately, the default configuration turns out to generate a lot of false positives. In addition, many alerts belong to failed exploit attempts executed by attackers who scan networks for vulnerable hosts. As noted above, these are general problems of signature-based systems.

The process of selectively disabling signatures that are not appli- cable to the local environment, or “tuning,” takes time, knowledge and experience. With respect to Snort, a particular problem is that many of its signatures are too general. For example, Snort’s signa- ture #1560:

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS

(msg:"WEB-MISC /doc/ access"; uricontent:"/doc/"; flow:to_server,established; nocase; sid:1560; [...])

searches for the string /doc/ within URIs of HTTP requests. While this signature is indeed associated with a particular vulner- ability (CVE-1999-0678 [9]), it only makes sense to use it if you have detailed knowledge about your site (for example, that there is no valid document whose path contains the string /doc/). Other- wise, the probability of a signature match reflecting a false alarm

266

is much higher than that it indicates an attacker exploiting an old vulnerability.

Another problem with Snort’s default set is the presence of over- lapping signatures for the same exploit. For example, signatures #1536, #1537, #1455, and #1456 (the latter is disabled by default) all search for CVE-2000-0432, but their patterns differ in the amount of detail. In addition, the vulnerability IDs given in Snort’s signatures are not always correct. For example, signature #884 ref- erences CVE-1999-0172 and Buqtraq [6] ID #1187. But the lat- ter corresponds to CVE-2000-0411.

As already noted, we cannot expect to avoid these limitations of Snort’s signatures by just using them semantically unmodified in Bro. For example, although we convert the Snort’s fixed strings into Bro’s regular expressions, naturally they still represent fixed sets of characters. Only manual editing would give us the additional power of regular expressions. We give an example for such an improve- ment in §3.5.1.

3.5 The Power of Bro Signatures In this section, we show several examples to convey the power

provided by our signatures. First, we demonstrate how to define more “tight” signatures by using regular expressions. Then, we show how to identify failed attack attempts by considering the set of software a particular server is runnning (we call this its vulnerabil- ity profile and incorporate some ideas from [22] here) as well as the response of the server. We next demonstrate modelling an attack in multiple steps to avoid false positives, and finally show how to use alert-counting for identifying exploit scans. We note that none of the presented examples are supported by Snort without extending its core significantly (e.g. by writing new plug-ins).

3.5.1 Using Regular Expressions Regular expressions allow far more flexibility than fixed strings.

Figure 3 (a) shows a Snort signature for CVE-1999-0172 that generates a large number of false positives at Saarland University’s border router. (See §4.1 for a description of the university.) Fig- ure 3 (b) shows a corresponding Bro signature that uses a regular expression to identify the exploit more reliably. CVE-1999-0172 describes a vulnerability of the formmail CGI script. If an at- tacker constructs a string of the form “...; ” (a | instead of the ; works as well), and passes it on as argument of the recipient CGI parameter, vulnerable formmails will ex- ecute the included shell commands. Because CGI parameters can be given in arbitrary order, the Snort signature has to rely on iden- tifying the formmail access by its own. But by using a regular expression, we can explicitly define that the recipient parame- ter has to contain a particular character.