Needed to parse the receive logs of the incoming SMTP traffic (Exchange 2010) to find out which devices are relaying mail through the system. The receive connector was very loose about the devices that could relay traffic through. So, needed to work out what will happen, so here it is. The basic flow is this:
RECVxxxxxx.log -> nxlog -> logstash -> Elasticsearch
Enable SMTP Receive logs to be generated. Do this in the Exchange Management Console. I have not detailed here, as a quick google will demonstrate how to do this. Ultimately, for Exchange 2010, the logs will be written to C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\Logs\ProtocolLog\SmtpReceive. The format of the file is detailed at the top of each file:
#Software: Microsoft Exchange Server #Version: 14.0.0.0 #Log-type: SMTP Receive Protocol Log #Date: 2019-12-19T00:00:02.159Z #Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context 2019-12-19T00:00:02.159Z,VALI\Servers and Applications,08D773C7FE2A833E,0,10.2.0.19:25,10.2.0.29:54567,+,, 2019-12-19T00:00:02.174Z,VALI\Servers and Applications,08D773C7FE2A833E,1,10.2.0.19:25,10.2.0.29:54567,*,SMTPSubmit SMTPAcceptAnyRecipient SMTPAcceptAnySender SMTPAcceptAuthoritativeDomainSender AcceptRoutingHeaders,Set Session Permissions 2019-12-19T00:00:02.174Z,VALI\Servers and Applications,08D773C7FE2A833E,2,10.2.0.19:25,10.2.0.29:54567,>,"
To send this through to ElasticSearch (ES), we need to parse the files. I have chosen NXlog here, as it was already existing the environment. To reduce the amount of data entering ES I decided to only send in lines with SMTP command mail-from in the line. An example of this line is follows:
019-12-19T00:00:02.190Z,SERVER\Servers and Applications,08D773C7FE2A833E,19,10.2.0.19:25,10.2.0.29:54567,<,mail from: <payroll@example.com
NXLOG
Once we have the logs being written to the disk, we now need something to parse them. This is what nxlog will do for us. Below is an extract of the relevant parts of the configuration file.
define EXBASEDIR C:\Program Files\Microsoft\Exchange Server\V14
<Extension csv_parser>
Module xm_csv
Fields datetime, connectorid, sessionid, sequencenumber, \
localendpoint, remotendpoint, event, data, context
</Extension>
<Input smtp_receive>
Module im_file
File '%EXBASEDIR%\TransportRoles\Logs\ProtocolLog\SmtpReceive\RECV*.LOG'
<Exec>
if $raw_event =~ /FROM/
{
csv_parser->parse_csv();
$EventTime = parsedate($datetime);
}
else
{
drop();
}
</Exec>
</Input>
<Output out_exchangercv>
Module om_tcp
Host IP ADDRESS OF LOGSTASH
Port 5142 # Replace with your desired port
Exec $SyslogFacilityValue = 2;
Exec $SourceName = 'exchange_smtpreceive_log';
Exec to_syslog_bsd();
</Output>
<Route exchange_smtp>
Path smtp_receive => out_exchangercv
</Route>
Lets unpack this configuration.
- The base directory of the Exchange logs is defined. Not really relevant in this case, but if we wanted to load in other logs, it would be valuable to as this is DRY (Dont Repeat Yourself).
- We load the CSV parsing module and define the fields that we want to parse. NOTE: I think these need to match exactly what it is expecting
- The magic happens in the Input statement block.
- We tell nxlog where the files are to read
- if the log file line contains FROM
- parse the line
- … otherwise we drop the line and dont send it too logstash
- The
<Output>section defines where the output will go. In this case will be sent to our logstash server on port 5142 - The <Route> section just ties inputs to outputs
Once we (re)start the nxlog service, it should send relevant lines towards our logstash server.
HINTS FOR DEBUGGING
There are some hints to assist with debugging this side of the connection.
- Use
log_info("raw_event is: " + $raw_event);to log the relevant incoming information to the log file. I used it in the
ifsection of the input block to write out when it got a valid line. This way I new that information was being sent to the logstash server. - Use the following flags in the INPUT block so that nxlog does not save where it was in reading the files while troubleshooting. Set the FILE name to be one file instead of a wildcard name.
SavePos FALSE ReadFromLast FALSE
LOGSTASH
Now that nxlog is sending lines to our LS server, we need it to listen on port 5142. Here is the logstash configuration
input {
tcp {
type => "ExchangeSMTPRcv"
port => 5142
}
}
filter {
if [type] == "ExchangeSMTPRcv" {
csv {
separator => ","
columns => ["date-time","connector-id","session-id","sequence-number","local-endpoint","remote-endpoint","event","data","context"]
}
mutate {
gsub => [ "remote-endpoint", ":.*","" ]
gsub => [ "local-endpoint", ":.*","" ]
remove_field => ["message", "date-time","port","event"]
}
}
}
output {
if [type] == "ExchangeSMTPRcv" {
elasticsearch {
hosts => ["localhost"]
index => "logstash_exchsmtpreceive-%{+YYYY.MM.dd}"
}
# stdout{
# codec => rubydebug
# }
}
}
Lets unpack
- Input section: defines LS to listen for TCP connections on port 5142. It sets the “Type” of the message to be “ExchangeSMTPRcv”
- Filter section: If the message is of type “ExchangeSMTPRcv” then
- Split the line by the , character
- The field names are defined in columns array
- Remove the port number from the remote-endpoint and local-endpoint columns. For example, the remote-endpoint might be decoded as 10.2.0.29:54567 by logstash. The gsub will change the field to be 10.2.0.29
- remove_field removes the original message, date-time, port and event fields from the block that goes to ES
- Output section: Writes the entry to ES into the logstash_exchsmtpreceive-%{+YYYY.MM.dd} index
For debuggingm uncomment the stdout section of the output section. This will write the decoded information to stdout if the logstash program is run standalone and not as a service.
Kibana
Once the files are being written to the database, then use kibana to graph the output