Tag Archives: OSSEC

OSSEC troubleshooting

Today we continue the saga of things I was supposed to write down but didn’t, for reasons unknown. Suppose you migrated your OSSEC management server, or freshly installed what will be the new manager on a new OS. You import the keys, as described in my previous post, but the connection fails for one or both of these reasons:

  • ossec-remoted(1403): ERROR: Incorrectly formatted message from '<client_ip>'. – Pick your own adventure-style error message.
  • ossec-agentd(1407): ERROR: Duplicated counter for '<server_name>'. – Incorrect serials.

This has happened several times over the course of the last decade, due to client/server version mismatch, drive failures, and what have you. There’s a pretty brute-force way to solve these problems, though:

  • stop both server and client;
  • on the client, delete everything inside /var/ossec/queue/rids;
  • reimport the key on the client (unsure if this step is really needed);
  • start the server;
  • test that the client is working, via ossec-agentd -d -f.
  • if the client is working, start the service.

That’s it. There’s nothing that a good ol’ rm -rf * can’t solve.

Better OSSEC syslog parsing for Splunk

Just as predicted by the documentation, the syslog parsing of the OSSEC app for Splunk was a bit meh: while it would work in several instances it would terribly fail in others, like HTTP access for example. Below you can find the current version I’m using, which also provide additional fields that can be used for reports.

[ossec-syslog-message]
#REGEX = ossec:.*?(Location:.*;)\s*(user: [^;]+;\s*)?(\w{3} \d+ [\d:]+ \w+ )?(.*)$
#FORMAT = message::$4
REGEX = ossec:.*?(Location: (.*?);)\s*(srcip: ([a-f0-9:\.]+);\s*)?(user: [^;]+;\s*)?([a-f0-9:\.]+ ([a-zA-Z\-]+ [a-zA-Z\-]+) )?(\[\w{3} \w{3} \d+ \d+:\d+:\d+\.\d+ \d+\]\s*|\w{3}$
FORMAT = ossec_location::$2 ossec_srcip::$4 ossec_httpusergroup::$7 ossec_msgtimestamp::$8 message::$9

What you see commented out are the original instructions that can be safely removed. The new REGEX is more complex than the original, maybe too much, but through this I can extract more information that were previously hidden, or not easily accessible, and at the same time remove redundant timestamps while having all the important messages correctly extracted.

If you have suggestions, feel free to comment below.

OSSEC Agent/Server + Splunk installation

There is a lot of documentation to be read about the installation of OSSEC, but it’s usually sparse and focused either on a local autonomous setup or on hundreds of VMs setups. In this article we will navigate through the necessary steps to set up a small OSSEC installation with the OSSEC agent running offsite on a web/mail server and the OSSEC server running onsite. Additionally we will take a look at Splunk and install it on the OSSEC server machine, which will make it easier to manage bigger volumes of data later on.

Prerequisites

In order to compile and install OSSEC you will need build-essential on Ubuntu machines and MySQL/PostgreSQL for database support. You can read more details about this here.

Agent/Server installation

Installing the agent and the server is as easy as running the script (after checksumming it) and answering a few questions, although you should keep (most of) the defaults since they’re solid, and then build up on them.

# wget http://www.ossec.net/files/ossec-hids-.tar.gz
# wget http://www.ossec.net/files/ossec-hids--checksum.txt
# cat ossec-hids-_checksum.txt
MD5 (ossec-hids-.tar.gz) = MD5SUM
SHA1 (ossec-hids-.tar.gz) = SHA1SUM
MD5 (ossec-agent-.exe) = MD5SUM_EXE
SHA1 (ossec-agent-.exe) = SHA1SUM_EXE

# md5sum ossec-hids-.tar.gz
MD5 (ossec-hids-.tar.gz) = MD5SUM
# sha1sum ossec-hids-.tar.gz
SHA1 (ossec-hids-.tar.gz) = SHA1SUM

# tar -zxvf ossec-hids-*.tar.gz
# cd ossec-hids-*
# ./install.sh
Basic server/agent configuration

After the server configuration, you will need to manage the agents. On the server you will use manage_agents command to insert a number of agents with their ids, names and ip addresses.

****************************************
* OSSEC HIDS v2.8 Agent manager.       *
* The following options are available: *
****************************************
   (A)dd an agent (A).
   (E)xtract key for an agent (E).
   (L)ist already added agents (L).
   (R)emove an agent (R).
   (Q)uit.
Choose your action: A,E,L,R or Q:

After adding the agents on the server, you need to extract the agent keys.

Choose your action: A,E,L,R or Q: e

Available agents:
   ID: 001, Name: NAME, IP: IP
Provide the ID of the agent to extract the key (or '\q' to quit): 001

Agent key information for '001' is:
IMPORTANT_HASH

You now need to add the hash to the agent, through manage_client.

****************************************
* OSSEC HIDS v2.8 Agent manager.       *
* The following options are available: *
****************************************
   (I)mport key from the server (I).
   (Q)uit.
Choose your action: I or Q: i

* Provide the Key generated by the server.
* The best approach is to cut and paste it.
*** OBS: Do not include spaces or new lines.

Paste it here (or '\q' to quit): IMPORTANT_HASH

Agent information:
   ID:001
   Name:NAME
   IP Address:IP

Confirm adding it?(y/n): y
Added.

If you remembered to configure the firewall rules properly, allowing traffic on UDP 1514, you should now have them synced upon restart. If everything is working as expected you will find the ossec-agentd connection in the logs within /var/ossec/logs/ossec.log: ossec-agentd(4102): INFO: Connected to the server (hostname/ipaddress:1514).

Adding global agent configurations

One of the smart moves that extend the capability of OSSEC is the possibility to push configurations to the agents. Anyone who managed a botnet knows how powerful this can be, and OSSEC is no exception. Let’s suppose we’re behind a static IP, say, 1.2.3.4: by logging in through SSH, moving files through FTP and changing configuration files around we would generate a lot of white noise, but we can fix that by adding a simple agent configuration on our server side:


  
    1.2.3.4
  

After a reset of the OSSEC processes the agent.conf will be pushed/pulled, and the IP should be now successfully white-listed. This method also allows to set specific rules for sets of agents, by specifying the names to which the configurations apply.

Agent configuration: we need to go deeper

As explained in this article, stopping to the defaults is no good practice. While all the base scenarios have been covered, specific needs have not. Using multi-user hosting or logging? You need to add these logs manually. Mail servers? These too. For some reasons you have verbose MySQL logging? This will need to be added too. That’s easily done by simply appending the specified logs and type to either the agent ossec.conf or the server agent.conf, whichever suits your needs best:

  
    syslog
    /var/log/dovecot.log
  

  
    apache
    /var/log/domains-*.log
  

Remember that you can use wildcards and strftime for the logs, but not together. Also there are a few pitfalls in using wildcards you should be aware of.

Tweaking the server for Splunk

At this point we have a working agent/server configuration, but we want to push it a step further to make use of Splunk. Even though my setup has OSSEC and Splunk sharing the same machine I chose a syslog client configuration, and the reason is simple: through the use of syslog_output I am able to increase the granularity by raising or lowering the alert level as I see fit, while also allowing me to add a separate OSSEC server elsewhere without the need to reconfigure Splunk. It’s a win-win. The changes are to be made inside ossec.conf:

  
    127.0.0.1
    PORT_NUMBER
  

You should put the syslog_output before the <rules> tag. This is all it takes to be ready for Splunk

Where to start Splunking

Silly puns aside, we will need the Splunk software and the Reporting and Management for OSSEC. Given my setup I downloaded the deb package on the server, and the app tgz on my workstation. The installation is as easy as running a few commands:

# dpkg -i splunk---.deb
# /opt/splunk/bin/splunk enable boot-start -user splunk

On a Ubuntu server this will install the required files, and make it start on boot running as splunk user. Before running it though, we need to make a change that will allow us to receive information from OSSEC. The following code can be added in the inputs.conf after the [default] section:

[udp://127.0.0.1:PORT_NUMBER]
disabled = false
sourcetype = ossec

This will start the UDP server, as per our mission. There are other modes available if you chose not to use the syslog_output method, but I will not go into that for now, I will just leave you the app documentation as reference.

At this point most of our work is done. Once the server is started (with service splunk start in my case) you can connect to it through its web interface, which should be up at http://ipaddress:8000/ and perfectly running. After the login you can navigate to App > Manage Apps… and click Install app from file, selecting the app tgz we downloaded earlier. If everything has been done correctly data should be now flowing, and a simple sourcetype=”ossec” query should hold all the collected information.

What to do with it, you ask? Well, that’s your job now 🙂