Security Logging Service


Overview

Penn’s critical hosts and applications are distributed across its Schools and Centers, but unfortunately, only a small portion of security problems can be proactively identified using external scans; the rest must be observed by logging and monitoring system-level events. Therefore, ISC provides the Security Logging Service, making monitoring easier for Penn's system owners, and allowing ISC Information Security to more quickly and effectively observe compromises and campus-wide threats. ISC is offering the Security Logging Service at no charge for security-relevant data.

Benefits

You are invited to participate in this Service, which is intended to:

  • Provide a secure, centralized repository for storing security-relevant logs from different sources (Windows, Linux, Apache, etc.).
  • Provide a platform from which to search, view, analyze, alert and report on security events to spot anomalies.
  • Provide Information Security a campus-wide view of system events, to help more effectively detect and alert on threats to campus systems and data.

By participating in the Security Logging Service, you'll be helping both your School or Center and the Penn community better understand and address the security risks affecting its systems.

How it Works

The service is based on Splunk, a powerful tool for collecting and analyzing machine data. To learn more about how Splunk can be used to understand system events, visit Splunk's website on machine data.

Your systems must first be configured to send filesystem, network, and application logs to the Security Logging Service. This is usually done through either:

  • A forwarding agent is installed on our host, sending its logs to the Security Logging Service.
  • Sending syslog events to a server running a forwarding agent, which then relays them to the Service.

ISC will work with you to find the best means for getting your system logs into the Service. Once your systems are reporting logs, you will be given access to the Service, where you will be able to search your logs using the Splunk Search Processing Language as well as visualize security-relevant events associated with common platforms or applications (Windows, Linux, Apache) using several basic pre-made dashboards.

The first step to using Splunk to analyze your servers' activity is to work with the ISC Splunk team to configure the connections between your system and the Splunk server.

Submit a request via help@isc.upenn.edu to send the following information to the Splunk Support team:

  • The number of hosts you'd like to have submit logs.
  • The operating systems your hosts are running.
  • The file system, network and application logs you'd like to send to the Service.
  • The approximate daily volume of log data you anticipate sending to the Service.

Deciding what events to log and send to the Security Logging Service will vary significantly between systems and organizations. In general, ISC Information Security recommends to begin logging the following types of events:

  • Authentication events (e.g., log-ins).
  • Event logs that can provide visibility into anomalous behavior (e.g., error logs).

For an additional and more detailed description of log sources and types, please refer to the Security Logging Guidance document and our recommended standards for event logging. 

  1. ISC asks that you prioritize forwarding logs from your registered Critical Components, should you have any.
  2. The Security Logging Service is being provided as-is, and its searching and alerting capabilities are designed to augment, not replace, existing School/Center IT and business processes.
  3. ISC will be evaluating aggregate logs from all hosts reporting to Splunk in order to identify trends and alert on possible signs of compromise or attack across campus.
  4. ISC will work with log providers to better analyze the security implications of an event or series of events to the extent possible based on available resources. This will include a collaborative investigation to identify and reduce false positives and/or true negatives.
  5. The saved searches provided by ISC are meant to guide system owners' use of Splunk for security monitoring. System owners are encouraged to develop their own searches to better understand their systems' security and performance issues.
  6. Positive results for searches compiled by ISC are not confirmation that security events have occurred, are occurring, or will occur on a given system.
  7. Similarly, the absence of results for searches compiled by ISC does not mean that security events have not, are not, or will not take place on your system.
  8. ISC has purchased a Splunk license permitting us to index up to 250GB of data per day. While we do not anticipate exceeding this quantity, should this happen we will work with the IT community to ensure that all Schools/Centers have an equal opportunity to participate in the service.
  9. ISC can make no guarantees as to the length of time your logs can be retained, due to variability in (a) the number of clients signing up for the service and (b) the volume of logs each client will be forwarding to the Service.
  10. ISC is offering the Service to Penn IT departments at no charge for security-relevant logs.
  11. While Splunk is a powerful data analysis and visualization tool, it's currently being provided for the primary purpose of assisting with the identification of information security-related events (e.g., identification of threats or compromise to Penn networks, systems, and data). Other uses of the product are not supported by ISC at this time.
  12. The Security Logging Service is configured to preserve logs and events for 60 days.

The purpose of the Security Logging Service is to improve Penn's ability to detect and respond to threats to its information system. As such, an effort should be made by all parties to limit the amount of sensitive data being sent to the Service to just that necessary for detecting emerging threats and signs of compromise.

By configuring your hosts to send system logs to the Security Logging Service, you grant ISC permission to analyze the logs for signs of anomalous, suspicious or malicious activity. Access control provided by the Splunk product will logically separate your data from other clients' data. Other clients of the Service will not have access to your log data.