NIST SP 800-61
Handling an Incident
4.1. Lessons Learned
One of the most important parts of incident response is also the most often omitted: learning and
improving. Each incident response team should evolve to reflect new threats, improved technology, and
lessons learned. Holding a "lessons learned" meeting with all involved parties after a major incident, and
optionally periodically after lesser incidents as resources permit, can be extremely helpful in improving
security measures and the incident handling process itself. Multiple incidents can be covered in a single
lessons learned meeting. This meeting provides a chance to achieve closure with respect to an incident by
reviewing what occurred, what was done to intervene, and how well intervention worked. The meeting
should be held within several days of the end of the incident. Questions to be answered in the meeting
include:
- Exactly what happened, and at what times?
- How well did staff and management perform in dealing with the incident? Were the documented procedures followed? Were they adequate?
- What information was needed sooner?
- Were any steps or actions taken that might have inhibited the recovery?
- What would the staff and management do differently the next time a similar incident occurs?
- How could information sharing with other organizations have been improved?
- What corrective actions can prevent similar incidents in the future?
- What precursors or indicators should be watched for in the future to detect similar incidents?
- What additional tools or resources are needed to detect, analyze, and mitigate future incidents?
Small incidents need limited post-incident analysis, with the exception of incidents performed through
new attack methods that are of widespread concern and interest. After serious attacks have occurred, it is
usually worthwhile to hold post-mortem meetings that cross team and organizational boundaries to
provide a mechanism for information sharing. The primary consideration in holding such meetings is
ensuring that the right people are involved. Not only is it important to invite people who have been
involved in the incident that is being analyzed, but also it is wise to consider who should be invited for the
purpose of facilitating future cooperation.
The success of such meetings also depends on the agenda. Collecting input about expectations and needs
(including suggested topics to cover) from participants before the meeting increases the likelihood that the
participants' needs will be met. In addition, establishing rules of order before or during the start of a
meeting can minimize confusion and discord. Having one or more moderators who are skilled in group
facilitation can yield a high payoff. Finally, it is also important to document the major points of
agreement and action items and to communicate them to parties who could not attend the meeting.
Lessons learned meetings provide other benefits. Reports from these meetings are good material for
training new team members by showing them how more experienced team members respond to incidents.
Updating incident response policies and procedures is another important part of the lessons learned
process. Post-mortem analysis of the way an incident was handled will often reveal a missing step or an
inaccuracy in a procedure, providing impetus for change. Because of the changing nature of information
technology and changes in personnel, the incident response team should review all related documentation
and procedures for handling incidents at designated intervals.
Another important post-incident activity is creating a follow-up report for each incident, which can be
quite valuable for future use. The report provides a reference that can be used to assist in handling similar
incidents. Creating a formal chronology of events (including timestamped information such as log data
from systems) is important for legal reasons, as is creating a monetary estimate of the amount of damage
the incident caused. This estimate may become the basis for subsequent prosecution activity by entities
such as the U.S. Attorney General's office. Follow-up reports should be kept for a period of time as
specified in record retention policies.