Script to archive server logfiles.

[kkomarla]

Is there a mechanism to store more than 50 server log files in Integrity server.

Determined by the configured log level for each log category those 50 files can cover up to 10 days but also less than 1 day is also possible. In case a deeper analysis is required it happens, that the required server logs are no longer available.

We could stay with these 50 files but we do no longer simply delete each older file but zip it which will lead to dramatic file size reduction. A file can be shrinked by round about 90% from 10 MB to 0.8 MB. By creating (if one does not exist) such an automatism it should be possible to easily provide server log files from the last 3-6 months with just an acceptable storage consumption. This could really help in terms of analysis of issues which are discovered not directly and the root cause analysis starts not until the related server log files are already deleted.

My question is "what are the Pro's and Con's in doing so?"

[KaelLizak]

Hello Karthik,

You migth be interested in CS156155: How to Change the Size or Number of server.log files on the Integrity Server. This covers how to change how many server.log files are retained by the Integrity server, and how to configure how large they get before being rotated.

Obviously, based on the information in the article and your experience, the rate of turnover of articles depends not on a particular amount of time, but on your Integrity server's load and the logging levels. I am not aware of a script that would automate moving the log files out and/or zipping them automatically, but that would be a useful tool to share with the Integrity Community.

Zipping files:

Advantage:

• At least a 10:1 savings in terms of space used.

Disadvantage:

• more difficult to search through the files for particular messages (although there are tools such as zcat, gzcat, and zgrep on Unix platforms that make this possible, and I imagine there are Windows parallels as well).

Swapping out files to other file systems:

Advantage:

• You don't lose data as the log files roll over.
• You can more easily gather statistical data.

Disadvantage:

• The value of this information diminishes rapidly with time: You may care about what was showing up in log files last week, or maybe even last month, but what was in log files a week ago becomes less valuable.
• Obviously, the value here is subjective based on your industry, your organization's practices and policies, and how detailed the log files are.

The other advantage of this kind of script is you can have it search for key phrases as it archives each rolled over server.log file. In particular, you can have it look for things like:

• cache warnings: If these come up often, it suggests that you might want to adjust your Source Integrity cache settings
• LDAP warnings: If ldap errors come up often for a particular user, it could suggest someone is attempting to compromise their account, or possibly just that they have a problem remembering or typing their password. Hopefully your LDAP administrator would also notice this sort of thing.

I'm sure there are other pros and cons I haven't thought of on this topic that I haven't thought of. If you've got some, or if you take issue with any of the ones I've presented, please chip in.

Regards,
Kael

[kkomarla]

Hello Kael,

Thanks for the detailed explanation.

Yes, as mentioned I have gone through the article CS156155: How to Change the Size or Number of server.log files on the Integrity Server.

Zipping files:

I agree with you, that searching through the files is not going to be easy. But I thought of saving the files date wise which would help me in narrowing down the search to an extent.

The three things that I have in mid is:

• I am planning to use a command line tool to zip up the files like Winzip or 7-Zip which are easy to use. The self-contained command line version (7za.exe) is the most portable choice.
• Then I hope to save the newly created zip files based on date (name the file as 10-12-2014 or something similar).
• Finally to delete the original files (unzipped ones). Also to save the newly zip files in the same location as the original server log files.

Once I have a script in place I shall get back to you so that it can be further enhanced for other user requirements.

Kind Regards,

Karthik Komarla