# File Space Accounting

# Introduction

File Space Accounting allows recursive monitoring of file usage and provides usage reports that can be grouped by user ID, file type, project, hosts, and more.

The StorageAnalyzer collects file space data locally from every client where the StorageAnalyzer is enabled. A central configuration file on the Core Server is used to specify which file spaces to account. Each file space is accounted locally since this accounting process is I/O intensive. Subdirectories that are mounted over the network are not accounted by default due to I/O intensity, and because networked file systems are often mounted on many hosts. But network-mounted file systems are accounted if file space is configured to do so in the file systems configuration.

The first configuration that affects file space accounting is the file systems configuration file. File systems excluded in this file are also excluded from file space accounting. As mentioned previously, all non-local file systems are excluded by default.

This provides a clear understanding of how file space is being utilized. This will produce the following aggregated data types used for historical reporting:

(29) Filespace

The following section will guide you on configuring file space accounting.

# Requirements

An Open iT Client connected to an Open iT Server or a coexistent Open iT setup
Activated StorageAnalyzer and file space collector

# Configuring File Space Accounting

Open iT collects file space data from all clients. A central configuration file on the Open iT server specifies which file spaces to account for. The process is I/O intensive; therefore, each file space is accounted for locally. By default, subdirectories mounted over the network are not included in the accounting. However, network-mounted file systems can be included if configured to do so.

The filespace_win.conf configuration file contains a list of mappings to filter the components considered in storage monitoring.

The server distributes the filespace_win.conf configuration to the clients automatically. Any modification on the client will be lost.

The file has three main configuration parts:

Name	Description
site	This includes configuration for various sites, domains, or workgroups that will be included in storage monitoring.
exclude	This includes configuration for various files that will be excluded in storage monitoring.
map	This includes mapping configuration for specific hosts and files to a specific account.

Parts of the File Space Configuration File

# Adding Site to File Space Storage Monitoring

In the server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

Locate the line:

filespace_win.conf
all
{
# Pattern that must match for this site/domain/workgroup to be considered.
# Use * to match all possible sites/domains/workgroups.
# Example of using a workgroup:
# site=WORKGROUP
site=*
...

Add new entries as needed. following the format:
```
site=<site_name>
```
Example
```
site=*
site=SVG.OPENIT.LOCAL
site=LOCAL
```
The example means to include storage monitoring for the domain SVG.OPENIT.LOCAL and the workgroup LOCAL.
Save the changes.

# Excluding Files to File Space Storage Monitoring

Excluded file systems indicated in the configuration file are also excluded in file space accounting. All non-local file systems are also excluded by default.

In the server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

Locate the line:

filespace_win.conf
# Files to be excluded from accounting.
# Default is to exclude names enclosed in curly braces and names ending in
# .sys (system files).
# NOTE: Some of the files excluded by default may not be accessible, and
# warnings will be reported and incomplete reports generated if they are
# included.
exclude={*}
exclude=*.sys

Add new entries as needed. following the format:
```
exclude=<file_name>
```
Example
```
exclude=*tmp
exclude=*temp
```
The example will exclude temporary files (files ending with .tmp and .temp) in file space storage monitoring.
Save the changes.

# Adding Maps to File Space Storage Monitoring

In the server, open filespace_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration. Read through the written instructions for configuration.

Locate the line:

filespace_win.conf
# Mappings.
map
{
  # Global match. Account all systems on "My Computer" if no other config is available.
  global
  {
	host=*
	match
	{
		all
		{
			dir=*
			account=UNDEF
		}
	}
}

Add new blocks under the map block, following the format:
Example
```
map
{
...
  temp
  {
    host=mnl164win
    match
    {
       temps
       {
         dir=?:\temp*
         account=TEMP
       }
    }
  }
```
The example will include temporary files in the host MNL164WIN for storage monitoring.
Save the changes.

The file_space configuration file contains a list of mappings to filter the components considered in storage monitoring.

The server distributes file_space configuration to the clients automatically. Any modification on the client will be lost.

The file has three main configuration parts:

Name	Description
DOMAIN	This includes configuration for various domains or workgroups that will be included in storage monitoring.
EXCLUDE	This includes configuration for various files that will be excluded in storage monitoring.
map	This includes mapping configuration for specific hosts and files to a specific account.

Parts of the File Space Configuration File

# Adding Domain to File Space Storage Monitoring

In the server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.
Add a new line for the DOMAIN using the following syntax:
DOMAIN syntax
```
DOMAIN:         <domain_name>
```
DOMAIN Example
```
DOMAIN:         samplecompany.com
```
This example indicates that the hosts to be accounted for are within the samplecompany.com domain.

Lines starting with hash signs (#) are considered comments and are ignored in the processing.
Save the changes.

# Excluding File to File Space Storage Monitoring

In the server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.
Add a new line for the EXCLUDE using the following syntax:
EXCLUDE syntax
```
EXCLUDE:      <files_or_directories_to_exclude_from_accounting>
```
EXCLUDE Example
```
EXCLUDE:      \.snapshot
```
This example will exclude .snapshot files from the accounting.

Add new entries as needed. Consider using regular expressions to better match the files or directories to be excluded. Make sure that all instructions and rules are followed to avoid malfunctions.

Entries are processed from top to bottom, and the last pattern that matches is prioritized.

Lines starting with hash signs (#) are considered comments and are ignored in the processing.
Save the changes.

# Adding Maps to File Space Storage Monitoring

In the server, open file_space in the Configuration directory, which is by default in /var/opt/openit/etc. Read through the written instructions in the file.
Add a new mapping line using the following syntax:
maps syntax
```
<hostname>:<directory>      <account_name_or_group>
```
EXCLUDE Example
```
fs1:/testfs    TEST
```
The example above enables file space accounting on host fs1 and the directory /testfs is accounted to TEST.

Add new entries as needed. Consider using regular expressions to better match the directories and accounts. Make sure that all instructions and rules are followed to avoid malfunctions.

Entries are processed from top to bottom, and the last pattern that matches is prioritized.

Lines starting with hash signs (#) are considered comments and are ignored in the processing.
Save the changes.

# Configuring File Type Classification

File space accounting collects all file data included in file systems by default. Three possible methods (levels) classify the files:

Level	Description
Level 0 Accounting	Select this as a faster alternative if the file type is of no interest. The file type will be regarded as undefined and set to undef.
Level 1 Accounting	This is the default method that uses filenames to decide on the file type.
Level 2 Accounting	This is the most advanced and slowest method that uses the file contents to decide on the file type.

File Type Levels

Open filespace.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.
Locate the filespace block.

Change the value of the filetype attribute to the desired value from 0 to 2. By default, value is set to 1.

filespace.conf
filespace()=""
{
   log()=""
    {
      debug-level(Integer)="0"
    }
    working-dir(DirName)="${OpeniT.directories.data}/FileSpace"{}
    temp-dir(DirName)="${OpeniT.directories.temp}"{}
        ...
    filename(FileName)="${OpeniT.directories.conf}/filename_win.conf"{}
    filetype(Integer)="1"{}
    temperatures(FileName)="${OpeniT.directories.conf}/file_temperature"{}
...

Save the changes.

For Unix systems, you may refer to the Configuring File Type Classification in Unix page.

# Configuring File Names to File Type Mapping

Configure the filename_win.conf file to map file extension to file types using the default file type Level 1.

The server distributes filename_win.conf configuration to the clients automatically. Any modification on the client will be lost.

Before mapping, verify the level and file used:

Open filespace.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.
Locate the filespace block.

Verify the values of the filename and filetype attributes.

filespace.conf
filespace()=""
{
   log()=""
    {
      debug-level(Integer)="0"
    }
    working-dir(DirName)="${OpeniT.directories.data}/FileSpace"{}
    temp-dir(DirName)="${OpeniT.directories.temp}"{}
        ...
    filespace(FileName)="${OpeniT.directories.conf}/filespace_win.conf"{}
    filename(FileName)="${OpeniT.directories.conf}/filename_win.conf"{}
    filetype(Integer)="1"{}
    temperatures(FileName)="${OpeniT.directories.conf}/file_temperature"{}
...

After verification:

In the server, open filename_win.conf in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.
Read through and follow the written instructions for configuration. Use any syntax below:
filename_win.conf (Using Comma)
```
<file_type>
{
    <file_extention1>,<file_extention2>,<file_extentionN>
}
```
filename_win.conf (Using New Line)
```
<file_type>
{
    <file_extention1>
    <file_extention2>
    <file_extentionN>
}
```
filename_win.conf (Using both Comma and New Line)
```
<file_type>
{
    <file_extention1>,<file_extention2>,<file_extention3>
    <file_extention4>
    <file_extentionN>
}
```
Example
```
binary
{
    exe,com,bin
    ocx,sys,scr
}
```
This example will map exe, com, bin, ocx, sys, and scr to file type binary.

By default, the file already has defined mappings. Add new entries as needed. Globbing is allowed in extension mapping. Use only lower case for file extensions. Make sure that all instructions and rules are followed to avoid malfunctions.
Save the changes.

Configure the filename_type file to map file extension to file types using the default file type Level 1.

The server distributes filename_type configuration to the clients automatically. Any modification on the client will be lost.

Before mapping, verify the level and file used:

Open openit.cfg in the openit directory, which is by default in /etc/opt/openit/.
Locate the STORAGE_FILE_SPACE_LEVEL and STORAGE_FILENAME_TYPE_MAP.

Verify that the values are 1 and filename_type, respectively.

openit.cfg
STORAGE_FILE_SPACE_LEVEL    1
STORAGE_FILENAME_TYPE_MAP   filename_type

After verification:

In the server, open filename_type in the etc directory, which is by default in /var/opt/openit/etc.
The mapping file is line oriented and contains two types of lines: regular expression mappings and the comments.

The regular expression mappings begin with match: followed by a Perl type regular match expression and a file type.
filename_type
```
match:      <Perl_type_regular_expressions>       <file_type>
```
The Perl type regular expression may match files depending on directory or file name.

The regular expression uses the entire file path of the file.
Example
```
match:      \/bin\/[^/]+$       binary
```
This example will map any string that ends with /bin/<string> to file type binary. <string> is a single file or name that does not include another /.

By default, the file already has defined mappings. Add new entries as needed. Make sure that all instructions and rules are followed to avoid malfunctions.
Save the changes.

# Configuring File Temperature

The file_temperature configuration file contains various temperature levels for the files accounted for. There are five default temperature levels, each with its own default label and upper boundary (in days).

file_temperature
# Temperature settings for file_space accounting.

# This file has the default settings for five
# temperature levels.
# Temperature labels  Upper boundary(days)
# ------------------  --------------------
hot                   30
warm                  90
tepid                 360
cold                  720
frozen                infinity

In this case, the lower boundary is considered as the upper boundary of the previous level. The lower boundary of the first level is assumed to be 0. The last level should have no number but infinity. This means that there will be no upper limit for this temperature level.

The server distributes file_temperature configuration to the clients automatically. Any modification on the client will be lost.

In the server, open file_temperature in the Configuration directory, which is by default in C:\Program Files\OpeniT\Core\Configuration.
Modify the configuration file as needed.

Temperature levels must be in ascending order of the upper boundary values.
Save the changes.

In the server, open file_temperature in the Configuration directory, which is by default in /var/opt/openit/etc.
Modify the configuration file as needed.

Temperature levels must be in ascending order of the upper boundary values.
Save the changes.

There may be any number of temperature levels, but there is no way of changing the levels for old data. The temperature levels of data collected earlier remains unchanged, even if the default settings are changed.

# Configuring File Space Period and Offset

Go to the scheduler directory, which is by default in C:\Program Files\OpeniT\Core\Configuration\scheduler, and open collect_storage_filespace-win.oconf.
Locate and set the value of root.scheduler.jobs.collect_filespace_win.scheduling.start-triggers.trigger#1.period. The default value is P7D.
collect_storage_filespace-win.oconf
```
66| period
67| {
68|     description=Run weekly
69|     type=timespan
70|     value=P7D
71| }
```
Locate and set the value of root.scheduler.jobs.collect_filespace_win.scheduling.start-triggers.trigger#1.offset. The default value is P-5DT2H.
collect_storage_filespace-win.oconf
```
72| offset
73| {
74|     description=Rather than running at midnight Monday, start 2 hours before midnight on Saturday
75|     type=timespan
76|     value=P-5DT2H
77| }
```
Specify the period and offset values to your desired timespan based on the ISO-8601 duration fromat PnDTnHnMn.nS.
Save the changes.

For Unix systems, you may refer to the Configuring File Space Period and Offset in Unix page.

# Sample Historical Report

File Storage Report for svg_host0045