RTG README
----------
Contents:
1. Introduction
2. Prerequisites
3. Quick start
4. Compiling
5. Configuration
6. Monitoring Gauges
7. Graphs and HTML Generation
8. Generating Reports
9. Maintenance
10. Troubleshooting
11. OS-Specific Configuration
12. Security
13. Installing the MySQL Perl DBI
14. Future
15. Acknowledgments
1. Introduction
RTG is a flexible, scalable, high-performance SNMP statistics monitoring
system. It is designed for enterprises and service providers that need to
collect time-series SNMP data for a large number of objects from a large
number of network elements quickly. All collected data is inserted into a
MySQL database allowing sophisticated analysis and reporting. The package
includes utilities that generate RTG configuration files, HTML, sample
reports and graphical data plots. RTG is freely available under the terms
of the GNU General Public License.
The primary advantages of RTG are:
- Runs as a daemon, incurring no cron or kernel startup overhead
- Written entirely in C for speed
- Can poll at sub one-minute intervals
- No averaging
- Inserts data into a relational database where complex queries and
reports may be generated
If these features are not important to you, RTG is likely the wrong tool
for the job. Alternatives we highly recommend are MRTG, RRDtool and
Cricket. Please see the FAQ.
2. Prerequisites
RTG comes as C source code and is intended to run on UNIX systems. RTG
requires a UNIX system with POSIX thread support, 64-bit long integers and
a sane compiler (gcc works great). RTG requires two external packages:
- MySQL (http://www.mysql.com/)
- Net SNMP (http://net-snmp.sourceforge.net/)
RTG can also use the older UCD SNMP, but we recommend sticking with
Net SNMP.
To use the built in reports and configuration generators and to generally
get the most out of RTG, it is strongly recommended that you also install:
- Apache (http://www.apache.org)
- PHP (http://www.php.net)
- Perl DBI (http://www.mysql.com/downloads/api-dbi.html)
3. Quick start
RTG is complex to setup, so please read all of the documentation.
a. Build RTG and install into default location /usr/local/rtg:
$ gzip -d rtg-x.y.tar.gz
$ tar -xvf rtg-x.y.tar
$ cd rtg-x.y/
$ ./configure
$ make
# make install
b. Make certain the MySQL database is installed and running.
c. Configure the database. You must know the MySQL root password.
The createdb script will setup the database for RTG:
$ /usr/local/rtg/etc/createdb mysqlroot
d. Edit the rtg-x.y/etc/routers file, adding each router you wish
to SNMP poll, one per line. To specify a per-router non-default
community string, add "router:community" in the routers file.
To specify a per-router non-default response bit width, add
"router:community:bits" in the routers file.
e. Edit rtg-x.y/etc/rtgtargmkr.pl to change the default SNMP read
community if necessary. To use 64bit OIDs (recommended if supported
by the device), change the "bits" field in rtgtargmkr.pl to 64.
f. Use the rtg-x.y/etc/rtgtargmkr.pl script to manage the RTG
target file. The first run will create a targets.cfg file.
$ ./rtgtargmkr.pl
g. Start up the poller to use the target list:
$ /usr/local/rtg/bin/rtgpoll -v -t targets.cfg
If the poller does not find a configuration file, it will create
one in the current directory called "rtg.conf". You may need to
edit this file if your installation is non-standard. If the
poller is successful, the "Polls" counter in the statistics banner
will increase and then a countdown to the next poll is displayed.
The DBInserts should increment after the second polling round.
h. RTG is running. Manipulate or mine the data in the database as
needed, or read the remaining sections for graphing and reporting
instructions.
4. Compiling
Please also see the OS-Specific configuration section of this document
for details on building RTG on specific platforms.
RTG uses GNU autoconf to determine your build type. Compiling should be a
simple matter of:
./configure
make
make install
See "./configure --help" for a complete list of build options.
RTG requires the Net SNMP and thread-safe MySQL libraries; compilation
will fail if configure cannot find these libraries.
Newer MySQL binary distributions include the thread-safe libraries by
default. If you compile your own MySQL, be certain to include the
"--enable-thread-safe-client" option when configuring. If you are uncertain,
check to see if you have a "libmysqlclient_r*" library installed on your
system; this is the reentrant, thread-safe library.
The MySQL database may reside on a different machine than RTG, however RTG
still needs the MySQL libraries. If your MySQL libraries are not in a
standard location, use the --with-mysql=/mysqlpath argument with
configure. If your SNMP libraries are not in a standard location, use the
--with-snmp=/snmppath argument with configure.
RTG has successfully built on: Solaris 2.x, Linux 2.x, and FreeBSD 4.x
Please let us know if you build RTG on other platforms and what
modifications were required. If you experience trouble compiling RTG, see
the Troubleshooting section of this document.
5. Configuration
rtgpoll takes a number of options, rtgpoll -h displays the options.
rtgpoll uses two ASCII files to direct its actions: a configuration file
and a target file. The configuration file specifies general RTG
configuration parameters while the target file lists the devices that RTG
will SNMP poll. The default name of the configuration file is rtg.conf.
A single rtg.conf file can control all programs (including rtgpoll,
rtgplot, *.php and *.pl). Each program attempts to find an rtg.conf
in the following path order:
1. ./rtg.conf
2. /usr/local/rtg/etc/rtg.conf
3. /etc/rtg.conf
If rtgpoll or rtgplot cannot find an rtg.conf file in any of these
paths, it will attempt to create one in the current working directory.
Most users maintain a master rtg.conf file customized for their
environment. rtg.conf contains the following configurable fields:
Interval 300
HighSkewSlop 3
LowSkewSlop .5
OutOfRange 93750000000
SNMP_Ver 1
SNMP_Port 161
DB_Host localhost
DB_Database rtg
DB_User snmp
DB_Pass rtgdefault
Threads 5
Interval is the time between successive polls of the target list,
default is 300 seconds (5 minutes). HighSkewSlop defines the maximum
number of Intervals allowed between two consecutive poll values before the
time in seconds between said points is deemed too large to calculate a valid
rate. With the default Interval and HighSkewSlop values, that time would be
300 * 3 (15 minutes). LowSkewSlop defines the minimum number of Intervals
alloweed between two consecutive poll values before the time in seconds
between said points is deemed two low to calculate a valid rate. With
the default Interval and LowSkewSlop values, that time would be
300 * 0.5 (2.5 minutes). OutOfRange defines an upper bound
above which rtgpoll will never attempt an insert into the database.
OutOfRange should be a multiple of the maximum number of bytes
possible in the defined Interval for your highest speed link. The default
OutOfRange value will suffice in most installations. SNMP_Ver specifies
the SNMP version the poller will use. The number of threads rtgpoll
will use is defined in the variable Threads.
Variables in rtg.conf must match the names above exactly. Comments
and blank lines are allowed and the ordering of variables in rtg.conf
does not matter.
The target file specifies the objects to be SNMP polled. Comments must be
preceded with a '#' sign. Elements in the target file are tab delimited.
The format of the target file is fixed:
# Host OID 64/32 Community Table ID Description
where
Host = IP or hostname of target
OID = Full SNMP OID, e.g. .1.3.6.1.2.1.31.1.1.1.10.19
64/32/0 = Specify 64/32 bit objects or 0 for gauge objects
Community = SNMP Community
Table = MySQL table in the database to use
ID = A unique ID that is used with each insert
Description = Free text
rtgpoll first reads the configuration file, then the target file. For
each SNMP poll, rtgpoll will attempt an SQL INSERT of the form:
INSERT INTO Table VALUES (ID, NOW(), bigint)
Where Table is the name of the database table and ID is an integer. Both
Table and ID come from the target list, NOW() is the current timestamp and
bigint is the delta value between successive SNMP polls.
RTG makes no attempt at determining rate; one must look at the time
difference successive entries in the database. The RTG graphing and
reporting tools automatically calculate rate.
6. Monitoring Gauges
RTG can monitor OIDs that return a gauge value allowing one to monitor
items such as temperature, CPU, users, etc. If an entry in the target
file has 0 (zero) specified as the OID bit width instead of 64 or 32,
the poller assumes that it is monitoring a gauge and does not attempt
to calculate an interval delta value. Currently there are no
automated scripts to generate a gauge target file, but should be
trival for users to implement for their environment.
7. Graphs and HTML Generation
RTG includes PHP pages to serve interactive graphical traffic plots via
the web. Using PHP demonstrates RTG best; we strongly recommend using
this method. Note that we assume an Apache (http:/www.apache.org) web
server with PHP and MySQL support running on a UNIX system.
To configure an apache web server to serve graphical RTG plots via PHP:
a. Build a PHP-capable Apache server (see http://www.php.net)
b. Add "SetEnv LD_LIBRARY_PATH /usr/local/lib:/usr/local/lib/mysql"
to httpd.conf (Or wherever your MySQL libraries are found).
c. Uncomment (or possibly add) "AddHandler cgi-script .cgi" in httpd.conf
d. Add "ExecCGI" to the htdocs Options statement in httpd.conf
e. Uncomment (or possibly add) "AddType application/x-httpd-php .php"
f. Restart apache
g. Copy the PHP files and images to the htdocs directory:
cp /usr/local/rtg/web/* /usr/local/apache/htdocs
h. Copy the rtgplot binary to the htdocs directory and *add* .cgi
extension:
cp /usr/local/rtg/bin/rtgplot /usr/local/apache/htdocs/rtgplot.cgi
i. Point your web browser at the URL (substitute your site in):
http://mysite.com/rtg.php (Interactive Reports)
http://mysite.com/95.php (95th Percentile Queries)
http://mysite.com/view.php (MRTG-style overview + day/wk/mo plots)
The rtgplot executable generates traffic graphs in PNG format. You do not
have to use PHP or our included PHP web pages. One can easily construct
custom web pages by using the proper HTML tags to embed an RTG plot
anywhere. An RTG image can be placed in any web page by using an < IMG >
tag with the appropriate values. For example:
will plot two lines from the MySQL tables ifInOctets_2 and ifOutOctets_2
corresponding to interface 49 on router 2 for the time span 1028606400 to
1028692799 (UNIX epoch seconds). The factor argument allows for bits per
second; the units argument is displayed as the Y-axis label on the plot.
The scalex argument auto-adjusts the X time axis according to the available
data samples rather than according to the actual time span given. If you
are plotting non-continuous data, such as error plots, use the impulses=yes
argument.
8. Generating Reports
RTG includes two Perl scripts in $prefix/bin that generate per-customer
traffic reports. These Perl scripts use the Perl-MySQL DBI (see the
prerequisites section for details). The first script is report.pl:
USAGE: ./report.pl -[##d][##h][##m][##s]
OR: ./report.pl
Give report a customer name or partial customer name as it appears on the
interface description of the network device and report will generate a
traffic report for all matching customers in the network. For example:
Blah Industries Traffic
Period: [08/05/2002 00:00 to 08/05/2002 23:59]
In Out Avg In Avg Out Util Util Max In Max Out Max Ut Max Ut
Connection MBytes MBytes Mbps Mbps In % Out% Mbps Mbps In% Out%
-------------------------------------------------------------------------------------------------------
at-1/2/0.104 core1.abc 18 20 0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01
at-1/2/0.109 core1.xyz 411 586 0.11 0.16 0.07 0.10 0.43 0.50 0.28 0.32
The second script is 95.pl:
USAGE: ./95.pl -[##d][##h][##m][##s]
OR: ./95.pl
The 95.pl script generates 95th percentile traffic rates for a particular
customer for the previous day. For example:
Blah Industries Traffic
Period: [08/05/2002 00:00 to 08/05/2002 23:59]
RateIn RateOut MaxIn MaxOut 95% In 95% Out
Connection Mbps Mbps Mbps Mbps Mbps Mbps
-------------------------------------------------------------------------------
at-1/2/0.104 core1.abc 0.01 0.01 0.01 0.01 0.01 0.01
at-1/2/0.109 core1.xyz 0.11 0.16 0.43 0.50 0.33 0.45
It is convenient to supply these Perl scripts the proper arguments in a
cron job in order to take automated actions (email, bill, report, etc)
each night or week.
A sufficiently capable Perl hacker should be able to modify these example
scripts to suit the particular needs of any organization. We appreciate
any contributed Perl reporting scripts (or clean up our messy Perl).
9. Maintenance
Most installations run the rtgtargmkr.pl script regularly via a cron job.
rtgtargmkr will add new routers and interfaces as they appear in the
network (defined by the routers file). The script will also detect
interface name changes and suggest the proper SQL command to update the
interface description in the database (this is done as a safety mechanism
to have a human involved in the updating process).
rtgpoll accepts a number of signals. SIGHUP forces a reload of the target
file. If rtgpoll is actively polling, the target file is reloaded when
rtgpoll becomes idle. This is useful when automating the target list
creation based on your active network. SIGUSR1 increases the verbosity of
a running rtgpoll; SIGUSR2 decreases the verbosity.
A typical installation will run the rtgtargmkr script nightly and then
send rtgpoll a -HUP signal to force the poller to re-read the target list.
10. Troubleshooting
If you are experiencing problems with RTG, please first consult the FAQ.
You may find the RTG mailing list archives helpful. Questions not
answered by these two means should be sent to the mailing list
(http://fireflynetworks.net/mailman/listinfo/rtg). When submitting
questions to the mailing list, please be sure to include the RTG version,
your machine architecture, and target file size.
RTG is compiled with lots of debug information. Pass multiple '-v' flags
on the rtgpoll argument line to increase the verbosity. Often this will
help you (or the developers) solve the problem.
11. OS-Specific Configuration
FreeBSD:
a. The FreeBSD ports are the easiest way to install the required MySQL and
SNMP libraries, but require additional options. To install the MySQL
thread-safe libraries, issue "make THREAD_SAFE_CLIENT=yes all install"
from within the "/usr/ports/databases/mysql323-server" or
"/usr/ports/databases/mysql40-server" directory depending on desired
MySQL version. Alternatively, if you already have MySQL installed but
not the thread safe libraries, run "make install" from
"/usr/ports/databases/mysql323-server/work/mysql-3.23.55/libmysql_r".
b. Install Net-SNMP with "make all install" from the "/usr/ports/net/net-snmp"
directory. A FreeBSD port should be along soon. FreeBSD rocks.
Linux:
Linux users using RPMs of SNMP and MySQL need to install both the
util and devel RPMs of each package. From example:
Net-SNMP: net-snmp-*, net-snmp-devel-*, net-snmp-utils-*
MySQL: MySQL-client-*, MySQL-shared-*, MySQL-devel-*, MySQL-*
12. Security
RTG uses MySQL with a default password of "rtgdefault" solely to
ease installation. The password should be changed as soon as RTG is
installed and operational. In addition, the MySQL server port
should be open only to trusted hosts and networks. *The default
RTG installation can leave your statistics data vulnerable!* Please
refer to the MySQL website for details on securing your database.
13. Installing the MySQL Perl DBI
The Perl reports and Perl configuration generators use the Perl DBI to
interface with the MySQL database.
a. From http://www.mysql.com/downloads/api-dbi.html download:
- DBI-x.xx.tar.gz
- DBD-mysql-x.xxxx.tar.gz
b. Uncompress and untar both packages:
$ gzip -d DBI-x.xx.tar.gz; gzip -d DBD-mysql-x.xxxx.tar.gz
$ tar -xvf DBI-x.xx.tar; tar -xvf DBD-mysql-x.xxxx.tar
c. Build DBI package:
$ cd DBI-x.xx; perl Makefile.PL; make; make install
d. Build the MySQL DBD package:
$ cd DBD-mysql-x.xxxx; perl Makefile.PL; make; make install
14. Future
RTG is intended to be a foundation upon which to build. Modules, features
and patches are always welcome and appreciated. To contribute, see
the "TODO" file.
15. Acknowledgments
RTG was inspired by the likes of MRTG, RRDtool (Tobias Oetiker, Dave Rand)
and Cricket (Jeff Allen). RTG uses (and useless without) the excellent
work of many others, including: MySQL, UCD/Net-SNMP, gd, png, cgilib
and SNMP_Session. Thanks to SourceForge (http://www.sf.net) for providing
an excellent service to the developer community. Finally, thanks to all
those who have tested and provided patches or suggestions to RTG!