A User's Guide to the Scripts

So far, the discussion has been a bit theoretical, setting up the general idea of how DNS management might be accomplished with Ganymede. This chapter deals a bit more directly with the processes involved in actually working with the DNS data and converting it between the BIND and XML formats. As described earlier, Ganymede doesn't emit zone files directly, but rather generates an XML file which is then converted into zone files by an external script. Also, Ganymede doesn't read zone files directly when initially populating the database with DNS information. It relies on another external script to convert the zone files into XML, which it can then import.

There are two main (Perl) scripts involved, dns2xml and xml2dns, that handle the necessary conversions, along with some additional scripts such as d2xpp that work in conjunction with them. The next few sections describe the roles they play and how to use them. A more development-oriented overview can be found here.

d2xpp

Because of the variations in the formatting and structure of zone files, the first task in the importing process is to try to transform them into a relatively uniform format that dns2xml can process. The d2xpp script is a preprocessor for dns2xml that performs this transformation. It serves three main purposes:
Expansion of Typing Shortcuts
Traditionally, DNS zone files have been edited by hand. To reduce the amount of typing required by network administrators, and to reduce the visual clutter in general, there are several typing shortcuts which can be used when entering the resource record data for a zone. The main ones are as follows:

The d2xpp script will attempt to expand these shortcuts, substituting the origin for @, adding the origin to any DNS names that don't end in a period, and adding explicit owner and class fields to each resource record as needed.

The following zone file fragment (for zone mydomain1.com.) illustrates the transformation.

Before:

@	IN	SOA ns1.mydomain1.com. root.ns1.mydomain1.com.(
			1999082501	; Serial (YYYYMMDDNN)
			3600		; Refresh
			600		; Retry
			604800		; Expire
			10800 )		; Minimum TTL

@		NS	dns1.mydomain1.com.
@		NS	dns2.mydomain1.com.

www	IN	A	1.2.3.4
                A       1.2.3.5
                A       1.2.3.6
cheddar	IN	A	1.2.3.7
swiss	IN	A	1.2.3.8

After:

mydomain1.com.       IN	SOA ns1.mydomain1.com. root.ns1.mydomain1.com.(
			        1999082501	; Serial (YYYYMMDDNN)
			        3600		; Refresh
			        600		; Retry
			        604800		; Expire
			        10800 )		; Minimum TTL

mydomain1.com.		IN	NS	dns1.mydomain1.com.
mydomain1.com.		IN	NS	dns2.mydomain1.com.

www.mydomain1.com.	IN	A	1.2.3.4
www.mydomain1.com.      IN      A       1.2.3.5
www.mydomain1.com.      IN      A       1.2.3.6
cheddar.mydomain1.com.	IN	A	1.2.3.7
swiss.mydomain1.com.	IN	A	1.2.3.8

Processing of Directives
Zone files can also include directives, which specify some action to be taken. The three most common (only?) are as follows:

$ORIGIN domain-name
This changes the origin for the current zone file to domain-name. Any DNS names that don't end with a period will have this origin appended. There is an implicit $ORIGIN zone-name at the beginning of the file.

$INCLUDE file-name [origin]
This includes the file designated by file-name into the current file at the point of the directive statement. An optional origin can be provided for the processing of the included file. This origin is only used for the included file and reverts back to the original origin after the inclusion.

$TTL default-ttl
This sets the default time-to-live for all resource records in the file that don't specify their own ttl. The default-ttl can range from 0-2147483647.

The d2xpp script will take any origin directives into account when completing domain names, and will physically import any included files into the current file. However, per-record ttl values are not currently supported by the GANY_DNS document type definition, so d2xpp does not process the $TTL directive in any meaningful manner.

Error Checking
The d2xpp script is probably the most appropriate place to perform any sanity checking on the original zone file data. This includes things like flagging invalid IP addresses (999.999.999.999, for example), invalid characters in DNS names, aliases appearing where only canonical names should appear, and other potential sources of trouble. See the list of Common DNS Operational and Configuration Errors for more common errors.

Unfortunately, d2xpp currently performs no such error checking. It was hacked together at the beginning of this project just to have uniform data to feed to dns2xml, and has barely been touched since. It definitely needs to be enhanced/rewritten.

The end result of the preprocessing is a series of zone files that have all the common typing shortcuts expanded (most importantly, all DNS names are fully qualified, taking into account any $ORIGIN directives), and physically contain all the external files that were originally specified by the $INCLUDE directive. The transformed zone files are valid zone files in their own right, just fully expanded. As mentioned earlier, error-checking should also take place, but currently doesn't. At this point, the zone files should be in a form that the dns2xml script can reasonably work with.

Options

The options for running d2xpp are straightforward:
d2xpp -i input-directory -o output-directory [-d]

-i input-directory
This is the directory where all the original zone files are located.

-o output-directory
This is the directory where all the transformed zone files are to be located. Make sure it is different from input-directory.

-d
Turns on some debugging output.

Limitations

dns2xml

The dns2xml script is designed to take a series of zone files (preprocessed by d2xpp) and convert the DNS information contained within those files into XML format. See the chapter titled XML's Role for a discussion of why this conversion is performed, why XML is used, and an example of the conversion process.

Options

There are numerous options that can be specified when running dns2xml:

dns2xml -c conf-file -i in-dir -o out-file -n bind-conf -p subpart-method -x -s -h -d

-c config-file
This option specifies a configuration file for dns2xml to use. The default is a configuration file called dns2xml.conf in the input directory (where the preprocessed zone files are located.) Many of the command line options can be specified in this configuration file, but most importantly, it defines the (required) default forward and reverse zone information.

The configuration file is actually Perl code defining variables that get imported into dns2xml via a do statement. It looks like this:

# dns2xml.conf - Configuration file for dns2xml.
# Values on command line override corresponding values below.


# The directory where the preprocessed zone files are located.
$Input_Dir         =  "/path/to/preprocessed/zone/files/"; 

# The name of the XML file to be generated. 
$Output_File       =  "d2x_output.xml";

# The name of the BIND configuration file.
$Named_Config_File =  "named.conf";

# The level of debugging information output.
$Debug             =  0; # 0...3

# The method by which ambiguous systems are handled.
$Sub_Part_Method   =  ""; # none|sys_unique|ip_unique

# Explicitly generate all info that is usually implied in the XML.
$Expand            =  0;

# Don't generate the pretty-printing indentation for the XML output.
$Squash            =  0;

# The default info for forward zones. Edit as necessary. REQUIRED.
$Default_Forward =  
  {
   # Basics 
   'TYPE'     => 'DEFAULT',
   'NAME'     => 'FORWARD',
   'FILE'     => '',        

   # SOA Values
   'REFRESH'  => 10800,
   'RETRY'    => 600,
   'EXPIRE'   => 604800,
   'MINTTL'   => 21600,
   'HOST'     => 'ns1.mydomain1.com.',
   'MAILADDR' => 'root.ns1.mydomain1.com.',
   
   # Nameservers
   'NS' => {
	    'dns1.mydomain1.com.' => 1, # The '1' value just indicates
	    'dns2.mydomain1.com.' => 1, # that the hash key is defined.
	   },

   # Default MX hosts
   'MX' => {
	    '10 mxhost1.mydomain1.com.' => 1,
	    '15 mxhost2.mydomain1.com.' => 1,
	   },
  };                  

# The default info for reverse zones. Edit as necessary. REQUIRED.
$Default_Reverse =  
  {
   # Basics 
   'TYPE'     => 'DEFAULT',
   'NAME'     => 'REVERSE',
   'FILE'     => '',

   # SOA Values
   'REFRESH'  => 10800,
   'RETRY'    => 600,
   'EXPIRE'   => 604800,
   'MINTTL'   => 21600,
   'HOST'     => 'ns1.mydomain1.com.',
   'MAILADDR' => 'root.ns1.mydomain1.com.',
   
   # Nameservers
   'NS' => {
	    'dns1.mydomain1.com.' => 1,
	    'dns2.mydomain1.com.' => 1,
	   },

   # Default MX hosts
   'MX' => {
	    '10 mxhost1.mydomain1.com.' => 1,
	    '15 mxhost2.mydomain1.com.' => 1,
	   },
  };


1; # required to return an 'OK' value to the 'do' statement.

-i input-directory
This is the directory where all information, including zone and configuration files, is expected to be found, unless otherwise specified.

-o output-file
This is the name of the XML file that is to be generated. It defaults to d2x_output.xml in the input-directory.

-n bind-config
This is the name of the BIND configuration file that is used to determine the zones to process. It defaults to named.conf in the input-directory.

-p subpartition-method
Determines how to resolve ambiguous system information. An ambiguous system is one where there is a question of whether a particular DNS name in the system is the name of an interface of that system or whether it actually belongs in its own system. The situation arises when the DNS name in question shares an IP address with the system's DNS name, but also has an IP address that the system DNS name does not have.

Here's an example (with extremely simplified notation):

x  IN  A   1
x  IN  A   2  
x  IN  A   3
y  IN  A   1
y  IN  A   5
In this case, x would be designated the system by dns2xml, and y the name of the 1 interface of x. The problem lies with the 5 interface. It is not related to x, but is related to y, and y IS related to x. How does 5 actually fit in?

The choices for resolving the ambiguity are as follows:

sys_unique
We say that an IP address that appears in one <SYSTEM> can also appear in other <SYSTEM> elements, but that the uniqueness of DNS names between systems must be maintained. In the given problem, the 1 interface is repeated in both systems but the y DNS name only appears in System 2:

System 1:
x  IN  A   1
x  IN  A   2  
x  IN  A   3
 
System 2:
y  IN  A   1
y  IN  A   5

ip_unique
We say that the DNS name of an interface of one <SYSTEM> can have the same name as an entirely different <SYSTEM>, but interfaces can't overlap between the two systems.

In the given problem, the y DNS name is repeated in both systems but the 1 interface only appears in System 1:

System 1:
x  IN  A   1
x  IN  A   2  
x  IN  A   3
y  IN  A   1

System 2:
y  IN  A   5
The DNS data that leads to a situation requiring the subpartitioning option is probably abnormal, but if it occurs, both of these methods will try to fit this existing data into the XML heirarchy of <SYSTEM> elements as just described. The resource records are the same in both cases, they are just assigned to different systems depending on the method chosen.

-x
Generate explicit XML for all data that is normally described implicitly. For example, zone attributes are usually only printed if they differ from the default zone values. The missing attributes are implied using the default values. A similar situation occurs in systems, where the system PTRTYPE attribute can encode which PTR resource records are associated with the system without having to explicitly list any <PTR> elements in the system itself. Using implied values in this way reduces the size of the XML file considerably and reduces the visual clutter quite a bit.

This option is useful for debugging purposes and if the application that reads the XML doesn't know the rules for extracting and applying the implied values.

-s
Prints the XML without any "pretty" indentation. Reduces the file size a little, but makes it very hard for human reading.

-h
Prints the command line options.

-d debug-level
Provides debugging output at increasing levels of detail, from 0 to 3.

Limitations

xml2dns

As described above, the dns2xml script converts existing DNS zone files to XML. The xml2dns script, on the other hand, performs the conversion in the opposite direction. Ganymede emits an XML file containing DNS data, from which xml2dns then generates the appropriate zone files.

Options

The options for running xml2dns are as follows:
xml2dns -c conf-file -i xml-file -o out-dir -n bind-conf 
        -s serial-file -v verbosity -a -f -d -h

-c config-file
This option specifies a configuration file for xml2dns to use. The default is a configuration file called xml2dns.conf in the output-directory (where the generated zone files are to be located.)

The configuration file is actually Perl code defining variables that get imported into xml2dns via a do statement. It looks like this:

# xml2dns.conf - Configuration file for xml2dns.
# Values on command line override corresponding values below.


$Output_Dir        = "output";
$Input_File        =  "d2x_output.xml";
$Named_Config_File =  "x2d.named.conf";
$Debug             =  0;
$Serial_Num_File   =  "lastserial.dat";
$Alias_As_Cname    =  0;
$Verbosity         =  3;

1;

-i XML-input-file
This is the name of the XML file that contains the DNS data from which the zone files are to be generated. It defaults to d2x_output.xml in the output-directory.

-o output-directory
This is the directory to which the zone files should be generated. At this stage of development, it should be a test directory and NOT the same directory that contains the original zone files.

-n bind-config
This is the name of the BIND configuration file that is to be generated. It defaults to named.conf in the output-directory.

-a
Indicates that all aliases should be represented by A resource records instead of CNAME resource records.

-s serial-file
This is the file that contains the current serial numbers for the zones. The numbers in this file are updated when the zones are written out. It defaults to lastserial.dat in the output-directory.

-v verbosity-level
This indicates the level of "abbreviation" to use when generating the zone files (see the discussion on "typing shortcuts" in the dns2xml section for examples of abbreviations.) A 0 indicates that all shortcuts should be taken, while 1..3 indicate decreasingly aggressive shortcutting (fully-qualifying domain names, replacing @ with the origin, etc.)

This is basically only for the benefit of human readers and debugging. Since the files are automatically generated, the shortcuts don't serve any timesaving functions, and the level should probably just be left at 3.

-f
This adds ID information such as HINFO resource records. By default these records are not generated.

-h
Prints the command line options.

-d debug-level
Provides debugging output at increasing levels of detail, from 0 to 3.

Limitations

Testing the Scripts

At this point in the development of these scripts, the best thing to do is try a roundtrip of the conversion process. That is, use dns2xml to convert existing zone files to XML, and then use xml2dns to generate new zone files from that XML file. Ideally, these generated files should be identical in content (if not in actual order/spacing) to the pre-processed versions of the original files.

Given a directory of BIND zone files called named, the roundtrip process usually goes something like this:

  1. Create a new directory (say named_pp) to contain the pre-processed versions of the zone files in named.
  2. Run d2xpp on the original named directory, specifying named_pp as the output directory. This should generate pre-processed versions in named_pp of the zone files in named, and should also copy the BIND config file (usually named.conf) to the named_pp directory.
  3. Create a dns2xml.conf file (by editing a copy of the default one) and place that in named_pp.
  4. Run dns2xml with named_pp as the input directory and test.xml as the XML file to output.

At this point the complete XML file is generated and can be examined. The next stage is to generate new zone files from test.xml:

  1. Create a new directory (say named_out) to contain the newly generated zone files.
  2. Create an xml2dns.conf file (by editing a copy of the default one) and place that in named_out.
  3. Run xml2dns with test.xml as the input file and named_out as the output directory, using verbosity level 3.
  4. Compare the original files in named_pp with the generated files in named_out (sortdiff* can help with this.)
  5. If there are differences in content, try to figure out why and fix the code!

(*)The sortdiff script is a quick little hack that is basically a Perl wrapper around 'diff' that will examine two directories file-by-file, or two single files, that have first been sorted. This seems to be a simple but effective way to check content without regard to spacing or ordering. It's not really part of the dns/xml scripts so isn't described further here.