Asif Javed

Using strace to Debug Facter Custom Facts in Puppet

Posted on by Asif

When working with Puppet, you may need to send custom facts from your agent nodes to the Puppet server. But where exactly should you place these custom facts? If you’re running Puppet 8.4.0 on Ubuntu 24.04 and struggling to locate the correct directories for Facter, the strace command can be your secret weapon for troubleshooting.

The Problem: Missing Facter Directories

Let’s start by examining what happens when you try to locate Facter directories using standard debugging commands.

Running Facter in Debug Mode

First, execute the facter command with the --debug flag to see what’s happening under the hood:

$ facter --debug

The output reveals important information about Facter’s execution process:

INFO Facter - executed with command line: --debug
DEBUG Facter - Facter version: 4.3.0
DEBUG Facter::FactManager - Resolving facts sequentially
DEBUG Facter::FactLoader - Loading all internal facts
DEBUG Facter::FactLoader - Loading custom facts
DEBUG Facter::FactLoader - Loading external facts

Notice that Facter attempts to load external facts, but it doesn’t specify which directories it’s checking. To isolate just the external facts loading process, you can use grep:

$ facter --debug 2>&1 | grep external
DEBUG Facter::FactLoader - Loading external facts

This confirms that Facter is attempting to load external facts, but we still don’t know where it’s looking.

Checking Puppet Configuration

Next, let’s verify the Puppet configuration directories:

$ sudo puppet config print codedir confdir vardir environmentpath --section agent
codedir = /etc/puppet/code
confdir = /etc/puppet
environmentpath = /etc/puppet/code/environments
vardir = /var/cache/puppet

These are the standard Puppet directories, but they don’t tell us where Facter expects to find custom facts.

Searching for Facts Directories

You might try using find commands to locate facts directories:

$ sudo find / -type f -name "facter.*"
$ sudo find / -type d -name "facts.d"
$ sudo find / -type d -name "fact*.*"

If these commands return no results, it means the expected directories simply don’t exist on your system yet.

The Solution: Using strace to Reveal Hidden Paths

This is where strace becomes invaluable. The strace command traces system calls and signals, allowing you to see exactly which files and directories a program tries to access during execution.

Understanding the strace Command

Here’s the strace command we’ll use:

$ strace -fvttTyy -s 256 -o /tmp/strace.txt puppet agent -t

Let’s break down each option:

  • -f: Follow child processes created by fork() calls
  • -v: Verbose mode – print unabbreviated versions of system calls
  • -tt: Print timestamps with microsecond precision
  • -T: Show time spent in each system call
  • -yy: Print file descriptor paths
  • -s 256: Set the maximum string size to print (256 characters instead of the default 32)
  • -o /tmp/strace.txt: Write output to a file instead of stderr
  • puppet agent -t: The command to trace (runs Puppet agent in test mode)

Analyzing the strace Output

Once the trace completes, search for references to facts.d directories:

$ grep "facts\.d" /tmp/strace.txt

This reveals the directories that Facter actually attempts to access:

read(5</usr/lib/ruby/vendor_ruby/facter/custom_facts/util/directory_loader.rb>, 
"# frozen_string_literal: true\n\n# A Facter plugin that loads external facts.\n#\n
# Default Unix Directories:\n# /opt/puppetlabs/custom_facts/facts.d, 
/etc/custom_facts/facts.d, /etc/puppetlabs/custom_facts/facts.d\n#\n
# Beginning with Facter 3, only /opt/puppetl"..., 8192) = 4456

newfstatat(AT_FDCWD, "/etc/facter/facts.d/", 0x7fffa553a0a0, 0) = -1 ENOENT (No such file or directory)

The key information here shows that Facter is looking for:

  • /opt/puppetlabs/custom_facts/facts.d
  • /etc/custom_facts/facts.d
  • /etc/puppetlabs/custom_facts/facts.d
  • /etc/facter/facts.d/

All of these return ENOENT (No such file or directory), confirming they don’t exist on the system.

Checking for Facter Configuration File

The strace output also reveals attempts to read a Facter configuration file:

$ grep "facter\.conf" /tmp/strace.txt
newfstatat(AT_FDCWD, "/etc/facter/facter.conf", 0x7fffa553a970, 0) = -1 ENOENT (No such file or directory)

This shows that Facter looks for /etc/facter/facter.conf, but this file doesn’t exist either.

Configuring Custom Facts Directories

Now that we know where Facter is looking, we can create the appropriate configuration.

Step 1: Create the Facter Configuration File

First, create the /etc/facter/ directory if it doesn’t exist:

$ mkdir -p /etc/facter/

Then create the Facter configuration file:

$ sudo vi /etc/facter/facter.conf

Add the following JSON configuration to specify custom external fact directories:

{
  "global": {
    "external-dir": [
      "/etc/puppet/code/facts.d",
      "/etc/puppet/code/environments/production/facts.d",
      "/opt/puppet/custom_facts/facts.d"
    ]
  }
}

This configuration tells Facter to look for external facts in three custom locations that align with your Puppet directory structure.

Step 2: Create a Custom Facts Directory

Choose one of the directories you specified in the configuration file and create it:

$ mkdir -p /opt/puppet/custom_facts/facts.d

Step 3: Add a Custom Fact

Create a simple text-based fact file. Facter supports various formats including plain text key=value pairs:

$ echo 'role=caddy' | sudo tee /opt/puppet/custom_facts/facts.d/custom_facts.txt

This creates a custom fact named role with the value caddy.

Testing Your Custom Facts

Now it’s time to verify that everything works correctly.

Test with the External Directory Flag

First, test by explicitly specifying the external directory:

$ sudo facter --external-dir=/opt/puppet/custom_facts/facts.d/ role
caddy

This should return the value caddy, confirming that Facter can read the custom fact when pointed directly at the directory.

Test with Standard Facter Command

Since you’ve created the facter.conf configuration file, Facter should now automatically check your custom directories:

$ sudo facter role
caddy

Success! The custom fact is now available without specifying the directory explicitly.

Verify with Puppet Facts

Finally, test that Puppet can access the custom fact:

$ sudo puppet facts find | grep role -C 5

This command retrieves all facts from the Puppet agent and searches for your custom role fact with 5 lines of context. You should see your custom fact in the output.

Key Takeaways

  1. strace is a powerful debugging tool: When standard commands don’t reveal configuration paths, strace can show you exactly what files and directories a program attempts to access.
  2. Facter configuration is flexible: By creating /etc/facter/facter.conf, you can specify custom directories that match your infrastructure’s organization.
  3. External facts are simple: Custom facts can be as simple as text files with key=value pairs, making them easy to manage and deploy.
  4. System calls tell the truth: The newfstatat system calls in the strace output definitively show which paths Facter checks, eliminating guesswork.
  5. Test incrementally: Always verify your configuration at each step – first with explicit paths, then with automatic discovery, and finally through Puppet itself.

Conclusion

When documentation falls short or default configurations are unclear, low-level debugging tools like strace can provide the answers you need. By tracing system calls, you can discover exactly how applications like Facter locate their configuration files and data directories, enabling you to configure your Puppet infrastructure with confidence.

The next time you’re troubleshooting file location issues with any Linux application, remember: strace is your friend. It reveals the truth about what your programs are really doing, one system call at a time.