How to collect core dump file of a crashing program that is shipped in Red Hat Enterprise Linux

Resolution

• Red Hat Enterprise Linux 6/7/8 has a service called abrt, which automatically collects the core dump files on the server and stores them inside the /var/spool/abrt.

• ABRT is not present in RHEL 9, The Coredumps can be captured through systemd coredumpctl functionality.

1. Ensure that the system is set up to generate application cores by removing the core limits:

   1. For the current interactive shell.

      Check the current core ulimit.

      # ulimit -c
      0        
      

      Set the core ulimit to unlimited.

      # ulimit -c unlimited
      

   2. For a given user.

      Login as the user and check the current core ulimit as above.

      Edit /etc/security/limits.conf; see 'How to set ulimit values' for details.

   3. For a RHEL 6 service.

      Check the current core ulimit. Replace PID with the PID of your service.

      # cat /proc/31220/limits
      Limit                     Soft Limit           Hard Limit           Units     
      [snip]
      Max core file size        0                    unlimited            bytes     
      [snip]
      

      Add the desired ulimit command(s) to /etc/sysconfig/SERVICENAME; see 'How to set limits (ulimit) for services in RHEL6 and older with upstart, sysvinit' for details.

   4. For a RHEL 7+ service.

      Check the current core ulimit.

      # systemctl show rsyslog | grep LimitCORE
      LimitCORE=0
      

      Set the core ulimit to unlimited. Override the current ulimit by running,

      # systemctl edit SERVICENAME
      

      In the editor, add 'LimitCORE=infinity', example below.

      [Service]
      LimitCORE=infinity
      

      See 'How to set limits for services in RHEL and systemd' for more details.

2. Make sure abrt and abrt-addon-ccpp is installed

   # yum install abrt abrt-addon-ccpp abrt-tui
   

3. Ensure the ccpp hooks are installed:

   # abrt-install-ccpp-hook install
   # abrt-install-ccpp-hook is-installed; echo $?;
   

   The second command should return 0 (hooks installed)

4. Ensure that this service is up and the ccpp hook to capture core dumps is enabled:

   for RHEL6

   # service abrtd start
   # service abrt-ccpp start
   

   for RHEL6, configure services to start on boot. Optional

   # chkconfig abrtd on
   # chkconfig abrt-ccpp on
   

   for RHEL 7 or RHEL 8

   # systemctl start abrtd.service
   # systemctl start abrt-ccpp.service 
   

   For RHEL 7 or RHEL 8, configure services to start on boot. Optional

   # systemctl enable abrtd.service
   # systemctl enable abrt-ccpp.service
   

   Additionally, abrt-journal-core.service might need to be disabled since it conflicts with abrt-ccpp.service:

   # systemctl disable abrt-journal-core.service
   

• To look for core dumps using the GUI, run the command:

  for RHEL6

  # abrt-gui
  

  for RHEL7 or RHEL 8

  # gnome-abrt
  

• To get a list of crashes on the command line, issue the following command:

  # abrt-cli list
  

• To generate a report of a specific crash:

  # abrt-cli report DUMPDIR
  

• Choose the RHTSupport option to reports crashes into the Red Hat Technical Support system. Alternatively, you may use the abrt-cli command to get detailed information about particular crash:

  # abrt-cli info DUMPDIR
  

• Locate the coredump file path, and provide the entire directory to the Red Hat Support representative.

  Show information on the DUMPDIR.

  # abrt-cli info DUMPDIR
  id 68fb7093b18414f689d9c7bea9b12f55ea6822ba
  reason:         rsyslogd killed by SIGBUS
  [snip]
  Directory:      /var/spool/abrt/ccpp-2023-08-09-19:25:03-22545
  

  Copy the DUMPDIR

  # tar -cJf ~/rsyslog-crash-`hostname -s`.tar.xz /var/spool/abrt/ccpp-2023-08-09-19:25:03-22545
  

For more details refer to the following documents.

• RHEL6: Deployment GuideChapter 28. Automatic Bug Reporting Tool (ABRT)

• RHEL7: System Administrator’s GuideChapter 25. Automatic Bug Reporting Tool (ABRT)

• abrtd.service collect coredump file in a container.

Note for RHEL-6.1 and earlier:

The command invocations:

# abrt-cli list
# abrt-cli report
# abrt-cli info

were:

# abrt-cli --list
# abrt-cli --report
# abrt-cli --info

In RHEL versions 6.1 and earlier. Additionally, in RHEL-6.1 and earlier, abrt refers to crashes by their UUID (obtained from the output of abrt-cli --list) rather than the dump directory.

Enable dumping for unsigned packages

• To collect core dumps from unsigned packages, (such as third party or in-house developed applications), do the following:
  1. Edit /etc/abrt/abrt-action-save-package-data.conf
  2. Change OpenGPGCheck = yes to OpenGPGCheck = no
  3. Save changes

Enable core dumps for unpackaged software

• To collect core dumps from unpackaged software (such as in-house developed applications), do the following:
  1. Edit /etc/abrt/abrt-action-save-package-data.conf
  2. Change ProcessUnpackaged = no to ProcessUnpackaged = yes
  3. Save changes