Crash Dump Analysis: HsErr Files, Core Dumps, SIGSEGV

JVM crashes are uncommon but devastating when they happen. When the JVM exits with a crash signal (SIGSEGV, SIGABRT, etc.), it writes an HsErr file (HotSpot Error File) containing the crash context, register state, stack traces, and memory addresses. Learning to read these files is essential for diagnosing native crashes that Java-level tooling cannot help with.

This guide covers what happens during a JVM crash, how to read the HsErr file, how to extract useful data from core dumps, and the common causes of SIGSEGV in Java applications.

Introduction

JVM crashes are uncommon but devastating when they occur. Unlike Java-level exceptions that your application can catch and handle, a crash signal like SIGSEGV terminates the JVM process itself, taking down your service with it. The HotSpot Error File (HsErr) that the JVM writes on crash contains the diagnostic context to understand what happened — register state, stack traces, loaded libraries, and crash location — but only if you know how to read it. Learning to interpret these files transforms a mysterious crash into a diagnosable incident.

HsErr files tell you the crash signal, the exact function that failed, and the stack frame that led there. Core dumps go further and capture the entire process memory, enabling post-mortem analysis when HsErr is not enough. Both tools are essential for production environments where crashes happen at scale and you cannot reproduce them locally. Native crashes involving JNI, JVM JIT bugs, and stack overflows are the categories most commonly diagnosed through these files.

This guide walks through what’s in an HsErr file, how to extract actionable data from core dumps, and the common SIGSEGV patterns in Java applications. You will learn what each section of the crash file means, how to correlate crash time with deployments and GC activity, and the production configurations that ensure crash diagnostics are captured rather than lost to /tmp.

What Happens When the JVM Crashes

When the JVM crashes, the OS sends a signal to the JVM process. The JVM’s signal handler catches it, writes diagnostic information to the HsErr file, and terminates. The HsErr file is created in the working directory (or /tmp if the working directory is not writable).

# Example HsErr file location
# The JVM prints this on stderr when crashing:
#
# An error report file with more information is saved as:
# /var/log/myapp/hs_err_pid12345.log

The HsErr File Structure

An HsErr file has several sections. Here is what each contains:

#
# A fatal error has been detected by the Java Runtime Environment.
#
# SIGSEGV (0xb) at pc=0x00007f8a3c2d4567, pid=12345, tid=0x00007f8a4c5a8700
#
# JRE version: OpenJDK Runtime Environment (build 17.0.2+8)
# Java VM: OpenJDK 64-Bit Server VM (17.0.2+8 mixed mode, sharing)
# Problematic frame:
# C  [libc.so.6+0x18a567]  __strlen_sse2+0x177
#
---------------  S Y S T E M  ---------------
...
---------------  T H R E A D  ---------------
...
---------------  P R O C E S S  ---------------
...

Key Sections

Section	What It Contains
Header	Signal type, program counter (PC), PID, TID
Problematic frame	The native function that crashed (C, jni, compiled code)
SYSTEM	OS version, memory info, CPU info
THREAD	Register state, stack trace of the crashing thread
PROCESS	All threads, memory map, loaded libraries
STACK TRACE	Java and native frames for each thread
HEAP	Summary of heap memory at crash time
Registers	CPU register values at crash point

Common Signal Causes

SIGSEGV (Signal 11 - Segmentation Fault)

The JVM or native code accessed memory it should not have. Common causes in Java:

• JNI bugs: Passing a bad reference to native code
• JVM bugs: Rare, but present in some versions
• Native library bugs: Bugs in native libraries like netty, lz4, etc.
• Stack overflow: Native stack exhausted (different from Java stack overflow)

SIGABRT (Signal 6)

The JVM called abort(). Typically from:

• Assert failures in debug JVM builds
• fatal error handling (assert(), vm_assert())
• Internal JVM errors that lead to abort()

SIGFPE (Signal 8 - Floating Point Exception)

Arithmetic error: division by zero, integer overflow, etc. Rare in Java (the JVM handles these in bytecode), but possible in JNI.

Reading the Crash Thread Stack

The most important section is the thread that caused the crash:

Current thread (0x00007f8a54003800):
  JavaThread "main" [_thread_in_native, id=12345, stack(0x00007f8a5c000000,0x00007f8a5c400000)]

Stack: [0x00007f8a5c000000,0x00007f8a5c400000],
  growable_gap=65536

Native frames: (J=compiled Java code, j=interpreted, v=VMcode, C=native code)
J  org.apache.commons.compress.utils.Charsets.toIso(jj)[B
J  com.example.MyApp.process(N)V @ 0x00007f8a3e8a1234
j  com.example.MyApp.nativeProcess(Ljava/lang/String;)V+0
j  com.example.MyApp.main([Ljava/lang/String;)V+0

C  [libexample.so+0x1234]  Java_com_example_MyApp_nativeProcess+0xabcd
C  [libc.so.6+0x18a567]  __strlen_sse2+0x177   <-- HERE IS THE CRASH

The <-- HERE IS THE CRASH annotation (when present) marks the actual crash point. In this example, a JNI function called strlen with a bad pointer.

Production Failure Scenarios

Scenario 1: JNI Passing Bad Reference

Symptom: JVM crashes in native code during a JNI call.

Investigation: The problematic frame shows a libc function like memcpy, strlen, or strchr. The preceding native frame shows your JNI code.

Problematic frame:
C  [libc.so.6+0x18a567]  __strlen_sse2+0x177

JNI frame:
j  com.example.MyApp.nativeMethod(Ljava/lang/String;)V+0

Root Cause: A jstring passed to native code was already freed or never valid. This happens when a Java String escapes its scope and is garbage collected while the native code still holds a reference.

Fix: Always use GetStringUTFChars and ReleaseStringUTFChars properly, and ensure you do not store Java object references in native structures that outlive the JNI call.

Scenario 2: JVM Bug in JIT Compiled Code

Symptom: Crash in VtblStub or StubQueue with no obvious JNI involvement.

Investigation: The problematic frame shows JVM internal functions, often in the JIT compiler or generated code stubs.

Problematic frame:
v  Stub::RethrowError+0xc7
j  com.example.Worker.process()V+0

Fix: Check your JVM version for known bugs. Upgrade the JVM. If the crash is reproducible, create a minimal test case and report it to the JVM vendor. As a temporary workaround, disable the problematic optimization with -XX:CompileCommand=exclude.

Scenario 3: Stack Overflow in Native Code

Symptom: Crash with stack overflow in the error text, but your Java stack was not deep.

Investigation: The thread stack section shows the native stack:

Native frames: (J=compiled Java code, j=interpreted, v=VMcode, C=native code)
C  [libjvm.so+0x1234567]  JVM_handle_linux_signal+0x567
C  [libc.so.6+0x159234]   __restore_rt
...
C  [recursive_native_func+0x10]  <-- recursion in native code

Root Cause: Some JNI library or JVM-dependent code recursed beyond the native stack limit. Common with deeply recursive native algorithms or certain JNI callbacks.

Fix: Increase native stack size with -Xss256k (or higher) for the affected thread, or fix the recursion bug in the native library.

Analyzing Core Dumps

A core dump is a snapshot of the entire process memory at the time of the crash. HsErr files are much smaller and focused, but core dumps contain everything — useful for post-mortem analysis when the HsErr is not enough.

Taking a Core Dump

# On Linux, the JVM generates a core dump on crash automatically
# To take a manual core dump:
gcore <pid>

# Or using systemd
coredumpctl dump <pid> > core.dump

# Or using kernel
sysctl kernel.core_pattern=/tmp/core.%e.%p

Reading a Core Dump with GDB

# Load the core dump with GDB
gdb /path/to/java /path/to/core.dump

# Load the JVM library for symbol resolution
symbol-file /usr/lib/jvm/java-17-openjdk/libjvm.so

# Show the crashing thread's stack
thread apply all bt

# Show registers at crash
info registers

# Examine memory at an address
x/100x 0x00007f8a54003800

Implementation: Automatic Crash Analysis Script

#!/bin/bash
# analyze_crash.sh - parse a JVM HsErr file

ERRFILE=$1

echo "=== CRASH SUMMARY ==="
grep "SIGSEGV\|SIGABRT\|SIGFPE" "$ERRFILE" | head -3

echo ""
echo "=== PROBLEMATIC FRAME ==="
grep -A2 "Problematic frame" "$ERRFILE"

echo ""
echo "=== CRASHING THREAD STACK ==="
awk '/Current thread/,/^$/' "$ERRFILE" | grep -E "^[CJjv]" | head -20

echo ""
echo "=== JAVA STACK TRACES ==="
grep -E "^Stack:|^  J |^  j |^  v " "$ERRFILE" | head -40

echo ""
echo "=== LIBRARY LOAD MAP ==="
grep "lib" "$ERRFILE" | grep -v "Compile" | grep -v "Class" | head -20

Trade-off Table

Aspect	HsErr File	Core Dump	JFR + Log Files
Size	KB	GB	MB
Contains	Summary, stack traces	Full memory snapshot	Events over time
Generation overhead	Crash only	Crash + write pause	Continuous low overhead
Analysis speed	Fast	Slow	Fast
Post-incident analysis	Limited	Full	Event-based
Production impact	None (crash dump only)	Disk I/O spike	Low

Observability Checklist

• Configure HsErr file location with -XX:ErrorFile=/path/hs_err_%p.log
• Set -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/ to get heap dumps on OOM
• Ensure the working directory is writable (or HsErr goes to /tmp)
• Enable core dump generation with ulimit -c unlimited
• Set kernel.core_pattern to include PID to avoid overwriting
• Keep JDK debug symbols installed (openjdk-*-debug package)
• Archive HsErr files alongside incident reports
• Correlate crash time with deployments, traffic spikes, and GC activity
• Build a searchable index of HsErr files across your fleet
• Set up alerting for JVM crash events (not just OOM)

Security Notes

HsErr files and core dumps contain sensitive information:

• Full heap contents: All Java objects, strings, potentially PII and credentials
• Native memory: May contain data from native libraries, including crypto keys
• Thread stacks: Full call stacks including business logic, API calls, SQL
• Classloader contents: Reveals application structure and third-party libraries

Hardening:

• Store crash dumps in access-controlled storage with encryption at rest
• Set kernel.core_pattern to a non-world-readable location
• Do not include crash dumps in bug reports sent externally without sanitization
• Use jmap sanitization features to redact sensitive heap data before sharing
• Clean HsErr files after investigation is complete

Common Pitfalls / Anti-Patterns

Pitfall 1: HsErr File Goes to /tmp

Problem: Working directory is not writable, so the HsErr file is created in /tmp with a random name and deleted on reboot.

Solution: Explicitly set -XX:ErrorFile=/var/log/myapp/hs_err_%p.log and ensure the directory exists and is writable.

Pitfall 2: Missing Debug Symbols

Problem: The crash stack shows only addresses like 0x00007f8a3c2d4567 with no function names.

Solution: Install the debug symbols package for your JDK: apt-get install openjdk-17-dbg or yum install java-17-openjdk-debuginfo. Keep JDK debug symbols in your environment for production troubleshooting.

Pitfall 3: Mixing JVM Architectures

Problem: Crash in libjvm.so on an ARM JVM running on x64 machine or vice versa.

Solution: Always verify architecture matches. uname -m and java -version should agree. Cross-platform JVM binaries can appear to work but crash mysteriously.

Pitfall 4: Not Saving the Core Dump

Problem: The HsErr is generated but the JVM was not configured to produce a core dump, so you have no memory data for deeper analysis.

Solution: Ensure ulimit -c unlimited before starting the JVM, or use -XX:+CreateCoredumpOnCrash (Linux only, defaults to on unless explicitly disabled).

Quick Recap Checklist

• JVM crashes generate HsErr files with stack traces and context
• Set -XX:ErrorFile to control where HsErr files are written
• SIGSEGV in native code is the most common JVM crash type
• The “problematic frame” shows where the crash occurred
• JNI bugs are a frequent cause of SIGSEGV in Java applications
• Install JDK debug symbols for readable stack traces
• Use gcore to manually capture a core dump for post-mortem analysis
• HsErr + core dump together give you the full picture
• Crash dumps contain sensitive data — control access and encrypt at rest
• Correlate crash events with deployments to identify root causes

Interview Questions

Further Reading

• OpenJDK HsErr Parser - HotSpot interpreter source for understanding HsErr context
• GDB Documentation for JVM debugging - Using GDB with process core dumps
• Linux Core Dump Handling - How Linux manages core dumps
• JEP 439: Generational ZGC - How ZGC affects crash behavior vs other collectors
• Oracle JDK Crash Analysis Guide - Official JVM crash troubleshooting guide

Conclusion

JVM crashes generate HsErr files containing stack traces, register state, and memory context. SIGSEGV in native code is the most common crash type, often from JNI bugs. Set -XX:ErrorFile to control where HsErr files are written, and ensure your working directory is writable. For deeper analysis, configure core dump generation with ulimit -c unlimited. Install JDK debug symbols so stack traces resolve to function names rather than raw addresses.