Profiler Servlet
Integrated, on-demand profiling via the Async Profiler project.
Background
HBASE-21926 introduced a new servlet that supports integrated, on-demand profiling via the Async Profiler project.
Prerequisites
Go to the Async Profiler Home Page, download a release appropriate for your platform, and install on every cluster host. If running a Linux kernel v4.6 or later, be sure to set proc variables as per the Basic Usage section. Not doing so will result in flame graphs that contain no content.
Set ASYNC_PROFILER_HOME in the environment (put it in hbase-env.sh) to the root directory of the
async-profiler install location, or pass it on the HBase daemon's command line as a system property
as -Dasync.profiler.home=/path/to/async-profiler.
Usage
Once the prerequisites are satisfied, access to async-profiler is available by way of the HBase UI or direct interaction with the infoserver.
Examples:
- To collect 30 second CPU profile of current process (returns FlameGraph svg)
curl http://localhost:16030/prof - To collect 1 minute CPU profile of current process and output in tree format (html)
curl http://localhost:16030/prof?output=tree&duration=60 - To collect 30 second heap allocation profile of current process (returns FlameGraph svg)
curl http://localhost:16030/prof?event=alloc - To collect lock contention profile of current process (returns FlameGraph svg)
curl http://localhost:16030/prof?event=lock
The following event types are supported by async-profiler. Use the 'event' parameter to specify. Default is 'cpu'. Not all operating systems will support all types.
Perf events:
- cpu
- page-faults
- context-switches
- cycles
- instructions
- cache-references
- cache-misses
- branches
- branch-misses
- bus-cycles
- L1-dcache-load-misses
- LLC-load-misses
- dTLB-load-misses
Java events:
- alloc
- lock
The following output formats are supported. Use the 'output' parameter to specify. Default is 'flamegraph'.
Output formats:
- summary: A dump of basic profiling statistics.
- traces: Call traces.
- flat: Flat profile (top N hot methods).
- collapsed: Collapsed call traces in the format used by FlameGraph script. This is a collection of call stacks, where each line is a semicolon separated list of frames followed by a counter.
- svg: FlameGraph in SVG format.
- tree: Call tree in HTML format.
- jfr: Call traces in Java Flight Recorder format.
The 'duration' parameter specifies how long to collect trace data before generating output, specified in seconds. The default is 10 seconds.
UI
In the UI, there is a new entry 'Profiler' in the top menu that will run the default action, which is to profile the CPU usage of the local process for thirty seconds and then produce FlameGraph SVG output.
Notes
The query parameter pid can be used to specify the process id of a specific process to be profiled. If this parameter is missing the local process in which the infoserver is embedded will be profiled. Profile targets that are not JVMs might work but is not specifically supported. There are security implications. Access to the infoserver should be appropriately restricted.