Analyzing the Most CPU-Consuming Requests in OpenResty or Nginx
You should always check out Nginx's error log file (specified
error_log directive in
nginx.conf) for any errors or warnings.
If you prefer redirecting common Lua errors to the HTTP response body during Lua development, you can put a Lua pcall call on the top level of your Lua code to catch and redirect any Lua exceptions. But keep in mind, not all errors can be captured this way because you could have errors when sending out the response, then it is impossible to put such errors into the response body.
You can find additional tools in the stap++ project:
These tools can not only debug functional problems, but also profile online servers to find performance bottlenecks.
3rd-party Lua libraries or 3rd-party NGINX C modules that are not maintained by OpenResty might be subject to memory issues since they might not go through the same careful testing of the standard OpenResty components. In very rare use cases, standard OpenResty components may incur memory issues as well. To debug such memory issues which lead to either segmentation faults or weird nondeterminism in software behaviors due to memory corruptions, we can use Valgrind to reliably find the real culprit when the issues can be relatively easy to reproduce.
To maximize the effectiveness of Valgrind's memcheck tool, we should at least
disable the memory pools in the NGINX core and also enforce LuaJIT to use the
system memory allocator (by default, LuaJIT uses its own memory allocator).
Furthermore, we should also enable the internal assertions in various
OpenResty core components like
ngx_lua and LuaJIT.
To simplify the special build settings for Valgrind usage, OpenResty
openresty-valgrind pre-built Linux packages for various
common Linux distributions:
If your Linux system is not supported in the pre-built package repositories yet, you can build your own openresty-valgrind by following the steps in the openresty-valgrind's RPM spec file.
Among other things, remember to configure the following lines in your
nginx.conf's top level scope to ensure nginx runs as a single non-daemon
process when being run by valgrind:
daemon off; master_process off; worker_processes 1;
It is strongly discouraged to use Valgrind on non-Linux systems like Mac OS X or macOS since Valgrind's support in those systems are very limited and even buggy.
Test::Nginx test scaffold has provide builtin support
for running tests with Valgrind, which is referred to as the "Valgrind test mode".
You can find detailed documentation with examples in the "Programming OpenResty" book
One limitation with Valgrind When your OpenResty application requires nontrivial load of real traffic to reproduce or you have to enable special nginx child processes like worker processes and cache manager processes to reproduce the issue, then you can consider rebuilding your OpenResty with clang or gcc's AddressSanitizer (ASAN) tool, which also gives much better performance.
For example, you can build OpenResty with ASAN this way:
export ASAN_OPTIONS=detect_leaks=0 ./configure --with-cc="gcc -fsanitize=address" \ --with-cc-opt="-O1 -fno-omit-frame-pointer" --with-debug -j9 \ --prefix=/usr/local/openresty-asan --with-luajit-xcflags="-DLUAJIT_USE_VALGRIND" \ --with-no-pool-patch make -j9 sudo make install
Make sure you also set the environment
ASAN_OPTIONS=detect_leaks=0 when you run
this openresty (installed under
/usr/local/openresty-asan/) otherwise you would
see a lot of false positives on memory leaks.
We will provide the
openresty-asan package in our official OpenResty Linux package
repositories in the near future to make this easier.
But ASAN comes with a limitation too: it cannot check memory issues happened in just-in-time (JIT) compiled code (like the machine code generated by PCRE JIT and LuaJIT's JIT compiler) or any hand-written assembly code as in OpenSSL or LuaJIT's interpreter.