Pinpointing the Python Code Paths with High Disk I/O (using OpenResty XRay)
OpenResty 1.17.8.1 RC1 is out
doujiang24 , 21 Mar 2020 (created 23 Jan 2020)We have just kicked out a new release candidate, OpenResty 1.17.8.1 RC1, for the community to test out.
Source package
https://openresty.org/download/openresty-1.17.8.1rc1.tar.gz
PGP for this source tarball:
https://openresty.org/download/openresty-1.17.8.1rc1.tar.gz.asc
Windows binary packages
Win64 version:
https://openresty.org/download/openresty-1.17.8.1rc1-win64.zip
PGP for the Win64 zip file:
https://openresty.org/download/openresty-1.17.8.1rc1-win64.zip.asc
Win32 version:
https://openresty.org/download/openresty-1.17.8.1rc1-win32.zip
PGP for the Win32 zip file:
https://openresty.org/download/openresty-1.17.8.1rc1-win32.zip.asc
Acknowledgments
Special thanks go to all our developers and contributors! Also thanks Thibault Charbonnier for his great help in preparing this release.
Version highlights
The highlights of this release candidate are:
- Based on the very recent mainline NGINX core 1.17.8.
- Support for OpenSSL 1.1.1 by upgrading the OpenSSL patches.
- Dropped support for PUC-Rio Lua; from now on, only LuaJIT 2.x is supported.
- Dropped the
dtrace
andgcc-maybe-uninitialized-warning
NGINX core patches, and applied the newinit_cycle_pool_release
patch. Additionally, all versions of the OpenSSL patches are now bundled in the final release tarball. - Reduced the final release tarball size from 4.7MB down to 3.5MB.
- ngx_lua
- Mandatorily load lua-resty-core and switched to the FFI-based API when available. Many old CFunction implementations have been removed, and lua_load_resty_core has been deprecated.
- Made the
coroutine.wrap()
API propagate errors to its parent coroutine, aligning with standard Lua behavior. This change may be breaking for some applications relying on the previous - but invalid - behavior of ngx_lua. - Fixed several compilation errors for static PCRE builds and LuaJIT FFI checks.
- ngx_stream_lua
- Ported many features from the
ngx_lua module,
including:
- Mandatorily load lua-resty-core and switched to the FFI-based API when available. Many old CFunction implementations have been removed, and lua_load_resty_core has been deprecated.
- Made the
coroutine.wrap()
API propagate errors to its parent coroutine, aligning with standard Lua behavior. This change may be breaking for some applications relying on the - invalid - previous behavior of ngx_stream_lua. - Added the
pool_size
andbacklog
options to the tcpsock:connect() API. - Enabled many FFI-based APIs for the stream subsystem including
ngx.worker.*
,ngx.var
,ngx.ctx
, and many more... - Fixed several compilation errors for static PCRE builds and LuaJIT FFI checks.
- Ported more features and bugfixes from previous releases of the ngx_lua module...
- Ported many features from the
ngx_lua module,
including:
- lua-resty-core
- Implemented the new
ngx_req.add_header()
API, similar tongx_resp.add_header()
. - New features for the
ngx.pipe
API with support for environment variables, timeouts, and more. - Updated the
ngx_ssl.get_tls1_version_str()
API to recognize TLS 1.3 connection.
- Implemented the new
- LuaJIT
- Increased the maximum number of allowed upvalues from 60 to 120.
- Initial support for x390x architectures.
- Many fixes imported from Mike Pall's upstream LuaJIT repository...
Full Changelog
- upgraded the nginx core to 1.17.8.
- see the changes here: https://nginx.org/en/CHANGES
- feature: added support for OpenSSL 1.1.1 by upgrading the OpenSSL patches. Thanks spacewander for the patch.
- feature: added a new
--with-luajit-ldflags=OPTS
option for specifying custom LuaJIT linker flags. - feature: ensured all OpenSSL patches are now included in the release tarball. Thanks Thibault Charbonnier for the patch.
- optimize: added an NGINX core patch to ensure unused listening fds are closed when
reuseport
is used. Thanks spacewander for the patch. - optimize: removed many non-source files from the release tarball to reduce its final size. Thanks Thibault Charbonnier for the patch.
- change: renamed the
ssl_pending_session
patch tossl_sess_cb_yield
for NGINX cores 1.17.1 and above. Thanks spacewander for the patch. - change: removed the
gcc-maybe-uninitialized-warning
patch which is now obsolete. Thanks Thibault Charbonnier for the patch. - change: we no longer maintain the NGINX
dtrace
patch. - bugfix: support yielding in ssl_certificate_by_lua_* when
ssl_early_data
is on. Thanks spacewander for the patch. - bugfix: ensured we apply the
init_cycle_pool_release
patch to NGINX cores >= 1.13.6 instead of 1.13.6 only. Thanks Thibault Charbonnier for the patch. - tweak: updated the
--without-luajit-gc64
option to follow Mike Pall's commit which enables GC64 by default on x64 platforms. Thanks Thibault Charbonnier for the patch. - tweak: updated the
./configure --help
usage text output for recent NGINX cores. - win32/win64: upgraded PCRE to 8.44 and OpenSSL to 1.1.1d.
- upgraded ngx_lua to 0.10.16.
change: lua-resty-core is now mandatorily loaded, and the lua_load_resty_core directive is deprecated. Thanks Thibault Charbonnier for the patch.
change: given the above change, old CFunction APIs have been retired when newer FFI implementations are available via
resty.core
. Thanks Thibault Charbonnier for the patch.change: given the above changes, we now prevent compilation with PUC-Rio Lua; only LuaJIT 2.x is supported going forward. Thanks Thibault Charbonnier for the patch.
change: removed compatibility code for unsupported NGINX core versions (< 1.6.0). Thanks Thibault Charbonnier for the patch.
change: we now ignore
lua_regex_*
directives when NGINX is compiled without PCRE support, which allows for resty-cli to work without requiring PCRE. Thanks Thibault Charbonnier for the patch.change: removed extraneous error logging when
coroutine.resume()
throws runtime errors; this change ensures ngx_lua aligns with standard Lua behavior. Thanks Thibault Charbonnier for the patch.change:
coroutine.wrap()
now propagates runtime errors to the parent coroutine; this change ensures ngx_lua aligns with standard Lua behavior. Thanks Thibault Charbonnier for the patch.feature: ngx.pipe: added support for specifying environment variables in the
ngx_pipe.spawn()
FFI API. Thanks spacewander for the patch.feature: ngx.pipe: added support for specifying timeouts in the
ngx_pipe.spawn
FFI API. Thanks spacewander for the patch.feature: ngx.pipe: added support for calling
ngx_pipe.shutdown()
on a sub-process when a light thread is waiting on it. Thanks spacewander for the patch.feature: enabled the
ngx.thread
andngx.socket.udp
APIs in ssl_session_fetch_by_lua*. Thanks Tom Thorogood for the patch.feature: implemented the FFI interface for the
ngx_req.add_header
API. Thanks spacewander for the patch.feature: implemented the FFI interfaces for
ngx.crc32_short
andngx.crc32_long
. Thanks Thibault Charbonnier for the patch.feature: when a timer fails to run, we now log the location of its function's definition. Thanks spacewander for the patch.
bugfix: ensured arguments of APIs mutating URI or request/response headers do not contain unsafe characters. Thanks Dejiang Zhu for the patch.
bugfix: skip the ssl_session_store_by_lua* and ssl_session_fetch_by_lua* handlers when using TLS 1.3. Thanks spacewander for the patch.
bugfix: config: ensured LuaJIT FFI check and static PCRE builds succeed on all platforms. Thanks Thibault Charbonnier for the patch.
bugfix: config: fallback to
--undefined
option for ld 2.25 and below. Thanks Thibault Charbonnier for the patch.bugfix: config: ensured the
pcre_version
symbol is always preserved when PCRE is statically linked. Thanks Thibault Charbonnier for the patch.bugfix: config: ensured the
pcre_version
symbol is always preserved on Darwin platforms as well. Thanks Thibault Charbonnier for the patch.bugfix: config: fixed an issue preventing compiliation with dynamic modules; we now avoid specifying
-DLUA_DEFAULT_PATH
and-DLUA_DEFAULT_CPATH
viaCFLAGS
.bugfix: ensured set_by_lua_file directives containing NGINX variables re-computes their closure's code cache key. Thanks Thibault Charbonnier for the patch.
bugfix: fixed compilation with NGINX cores < 1.11.11. Thanks Thibault Charbonnier for the patch.
bugfix: error logs now set LuaJIT's currentline to
-1
if no currentline is available. Thanks spacewander for the patch.bugfix: ngx.pipe: ensured reading while a process died returns the "closed" error. Thanks Thibault Charbonnier for the patch.
bugfix: added missing arguments to an
ngx_log_error
call inngx_http_lua_pipe.c
. Thanks Thibault Charbonnier for the patch.optimize: improved code cache lookup performance by using
luaL_ref()
to avoid invoking the costlylj_str_new
for each Lua handler execution. Thanks Thibault Charbonnier for the patch.optimize: fixed the pre-allocated number of keys in the
ngx.socket.tcp
table. Thanks Thibault Charbonnier for the patch.optimize: removed declaration of the obsolete
ngx_http_lua_inject_logby_ngx_api
function. Thanks Thibault Charbonnier for the patch.refactor: simplified Lua chunk and file cache key generation. Thanks Thibault Charbonnier for the patch.
refactor: reduced cache key size by removing unnecessary chunkname component. Thanks Thibault Charbonnier for the patch.
misc:
ngx_http_lua_util.h
: removedngx_str_set
definition as it is always defined in NGINX 1.6.0+ (the minimum supported NGINX core version). Thanks Thibault Charbonnier for the patch.misc: removed dead code, guard non-OpenResty LuaJIT definitions, and fixed styling issues. Thanks Thibault Charbonnier for the patch.
misc: fixed some warnings from the clang static code analyzer.
style: fixed a minor alignment issue in
ngx_http_lua_ssl_certby.c
. Thanks Thibault Charbonnier for the patch.style: updated
nginx_version
guard macros and assume it is always defined. Thanks Thibault Charbonnier for the patch.
- upgraded ngx_stream_lua to 0.0.8.
- feature: this module can now be compiled as a dynamic module with via the
--with-dynamic-module=PATH
option of./configure
. - bugfix: config: fixed an issue preventing compiliation with dynamic modules; we now avoid specifying
-DLUA_DEFAULT_PATH
and-DLUA_DEFAULT_CPATH
viaCFLAGS
. - Ported many features from ngx_http_lua 0.10.16. Thanks Thibault Charbonnier for the ports.
- change: lua-resty-core is now mandatorily loaded, and the lua_load_resty_core directive is deprecated.
- change: given the above change, old CFunction APIs have been retired when newer FFI implementations are available via
resty.core
. - change: given the above changes, we now prevent compilation with PUC-Rio Lua; only LuaJIT 2.x is supported going forward.
- change: removed compatibility code for unsupported NGINX core versions (< 1.6.0).
- change: removed extraneous error logging when
coroutine.resume()
throws runtime errors; this change ensures ngx_stream_lua aligns with standard Lua behavior. - change:
coroutine.wrap()
now propagates runtime errors to the parent coroutine; this change ensures ngx_stream_lua aligns with standard Lua behavior. - feature: when a timer fails to run, we now log the location of its function's definition.
- bugfix: error logs now set LuaJIT's currentline to
-1
if no currentline is available. - bugfix: fixed compilation with NGINX cores < 1.11.11.
- misc: fixed some warnings from the clang static code analyzer.
- Ported many features from ngx_http_lua 0.10.15. Thanks Thibault Charbonnier for the ports.
- feature: added the
pool_size
andbacklog
options to the tcpsock:connect() API in order to support backlog queuing in cosocket connection pools. - feature: implemented the tcpsock:receiveany() upstream cosocket API.
- feature: allowed sending boolean and nil values in cosockets.
- feature: api.h: exposed the
ngx_stream_lua_ffi_str_t
C data type for other NGINX C modules. - feature: errors are now logged when timers fail to run.
- bugfix: we now avoid tcpsock:setkeepalive() putting connections into the pool when NGINX is already shutting down.
- bugfix: inlined Lua code snippets in
nginx.conf
failed to use the Lua source checksum as part of the Lua code cache key. - bugfix: config: ensured LuaJIT FFI check and static PCRE builds succeed on all platforms.
- bugfix: config: fallback to
--undefined
option for ld 2.25 and below. - bugfix: config: ensured the
pcre_version
symbol is always preserved when PCRE is statically linked. - bugfix: config: ensured the
pcre_version
symbol is always preserved on Darwin platforms as well.
- feature: added the
- feature: this module can now be compiled as a dynamic module with via the
- upgraded lua-resty-core to 0.1.18.
- change: we now require ngx_lua v0.10.16 and ngx_stream_lua v0.0.8.
- change: updated to support the ngx_http_lua module without the now removed CFunction APIs. Thanks Thibault Charbonnier for the patch.
- feature: updated the
ngx_ssl.get_tls1_version_str()
API to recognize TLS 1.3 connections. Thanks Thibault Charbonnier for the patch. - feature: implemented the
ngx.req
module and thengx_req.add_header()
API. Thanks spacewander for the patch. - feature: added support specifying timeouts in the
ngx_pipe.spawn()
API. Thanks spacewander for the patch. - feature: added support for specifying environment variables in the
ngx_pipe.spawn()
API. Thanks spacewander for the patch. - feature: allowed for calling
ngx_pipe.shutdown()
on a sub-process when a light thread is waiting on it. Thanks spacewander for the patch. - feature: implemented
ngx.crc32_short()
andngx.crc32_long()
via FFI. Thanks Thibault Charbonnier for the patch. - feature: ensured loading the
ngx.re
module without PCRE produces a friendly error. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based API for all
ngx.worker.*
APIs in the stream subsystem. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based API for
ngx.var
in the stream subsystem. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based API for
ngx.ctx
and 'ngx.status' in the stream subsystem. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based API for
ngx.encode_base64()
andngx.decode_base64()
in the stream subsystem. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based API for
ngx.exit()
in the stream submodule. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based APIs for
ngx.escape_uri()
andngx.unescape_uri()
in the stream subsystem. Thanks Thibault Charbonnier for the patch. - feature: enabled the FFI-based APIs for
ngx.md5()
,ngx.md5_bin()
, andngx.sha1_bin()
in the stream subsystem. Thanks Thibault Charbonnier for the patch. - bugfix:
ngx.re
: fixed the error stacktrace level when missing PCRE support. Thanks Thibault Charbonnier for the patch. - bugfix: ensured
ngx.pipe
APIs with an invalidself
argument throw an error with the proper stack level. Thanks Thibault Charbonnier for the patch. - bugfix: updated
ssl.get_tls1_version_str()
to return the error"unknown version"
when the TLS version number is not recognized. Thanks Thibault Charbonnier for the patch. - optimize: load the
resty.core.misc
module last to avoid metatable lookups onngx
. Thanks Thibault Charbonnier for the patch. - optimize: made
ngx_resp.add_header()
early-exit whenheader_value
is an empty table. Thanks spacewander for the patch.
- upgraded LuaJIT to 2.1-20200102.
- feature: Increased the maximum number of allowed upvalues from 60 to 120.
- feature: Added initial s390x support.
- Move all register allocations out of the asm_href loop.
- aarch64: Fix register allocation issue for XLOAD.
- aarch64: Use the xzr register whenever possible.
- aarch64: Allocate LJ_TISNUM early.
- aarch64: Fixed crash with side traces under register pressure.
- ARM: Fix up condition codes for conditional arithmetic insn.
- ARM64: Added support for FNMADD and FNMSUB.
- Fixed
os.date
for timezone change awareness. jit.prngstate
: Return a sane value (0) forLUAJIT_DISABLE_JIT
.- thread.exdata: Build
recff_thread_exdata
only forLJ_HASFFI
. - Removed redundant
emit_check_ofs
. - doc: mentioned the increase in the maximum number of upvalues in the miscellaneous section.
- doc: reworded the description of the table.isempty API.
- imported Mike Pall's latest changes:
- Properly fix pointer diff in string.find().
- x64: Enable
LJ_GC64
mode by default. - FFI: Eliminate hardcoded string hashes.
- Fix interaction between profiler hooks and finalizers.
- Don't use STRREF for pointer diff in string.find().
- Fix
tonumber("-0")
. - Fix hash table chaining (again).
- Fix declarations of _BitScanForward/_BitScanReverse.
- Add stricter check for
print()
vs.tostring()
shortcut. - Prevent integer overflow while parsing long strings.
- Fix stack check when recording BC_VARG.
- FFI: Add missing write barrier on C library index update.
- FFI: Workaround for platform dlerror() returning NULL.
- OSX: Use
__thread
attribute. - OSX: Don't set a default
MACOSX_DEPLOYMENT_TARGET
. - Build MinGW import library, too.
- Fix MinGW make clean.
- Update Android and iOS build docs.
- Add note about the unsuitabilty of
math.random()
for crypto. - Update MSVC build script and docs.
- More recent MSVC is partially C99 compliant.
- Fix narrowing of conversions to U32.
- doc: readme.md: improve completeness and readability.
- Fix unsinking of 64 bit constants.
- Fix bytecode dump unpatching.
- Fix
debug.getinfo()
argument check. - MIPS: Fix delay slot hint.
- Fix TNEW load forwarding with instable types.
- ARM: Fix GCC 7
-Wimplicit-fallthrough
warnings. - ARM: Fix condition code check fusion.
- ARM64: Avoid side-effects of constant rematerialization.
- upgraded lua-resty-lrucache to 0.10.
- feature: implemented the
cache:get_keys()
API. Thanks Datong Sun for the patch. - feature: implemented the
cache:count()
andcache:capacity()
APIs. Thanks Datong Sun for the patch. - feature: implemented a user
flags
attribute similar to that of thengx.shared
API. Thanks Thibault Charbonnier for the patch. - makefile: added a
lint
target to detect invalid test cases which we now use in Travis CI. Thanks Thibault Charbonnier for the patch. - doc: cleaned up outdated resources and grammatical improvements.
- doc: fixed a minor typo in the
cache:flush_all()
description. - doc: updated the
cache:set
usage section to reflect thettl
argument being optional.
- feature: implemented the
- upgraded lua-resty-string to 0.12.
- optimize: removed a duplicate string length lookup in
to_hex()
. Thanks Robert Paprocki for the patch.
- optimize: removed a duplicate string length lookup in
- upgraded lua-resty-redis to 0.28.
- feature: added new options
ssl
andssl_verify
to thered:connect()
API for connecting to Redis over TLS. Thanks Vinayak Hulawale for the patch. - feature: implemented the
red:set_timeouts()
API. Thanks zouyi for the patch.
- feature: added new options
- upgraded lua-resty-shell to 0.03.
- optimize: removed an unused function argument. Thanks Dejiang Zhu for the patch.
- doc: clarified return values and behavior upon reaching a timeout threshold. Thanks Dejiang Zhu for the patch.
- upgraded resty-cli to 0.25.
- upgraded ngx_echo to 0.62.
- bugfix: config: we now always use fouble quotes in
[...]
conditionals. - bugfix: config: avoided an error with NGINX 1.17.0 and above. Thanks Thibault Charbonnier for the patch.
- style: fixed a coding style issue found by
ngx-releng
.
- bugfix: config: we now always use fouble quotes in
- upgraded ngx_srcache to 0.32.
- bugfix: config: we should always use double quotes in
[...]
conditionals. - bugfix: config: avoided an error with NGINX 1.17.0 and above. Thanks Thibault Charbonnier for the patch.
- style: fixed minor coding style issues found by
ngx-releng
. - doc: documented a tip to make memcached store objects bigger than 1MB.
- bugfix: config: we should always use double quotes in
Feedback
Feedback on this new release candidate is most welcome. Feel free to create new GitHub issues or contact us via one of our mailing lists.