Pinpointing the Python Code Paths with High Disk I/O (using OpenResty XRay)
ChangeLog for 1.15.8.x
Yichun Zhang (agentzh) , 21 Mar 2020 (created 14 Nov 2018)Version 1.15.8.3 - 20 March 2020
- Applied the safe_map_uri_to_path patch to the 1.15.8 version of the nginx core.
- Applied the init_cycle_pool_release patch to the 1.15.8 version of the nginx core.
- ngx_lua
- bugfix: ensured arguments of APIs mutating URI or request/response headers do not contain unsafe characters. Thanks Dejiang Zhu for the patch.
See the HackerOne report for more details about the security vulnerabilities.
Version 1.15.8.2 - 8 September 2019
- bugfix: applied the nginx core patch for new HTTP/2 security advisories (CVE-2019-9511 CVE-2019-9513 CVE-2019-9516).
- win32/win64: upgraded PCRE to 8.43 and OpenSSL to 1.1.0k.
Version 1.15.8.1 - 16 May 2019
- upgraded the nginx core to 1.15.8.
- see the changes here: http://nginx.org/en/CHANGES
- change: we now enable the GC64 mode by default in our bundled LuaJIT build for x86_64 architectures; this can be disabled using
--without-luajit-gc64
. Thanks Thibault Charbonnier for the patch. - feature: bundled the lua-tablepool library.
- feature: bundled the lua-resty-signal library. Thanks OpenResty Inc. for supporting this work.
- feature: bundled the lua-resty-shell library. Thanks OpenResty Inc. for supporting this work.
- feature: ./configure: added new options
--without-stream_ssl_module
and--without-stream
. - feature: ./configure: added support for the
-h
option. - feature: ./configure: support
@rpath
placeholder in OS X. Thanks Datong Sun for the patch. - feature: updated the
socket_cloexec
patches to support the ngx.pipe API. Thanks OpenResty Inc. for supporting this work and spacewander for the development of the patch. - win32: upgraded OpenSSL to 1.1.0j.
- bugfix: nginx did not destroy the cycle memory pool before the daemon process exits. Thanks Yuansheng Wang for the patch.
- bugfix: win32/win64: the error log buffer size was merely 2048 bytes, and is now updated to 4096 bytes.
- resty-doc indexes: skipped nginx's njs docs due to the use of special xml tags.
- upgraded ngx_lua to 0.10.15.
- feature: added support for ARM64 (Aarch64). Thanks Cloudflare for sponsoring this work. Thanks Dejiang Zhu and Zexuan Luo for the development of the patch.
- feature: implemented the lua_load_resty_core directive which loads
resty.core
by default. Thanks Thibault Charbonnier for the patch. - feature: added the
pool_size
andbacklog
options to the tcpsock:connect() API in order to support backlog queuing in cosocket connection pools. Thanks OpenResty Inc. for supporting this work and spacewander for the development of the patch. - feature: implemented a new lua_sa_restart directive, which sets the
SA_RESTART
flag on nginx workers signal dispositions. Thanks Thibault Charbonnier for the patch. - feature: implemented the tcpsock:receiveany() upstream cosocket API. Thanks spacewander for the patch.
- feature: added support for the nginx builtin Link header, which allows for specifying multiple values for this header. Thanks tokers and Thibault Charbonnier for the patch.
- feature: errors are now logged when timers fail to run. Thanks spacewander for the patch.
- feature: added an FFI API to support the ngx.pipe API provided by lua-resty-core. Thanks OpenResty Inc. for supporting this work and spacewander for the development of the patch.
- feature: added an FFI API for retrieving
env
directives values from init_by_lua*. Thanks Thibault Charbonnier for the patch. - change: we no longer support the standard Lua 5.1 interpreter (PUC-Rio Lua). It is now strongly recommended to use the OpenResty LuaJIT releases, bundled with the official OpenResty releases.
- change: we now avoid running init_by_lua* in signaller processes and when testing the nginx configuration. Thanks spacewander for the patch.
- change: we now print an alert when a non openresty-specific version of LuaJIT is detected since many optimizations would be missing. Thanks Thibault Charbonnier for the patch.
- change: we now print a warning when writing to the Lua global variable
_G
. - change: we now avoid generating the Content-Type response header when getting all response headers. Thanks spacewander for the patch.
- bugfix: fixed a segfault in nginx core >= 1.15.0 when init_worker_by_lua* is used. Thanks Datong Sun for the patch.
- bugfix: ngx.process: type() didn't return 'master' in master process. Thanks spacewander for the patch.
- bugfix: tcpsock:setkeepalive(): worker processes might take too long to gracefully shutdown when the keep alive connections take a long max idle time. Thanks Dejiang Zhu for the patch.
- bugfix: setting the request "Host" header should clear any existing cached
$host
variable so that subsequent reads return the expected result. - bugfix: inlined Lua code snippets in
nginx.conf
failed to use the Lua source checksum as part of the Lua code cache key. - bugfix: silenced
-Wcast-function-type
warnings, allowing for compilation with gcc8. Thanks Alessandro Ghedini for the patch. - doc: noted that ngx.req.get_method() can be used in the body_filter_by_lua* and log_by_lua* phases. Thanks tokers for the patch.
- doc: updated the docs to reflect the change that we now no longer support the standard Lua 5.1 interpreter in this module. also recommended OpenResty's LuaJIT branch version instead of the stock LuaJIT.
- doc: mention that ngx.req.set_body_data() and ngx.req.set_body_file() must read the request body. Thanks Thibault Charbonnier for the patch.
- doc: fixed the links to ssl_session_store_by_lua*. Thanks chronolaw for the patch.
- typo: fixed a debug log in access and rewrite handlers. Thanks sbhr for the patch.
- upgraded ngx_stream_lua to 0.0.7.
- feature: added support for ARM64. Thanks Cloudflare for sponsoring this work. Thanks Dejiang Zhu and Zexuan Luo for the development of the patch.
- feature: implemented the lua_load_resty_core directive which loads
resty.core
by default. Thanks Thibault Charbonnier for the patch. - feature: implemented the preread reqsock:peek() API. Thanks Kong Inc. for sponsoring this work. Thanks Datong Sun for the development of the patch.
- feature: enabled ssl_certificate_by_lua* support. Thanks Kong Inc. for sponsoring the development of this feature, and thanks James Callahan and Datong Sun for the development of the patch.
- feature: implemented a new lua_sa_restart directive, which sets the
SA_RESTART
flag on nginx workers signal dispositions. Thanks Thibault Charbonnier for the patch. - feature: enabled resty.core.shdict support. Thanks Datong Sun for the patch.
- feature: enabled ngx.semaphore support. Thanks Datong Sun for the patch.
- feature: enabled ngx.errlog support. Thanks Thibault Charbonnier for the patch.
- feature: added an FFI API for retrieving
env
directives values from init_by_lua*. Thanks Thibault Charbonnier for the patch. - change: we now avoid running init_by_lua* in signaller processes and when testing the nginx configuration. Thanks Thibault Charbonnier for the patch.
- change: we now print an alert when a non openresty-specific version of LuaJIT is detected since many optimizations would be missing. Thanks Thibault Charbonnier for the patch.
- misc: re-rendered all files to include newly added template header. Thanks Datong Sun for the patch.
- doc: added missing links for ssl_certificate_by_lua* directives. Thanks chronolaw for the patch.
- upgraded lua-resty-core to 0.1.17.
- feature: added support for ARM64 (AArch64). Thanks Cloudflare for sponsoring this work. Thanks Dejiang Zhu and Zexuan Luo for the development of the patch.
- feature: implemented the ngx.pipe API which allows spawning processes and communicating with them in a non-blocking way. Thanks OpenResty Inc. for supporting this work and spacewander for the development of the patch.
- feature: ssl_certificate_by_lua* support for the stream subsystem. Thanks Kong Inc. for sponsoring the development of this feature, and thanks James Callahan and Datong Sun for the development of the patch.
- feature: resty.core.shdict: enabled the FFI-based API for the stream subsystem. Thanks Datong Sun for the patch.
- feature: ngx.semaphore: enabled the FFI-based API for the stream subsystem. Thanks Datong Sun for the patch.
- feature: ngx.errlog: enabled the FFI-based API for the stream subsystem. Thanks Thibault Charbonnier for the patch.
- feature:
os.getenv
is now able to retrieveenv
directives values from within init_by_lua*. Thanks Thibault Charbonnier for the patch. - change: we now require OpenResty's LuaJIT branch to work properly on ARM64 since we make use of the thread.exdata Lua API.
- change: increased stack level on subsystem violation error to expose the offending module. Thanks Datong Sun for the patch.
- change: updated calls to
error()
to be thrown from the user's stack level. Thanks Thibault Charbonnier for the patch. - feature: implemented the
ndk.*
API with FFI. - bugfix: ensured
resty.core
can be loaded in nginx < 1.11.7. Thanks Thibault Charbonnier for the patch. - bugfix: ensured
resty.core
can be loaded in binaries without PCRE support. Thanks Thibault Charbonnier for the patch. - bugfix: regex: disabled PCRE JIT compilation and regex caching in init_by_lua* when running on macOS. _Thanks Datong Sun for the patch.
- bugfix: ngx.process: type() didn't return 'master' in master process. Thanks spacewander for the patch.
- bugfix: now we only allow a single version of ngx_lua and ngx_stream_lua. Thanks spacewander for the patch.
- optimize: fixed misuses of Lua global variables in our Lua code (caught by the new version of the lj-releng tool).
- optimize: removed redundant requires in
resty/core.lua
. Thanks chronolaw for the patch. - doc: ngx.process: fixed a typo in the spelling of 'privileged agent'. Thanks spacewander for the patch.
- doc: ngx.process: updated the process type list. Thanks spacewander for the patch.
- doc: ngx.balancer: fixed a typo in the sample code. Thanks chronolaw for the patch.
- doc: ngx.ssl: documented the
get_tls1_version_str()
API function. - doc: ngx.semaphore: switch status to production ready.
- upgraded LuaJIT to 2.1-20190507: https://github.com/openresty/luajit2/tags
- feature: implemented a new table.isempty() API. Thanks OpenResty Inc. for supporting this work.
- feature: implemented a new table.isarray() API. Thanks OpenResty Inc. for supporting this work.
- feature: implemented a new table.nkeys() API. Thanks OpenResty Inc. for supporting this work.
- feature: implemented the new Lua and C API functions for thread.exdata. This API is used internally by the OpenResty and we advise not to use it yourself in the context of OpenResty. Thanks Zexuan Luo for preparing the final version of the patch.
- feature: luajit -bl: dump the constant tables (KGC and KN) for each Lua proto object as well.
- feature: luajit.h: defined the macro
OPENRESTY_LUAJIT
for our branch of LuaJIT. - change: run the LJ_64 branch in asm_hrefk when
LUAJIT_USE_VALGRIND
andLUAJIT_ENABLE_GC64
are both supplied. Thanks spacewander for the patch. - bugfix:
ffi.C.FUNC()
: it lacked a write barrier which might lead to use-after-free issues and memory corruptions. - bugfix: LuaJIT's
jit.v
module might lead to segfaults due to buggy Lua stack traversal code. - bugfix: 16-bit MCLink.mapofs and GCtrace.nsnapmap fields would overflow for large Lua programs, leading to segfaults; enlarged them to 32-bit.
- bugfix: fixed a segfault when unsinking 64-bit pointers. Thanks Luke Gorrie and Thibault Charbonnier for the patch.
- bugfix: guarded the
jit_prngstate()
builtin with theLJ_HAS_JIT
macro. Thanks abhay1722 for the patch. - bugfix: fixed assertion failure "lj_record.c:92: rec_check_slots: Assertion `nslots <= 250' failed" found by stressing our edgelang compiler.
- doc: documented what we change in our OpenResty branch of LuaJIT.
- imported Mike Pall's latest changes:
- Improve luaL_addlstring().
- Fix os.date() for wider libc strftime() compatibility.
- Fix MinGW build.
- DynASM/MIPS: Fix shadowed variable.
- DynASM/PPC: Fix shadowed variable.
- Fix overflow of snapshot map offset.
- Better detection of MinGW build.
- Actually implement maxirconst trace limit.
- MIPS/MIPS64: Fix TSETR barrier (again).
- Fix memory probing allocator to check for valid end address, too.
- DynASM/x86: Fix vroundps/vroundpd encoding.
- DynASM: Fix warning.
- ARM64: Fix exit stub patching.
- ARM64: Fix write barrier in BC_USETS.
- Windows: Add UWP support, part 1.
- From Lua 5.3: assert() accepts any type of error object.
- x86: Disassemble FMA3 instructions.
- DynASM/x86: Add FMA3 instructions.
- PPC/NetBSD: Fix endianess check.
- x86/x64: Check for jcc when using xor r,r in emit_loadi().
- FFI: Make FP to U64 conversions match JIT backend behavior.
- Bump copyright date to 2018.
- FFI: Add tonumber() specialization for failed conversions.
- Give expected results for negative non-base-10 numbers in tonumber().
- upgraded lua-resty-lrucache to 0.09.
- optimize: fixed misuses of Lua global variables in our Lua code (caught by the new version of the lj-releng tool).
- upgraded lua-resty-lock to 0.08.
- bugfix: we now enforce the use of lua-resty-core's resty.core.shdict module to avoid dead locking with the CFunction-based shdict API in ngx_lua.
- doc: made it clearer that we depend on lua-resty-core.
- doc: documented the limitations of certain ngx_lua execution contexts which prohibit yielding.
- upgraded lua-resty-limit-traffic to 0.06.
- bugfix: set_conn() method did not work at all due to a copy&paste error. Thanks pearzl for the patch.
- doc: added the resty.limit.count module to the library's README. Thanks Thibault Charbonnier for the patch.
- upgraded resty-cli to 0.24.
- feature: implemented new
--stap
and--stap-opts
command-line options to trace the underlying nginx C process directly (without going through the Perl wrapper layer). - feature: implemented a new
--main-conf
command-line option for specifying nginx's main {} context directives directly from the command line. Thanks Datong Sun for the patch. - feature: implemented a new
--gdb-opts
command-line option which implies--gdb
. The--valgrind-opts
option also implies--valgrind
. - bugfix: md2pod: infinite looping might happen when an empty
.md
file is given. - bugfix: nginx-xml2pod: ensured
<link>
tags contents are parsed. Thanks Thibault Charbonnier for the patch.
- feature: implemented new
- upgraded lua-resty-upstream-healthcheck to 0.06.
- bugfix: avoided shadowing a variable which would prevent the checking threads from being waited upon. Thanks Thijs Schreijer and Thibault Charbonnier for the report and fix.