Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

<del> FROM centos:5 </del> <ins> # We use Debian 6 (glibc 2.11, kernel 2.6.32) as a common base for other # distros that still need Rust support: RHEL 6 (glibc 2.12, kernel 2.6.32) and # SLES 11 SP4 (glibc 2.11, kernel 3.0). FROM debian:6 </ins> WORKDIR /build <del> # Centos 5 is EOL and is no longer available from the usual mirrors, so switch # to http://vault.centos.org/ RUN sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf RUN sed -i 's/mirrorlist/#mirrorlist/' /etc/yum.repos.d/*.repo RUN sed -i 's|#(baseurl.*)mirror.centos.org/centos/$releasever|1vault.centos.org/5.11|' /etc/yum.repos.d/*.repo </del> <ins> # Debian 6 is EOL and no longer available from the usual mirrors, # so we'll need to switch to http://archive.debian.org/ RUN sed -i '/updates/d' /etc/apt/sources.list &&  sed -i 's/httpredir/archive/' /etc/apt/sources.list </ins> <del> RUN yum upgrade -y && yum install -y  curl  </del> <ins> RUN apt-get update &&  apt-get install --allow-unauthenticated -y --no-install-recommends  automake  </ins> bzip2  <ins> ca-certificates  curl  file  g++  g++-multilib  </ins> gcc  <del> gcc-c++  </del> <ins> gcc-multilib  git  lib32z1-dev  libedit-dev  libncurses-dev  </ins> make  <del> glibc-devel  </del> <ins> patch  </ins> perl  <del> zlib-devel  file  xz  which  pkgconfig  </del> <ins> pkg-config  unzip  </ins> wget  <del> autoconf  gettext </del> <ins> xz-utils  zlib1g-dev </ins> ENV PATH=/rustroot/bin:$PATH <del> ENV LD_LIBRARY_PATH=/rustroot/lib64:/rustroot/lib </del> <ins> ENV LD_LIBRARY_PATH=/rustroot/lib64:/rustroot/lib32:/rustroot/lib </ins> ENV PKG_CONFIG_PATH=/rustroot/lib/pkgconfig WORKDIR /tmp <ins> RUN mkdir /home/user </ins> COPY host-x86_64/dist-x86_64-linux/shared.sh /tmp/ # We need a build of openssl which supports SNI to download artifacts from


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

COPY host-x86_64/dist-x86_64-linux/build-openssl.sh /tmp/ RUN ./build-openssl.sh <del> # The `curl` binary on CentOS doesn't support SNI which is needed for fetching </del> <ins> # The `curl` binary on Debian 6 doesn't support SNI which is needed for fetching </ins> # some https urls we have, so install a new version of libcurl + curl which is # using the openssl we just built previously. # # Note that we also disable a bunch of optional features of curl that we don't # really need. COPY host-x86_64/dist-x86_64-linux/build-curl.sh /tmp/ <del> RUN ./build-curl.sh </del> <ins> RUN ./build-curl.sh && apt-get remove -y curl </ins> # binutils < 2.22 has a bug where the 32-bit executables it generates # immediately segfault in Rust, so we need to install our own binutils.


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

COPY host-x86_64/dist-x86_64-linux/build-binutils.sh /tmp/ RUN ./build-binutils.sh <del> # libssh2 (a dependency of Cargo) requires cmake 2.8.11 or higher but CentOS # only has 2.6.4, so build our own COPY host-x86_64/dist-x86_64-linux/build-cmake.sh /tmp/ RUN ./build-cmake.sh # Build a version of gcc capable of building LLVM 6 </del> <ins> # Need at least GCC 5.1 to compile LLVM nowadays </ins> COPY host-x86_64/dist-x86_64-linux/build-gcc.sh /tmp/ <del> RUN ./build-gcc.sh </del> <ins> RUN ./build-gcc.sh && apt-get remove -y gcc g++ </ins> <del> # CentOS 5.5 has Python 2.4 by default, but LLVM needs 2.7+ </del> <ins> # Debian 6 has Python 2.6 by default, but LLVM needs 2.7+ </ins> COPY host-x86_64/dist-x86_64-linux/build-python.sh /tmp/ RUN ./build-python.sh <del> # Now build LLVM+Clang 7, afterwards configuring further compilations to use the </del> <ins> # LLVM needs cmake 3.4.3 or higher, and is planning to raise to 3.13.4. COPY host-x86_64/dist-x86_64-linux/build-cmake.sh /tmp/ RUN ./build-cmake.sh # Now build LLVM+Clang, afterwards configuring further compilations to use the </ins> # clang/clang++ compilers. <del> COPY host-x86_64/dist-x86_64-linux/build-clang.sh host-x86_64/dist-x86_64-linux/llvm-project-centos.patch /tmp/ </del> <ins> COPY host-x86_64/dist-x86_64-linux/build-clang.sh /tmp/ </ins> RUN ./build-clang.sh ENV CC=clang CXX=clang++ <del> # Apparently CentOS 5.5 desn't have `git` in yum, but we're gonna need it for # cloning, so download and build it here. COPY host-x86_64/dist-x86_64-linux/build-git.sh /tmp/ RUN ./build-git.sh # for sanitizers, we need kernel headers files newer than the ones CentOS ships # with so we install newer ones here COPY host-x86_64/dist-x86_64-linux/build-headers.sh /tmp/ RUN ./build-headers.sh # OpenSSL requires a more recent version of perl # with so we install newer ones here COPY host-x86_64/dist-x86_64-linux/build-perl.sh /tmp/ RUN ./build-perl.sh </del> COPY scripts/sccache.sh /scripts/ RUN sh /scripts/sccache.sh


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

curl -L https://github.com/llvm/llvm-project/archive/$LLVM.tar.gz |  tar xzf - --strip-components=1 <del> yum install -y patch patch -Np1 < ../llvm-project-centos.patch </del> mkdir clang-build cd clang-build


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

set -ex source shared.sh <del> curl https://cmake.org/files/v3.6/cmake-3.6.3.tar.gz | tar xzf - </del> <ins> CMAKE=3.13.4 curl -L https://github.com/Kitware/CMake/releases/download/v$CMAKE/cmake-$CMAKE.tar.gz | tar xzf - </ins> mkdir cmake-build cd cmake-build <del> hide_output ../cmake-3.6.3/configure --prefix=/rustroot </del> <ins> hide_output ../cmake-$CMAKE/configure --prefix=/rustroot </ins> hide_output make -j10 hide_output make install cd .. rm -rf cmake-build <del> rm -rf cmake-3.6.3 </del> <ins> rm -rf cmake-$CMAKE </ins>


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

cd .. rm -rf curl-build rm -rf curl-$VERSION <del> yum erase -y curl </del>


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

cd .. rm -rf gcc-build rm -rf gcc-$GCC <del> yum erase -y gcc gcc-c++ binutils </del>


Question:
The docs currently claim that the minimum Linux kernel version for is 2.6.18 (released September 20th, 2006). This is because RHEL 5 used that kernel version. However, RHEL 5 entered ELS on March 31, 2017. Should we continue to support RHEL 5 for , or should we increase the minimum Linux Kernel version to 2.6.27 (2nd LTS) or 2.6.32 (RHEL 6, 3rd LTS)? . Even bumping the min-version to 2.6.27 would allow us to remove most of the Linux-specific hacks in . Example: .
Given that RHEL is the only reason we keep comically old kernel versions around, I would propose that Rust only support RHEL until Maintenance Support ends. This is (essentially) what we already did for RHEL 4. Rust never supported RHEL 4, and back when RHEL 5 still had maintenance support. It would be nice to get someone from Red Hat or a RHEL customer to comment. This policy would allow us to increment the minimum kernel from 2.6.18 to 2.6.32 (and remove a lot of Linux-specific hacks). Note that RHEL has a weird (and very long) support system for RHEL 4, 5, and 6 (sources: and ). It has 5 major steps: Full Support Normal support of the OS Maintenance Support 1 Bug/Security fixes Limited hardware refresh Maintenance Support 2 Bug/Security fixes The end of this phase is considered "Product retirement" Extended Life Cycle Support (ELS) Additional paid product from Red Hat Gives updates for a longer period of time No additional releases/images Extended Life Phase (ELP) No more updates Limited Technical Support End date not given by Red Hat Current status of RHEL versions: RHEL 4 Not supported by Rust Currently in ELP (no end date specified) ELS ended March 31, 2017 RHEL 5 Supported by Rust Currently in ELS Maintenance Support ended March 31, 2017 ELS ends November 30, 2020 RHEL 6 Supported by Rust Currently in Maintenance Support 2 Maintenance Support ends November 30, 2020
cc which I think they could be related.
It also may be worth to drop support of Windows XP and Vista (especially considering that panics are broken on XP since June 2016, see: ). Previously it was discussed . cc
Besides , which other things would we be able to clean up?
Good question! There are 6 workarounds in for older Linux versions (that I could find). Increasing the minimum version to 2.6.32 (aka 3rd Kernel LTS, aka RHEL 6) would fix 5 of them. Code links are inline: (mentioned above, in 2.6.23) ( in 2.6.24, there is also the mention of a bug occuring on "some linux kernel at some point") to atomically set the flag on the pipe fds ( in 2.6.27) ( in 2.6.27) to permit use of ( in 2.6.28) ( in 4.5, not fixed by this proposal) As you can see, the workarounds fixed by this proposal all have a similar flavor.
I am Red Hat's maintainer for Rust on RHEL -- thanks for the cc. I try to keep an eye on new issues, but this one slipped past me. Red Hat only ships the Rust toolchain to customers for RHEL 7 and RHEL 8. If our customers would like to use Rust on older RHEL, they can do so via , and we'll support them in the same way we would for any other third-party software. Internally we do also build and use on RHEL 6, mostly because it's needed to ship Firefox updates. This is where it gets a little hairy, because each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel. I would have to apply local workarounds if upstream Rust loses the ability to run on 2.6.18 kernels. We prefer to keep fixes upstream as much as possible, both for open sharing and to avoid bitrot. Is there much of a benefit to removing the current workarounds? I agree all of that code is annoying and cumbersome, but it's already written, and as far as I know it hasn't been a maintenance burden. Do you see otherwise? If there are any known issues with such Linux compatibility code, I am definitely willing to take assignment for fixing them.
(Not a Rust maintainer, but I'm a Rust user and maintain a lot of other OSS software so I have well-developed feelings around supporting Very Old Software :-)) Do you have any sense of on what timeline Rust would be able to drop support for 2.6.18 kernels without causing you pain? In general I don't think people mind supporting configurations that have users and are painful to work around, but needing to support them for forever is a bitter pill to swallow! Particularly as they get harder and harder to test over time (already I have no idea how to test on really old kernels besides building it myself). So if there was an estimate "we'd like to be able to support this until Q2 2020", even if it's not set in stone, I think that would be very helpful!
The other benefit would be that Rust wouldn't have to use CentOS 5 for CI, which means we don't have to patch LLVM (and Emscripten LLVM) to compile on those systems. Of course that's also fairly limited in scope.
As soon as we stop shipping Firefox and Thunderbird updates on RHEL 6, I won't need Rust there anymore. AFAIK this does correspond to the end of Maintenance Support, November 30, 2020. Then I'll be on to RHEL 7 builders as my minimum, probably still with some RHEL 6 2.6.32 kernels involved. It should be fine for me if we update CI to CentOS 6. This is mostly concerned with how the release binaries are linked to GLIBC symbol versions, which is all in userspace. It's a broader community question whether any other Rust users still care about running on RHEL or CentOS 5. (Small caveat - glibc support for a symbol can still return , .)
I noticed that Red Hat also has Extended Life Cycle Support (ELS) for RHEL 6 until June 30, 2024. Will you need RHEL 5 to work during this time? I don't know how ELS works with Firefox updates. Also, is there any reason RHEL 4 support wasn't an issue prior to March 31, 2017 (while RHEL 5 was still normally supported)? This issue came up for me when dealing with opening files in , see for more info. No single RHEL 5 issue is that bad, it's mainly just the sum of a bunch of tiny issues.
I'm not on that team, but AFAIK we don't ship Firefox updates for ELS. The last build I see for RHEL 5 was . Maybe there could be an exception for a severe security issue, but I really doubt we'd rebase to newer Firefox for that, which means Rust requirements won't change. New Firefox ESR versions do require a newer Rust toolchain too, which is generally why we have to keep up. Otherwise we could just freeze some older compat rustc while upstream moves on. Rust wasn't required until Firefox 52.
that makes perfect sense to me, thanks for clarifying. So the proposed policy would be: Support RHEL until RHEL is retired (i.e. ends normal support). This would mean: Supporting RHEL 5 (v2.6.18) until November 30, 2020 Supporting RHEL 6 (v2.6.32) until June 30, 2024 Supporting RHEL 7 (v3.10) until May, 2029 This is a much longer support tail than any other Linux distros (that I know of), so it would also be the effective minimum kernel version of , , and their dependencies. How does that sound to people? EDIT: The alternative to this would be Support RHEL until it is retired, taking the table above and incrementing the RHEL versions by 1. It would also mean being able to drop RHEL 5 support now.
Sure, that's ideal for me. :smile: As mentioned, is just needed for kernel support, as far as I'm concerned. Userspace concerns like GLIBC symbol versions can stick to currently supported for my needs.
Red Hat should backport the features we need (CLOEXEC and getrandom, in particular) to whatever kernels that it wants Rust to support. I don't know any good reason why they don't do so, other than it's cheaper to convince the whole world to keep supporting older kernel versions than it is to do the backports. We should change that dynamic.
The alternative to this would be Support RHEL N until it is retired I think we should not officially support retired OS versions (well, maybe with some grace period), so I prefer this option. Supporting 14 year old kernels seems a bit too much to me. each RHEL N is bootstrapped on RHEL N-1 and often kept that way -- meaning a lot of our RHEL 6 builders are still running on a RHEL 5 kernel Is it possible to apply kernel patches to those builders to add support for functionality like CLOEXEC? Or build Firefox separately on RHEL 6?
I think it won't be fruitful for us to debate the merits of stable enterprise kernels, but no, this is not a possibility. An alternative take is "Red Hat should be responsible for the work in Rust to maintain the support they want" -- here I am, ready and willing. I definitely can't change those kernels. The ones that are stuck on N-1 are precisely to avoid rocking the boat, and backporting features is a big change. Some of our arches do eventually update the builders to their matching N kernel, but some don't, and I don't know all the reasons. If the workarounds are removed, then I will have to reapply them in our own builds. This assumes I keep using only our own binaries -- if I ever have to re-bootstrap from upstream binaries for stage0, I would also have to use some interception (like ) to hack in the workarounds. This is all possible, but much worse than the status quo of maintaining support here.
I think this sounds reasonable. We aren't going to be able to update these kernels, it's just a question of when we would drop support for these OSes. Especially given that we wont need RHEL 5 CI support, I think that leaving things as they are is fine. I opened this issue to verify two things: 1) That we weren't intending on supporting RHEL 5 "forever", and had a clear date when to drop support. 2) That someone from Red Hat actively cared about supporting these older kernels. It looks like both these things are true. We should remove RHEL 5 workarounds after November 30, 2020. This issue can be postponed until then. EDIT: For our use case in , we were able to add compatibility for RHEL 5 .
See for a recent case where supporting such ancient kernels required extra manual work ( to the 2 lines mentioned by above). I am not sure how often that comes up, but probably the hardest part is to even notice that it happens -- and even then those codepaths will likely linger untested.
Yes, but this will still be true in general. If the CI kernel is newer than whatever kernel baseline we choose, it will be possible for newer syscalls to pass undetected. I don't know if we have any control over that -- can we choose a particular VM image? So that leaves us with a stated policy of support. The more that developers are aware of such issues, the better, but I'm the one most likely to notice when my build actually fails. If I come back with a fix, I at least need the project to be receptive. I haven't been in the habit of building nightly on our RHEL6 builders, but maybe I should! Catching this early is better for everyone involved...
That seems reasonable. How quick is the Rust related bootstrapping? Could it be run automatically (or more frequently)? This is an interesting point. If I hadn't made that change, would have kept working on RHEL 5, it just would have leaked an FD. I'm not sure how we would even test for this sort of thing. This isn't a RHEL 5 specific concern, I just don't know how Rust tests these types of resource leak issues in general.
About 2 hours. Automation is harder, since I believe our build system needs my credentials, but I'll look into what is possible here. Yeah, that's a lot more subtle than an !

ulimit -c unlimited fi <ins> # There was a bad interaction between "old" 32-bit binaries on current 64-bit # kernels with selinux enabled, where ASLR mmap would sometimes choose a low # address and then block it for being below `vm.mmap_min_addr` -> `EACCES`. # This is probably a kernel bug, but setting `ulimit -Hs` works around it. # See also `dist-i686-linux` where this setting is enabled. if [ "$SET_HARD_RLIMIT_STACK" = "1" ]; then rlimit_stack=$(ulimit -Ss) if [ "$rlimit_stack" != "" ]; then ulimit -Hs "$rlimit_stack" fi fi </ins> ci_dir=`cd $(dirname $0) && pwd` source "$ci_dir/shared.sh"