Jason Cooper: Dev/maintainer workflow security

July 29, 2015

Related Material:

  1. 2014 ksummit-discuss: Jiri Kosina: Stable workflow
  2. 2014 ksummit-discuss: Jiri Kosina: Depth of the tree of maintainers
  3. 2014 ksummit-discuss: Li Zefan: Stable Issues
  4. 2014 ksummit-discuss: Andy Lutomirski: Reviewing new API/ABI
  5. 2014 ksummit-discuss: Takashi Iwai: Metadata addendum to git commit
  6. 2014 ksummit-discuss: Takashi Iwai: Guidance for subsystem maintainers
  7. 2014 ksummit-discuss: Dan Williams: Move fast and oops things
  8. 2013 ksummit-discuss: Jiri Kosina: Stable trees and pushy maintainers
  9. 2013 ksummit-discuss: Jiri Kosina: Depth of maintainers tree
  10. 2013 ksummit-discuss: Steven Rostedt: What to do when a maintainer is no longer available
  11. 2013 ksummit-discuss: John W. Linville: Maintainer hierarchy vs. the release process
  12. 2013 ksummit-discuss: Samuel Ortiz: DT, maintainership, develoment process
  13. July 24, 2013 LWN article: What's missing from our changelogs

Additional Participants: Geert Uytterhoeven, James Bottomley, Jiri Kosina, Josh Boyer, Josh Triplett, Kees Cook, Konstantin Ryabitsev, Mark Brown, Olof Johansson, Steven Rostedt, and Theodore Ts'o.

People tagged: Dave Jones, Edward Snowden, Kees Cook,

Jason Cooper recommends a discussion on developers' and maintainers' workflow, a topic that has seen much discussion in prior Linux Kernel Summits. However, Jason would like to focus on endpoint security with an eye towards protecting the kernel source tree, to the exclusion of other popular topics such as development environment and choice of editor. Jason suggests the possibility of using the Chatham House Rule for the discussion, which allows publication of content, but not of who said what (or the release of any information that might allow someone to correctly deduce who said what). Josh Triplett suggested a broader audience, such as Linux Plumbers Conference. Josh also suspects that it would be easier to attack distros than mainline. Theodore Ts'o agreed that distro security is an important topic for a venue like Plumbers, but believes that there are a number of important kernel-specific topics appropriate for Linux Kernel Summit. Ted also suspects that if a dozen students were to attempt to submit patches containing obfuscated security bugs, at least one of them would get into Linus's tree. Kees Cook agrees that there are important topics, including:

  1. Supply-chain security, including intentionally malicious commits (which Kees suspects have already happened) and personal security.
  2. Reactive security (AKA security bug-fix workflow), including getting fixes to end users (as opposed to merely getting them to the -stable trees) and documenting impact where known.
  3. Proactive security, namely stopping security holes from appearing in the first place, including scope analysis, threat analysis, exposure analysis, static checkers, and run-time mitigation.

Kees notes that supply-chain security got quite a bit of air time after the kernel.org incident a few years ago, and reactive security gets a lot of air time every time an exploit appears, so Kees is most interested in proactive security. However, Kees also notes that discussion of proactive security is only as good as the amount of additional resources invested in actually implementing the needed changes. Kees likes the idea of a panel session to keep the discussion focused. Jason Cooper suggests that a full discussion of all of Kees's topics could easily turn into the Kernel Security Summit. Jason also notes that the long discussions of reactive security has not yet produced a solution, so additional discussion could be valuable, especially if it make progress towards resolving the conflict between openness on the one hand and effective patch propagation on the other. Jason also argues that proactive security can and should be discussed publicly, and suggests that a summary of the state of the art might be sufficient. Finally, Jason points out that one big thing that has changed since the kernel.org incident is Snowden. James Bottomley points out that lack of review is the biggest source of security incidents, including heartbleed and shellshock. Finally, James suggests asking LF's Core Infrastructure Initiative for help in the form of a security analysis.

James Bottomley also believes that anything that significantly reduces the accidental security holes will also significantly reduce the malicious holes. Jason suggests a CVE post-mortem session to evaluate our processes, and Theodore Ts'o called attention to Kees Cook's presentation entitled Kernel Security Anti-Patterns: Low Hanging Fruit, and asked Kees if he would be willing to do an update. Kees said that he would, but that he would like some dicussion about also focusing on higher-level risks in addition to the code-level anti-patterns in this discussion.

James further suspects that it is not all that easy to use stolen credentials without being detected, and calls out the risks of converging on airport theatre-style security. Jiri Kosina pointed out that if someone managed to forge a commit both on your local git repo and that of ra.kernel.org, you might not notice immediately. If the malicious commit was a small modification to an existing commit, Jiri expects that the chances of noticing would be even smaller. Konstantin Ryabitsev noted that two-factor authentication (2FA) is intended to prevent this sort of attack, but also grumbled that only about 30 people have a token and that only 25 of 450 repositories require 2FA. Although Konstantin is not ready to make use of 2FA mandatory, he is very interested in any suggestions that would increase the number of developers using it. James Bottomley asked what threat vector 2FA is intended to address. James also argued that it might in fact be giving people a false sense of security, given that an attacker can bypass it by using git rebase to inject the malicious code into an existing commit. Mark Brown replied that 2FA actually made travel easier for him, and that he believes that 2FA is a useful defense-in-depth security measure. Konstantin Ryabitsev argues that 2FA counters several attack vectors involving multiple systems that both push to and pull from a given kernel.org git repo (which Steven Rostedt believes to have mitigated by maintaining a separate copy of his git repo). He also believes that 2FA mitigates a number of likely misconfigurations involving agent forwarding, local-content-access exploits (Jiri Kosina notes that the patch-script attack might be used for this purpose), and cleartext backups of home directories. James agrees that 2FA can protect the community against careless developers, but that it does not help careful developers much. Theodore Ts'o noted that even maximally careful developers will make the occasional mistake, and that 2FA is therefore a useful defense-in-depth security measure. Geert Uytterhoeven suggests that LUKS be used to protect mackup media, and Josh Triplett offered up gpg or other tools for encrypting backups.

Kees Cook noted that there are a lot of ways to compromise a developer workstation other than stealing SSH keys, but also that the open nature of Linux-kernel does make credential theft less useful to the attacker. However, Kees remains convinced that more should be done to push security fixes all the way out to the end user, noting that it is the Linux kernel's name that gets smeared when vendors fail to keep up with security updates. Kees reiterated his belief that the core issue with proactive security is not having enough people working on this: “Dave Jones, come back, we all still love Trinity!” James Bottomley noted that security is only as good as the weakest link in the chain, so that making randomly chosen links stronger won't necessarily help much. He also asked for a proposal for pushing security fixes to end users, given that the Linux kernel community's authority ends at -stable. Finally, he noted that LF CII might help get more people working on proactive security. Kees did not have a specific proposal, and agreed that the final push to end users was vendors' responsibility, but still feels that a real-world element is missing. Kees also remained skeptical of CII's ability to really help with proactive security.

Theodore Ts'o suggests a panel discussion during the technical-session day. Steven Rostedt suggests a panel with a core-day readout, and Olof Johansson agrees, hoping that a closed session might allow discussion of points on the convenience-vs-security tradeoff spectrum weighted more towards convenience. Olof also hopes for approaches gaining more convenience without sacrificing security, perhaps documented in an LWN article. Jason Cooper agreed, noting that this is the purpose of the Chatham House Rule. Jason would also like the open session after the closed session, so that the open session can contain distilled-down content from the closed session.