You probably tell your users to update their software frequently, and that’s for a good and simple reason: Software updates fix vulnerabilities and make systems more secure. Not only that, they make the entire digital ecosystem more secure. A network is stronger when the links are strong. And conversely, a strong network can be hacked if there’s a weak link. For example, Target’s well-protected credit card network was compromised through weaknesses in its HVAC vendor’s computer system.

Notwithstanding the security risks, government agents may see malicious software updates as a means for surveillance. The term malicious software, or malware, refers to software code designed to be harmful to the software user, often by breaking down protective security measures and giving access or control to unintended third parties. The U.S. or other governments may seek to force you to create and install such malicious software. For example, an investigator could demand that you change your software to bypass passcode lockouts, enable wiretapping, turn on cameras, or physically track someone.

This risk is not theoretical. You may have read about Apple’s refusal to comply with a government order to create and push code to the San Bernardino shooter’s iPhone. In that case, the FBI asked Apple to develop software that would bypass an Apple security feature that erases the phone’s data after 10 incorrect attempts to enter the passcode. The government ultimately dropped its case when the FBI found a third-party hacker that had developed software that could break into the suspect’s iPhone. Apple has since claimed to make substantial security improvements in subsequent iOS updates, including to address the password-attempts loophole.

The likelihood that government actors may attempt to force software makers to push out software updates that include malware designed to obtain data from targeted devices grows as more companies secure their users’ data with encryption. And, as companies close other technological loopholes, there will be increased pressure on law enforcement to find alternate vulnerabilities to exploit. For example, Apple recently closed a technological loophole that law enforcement routinely relied upon to extract iPhone data. As other backdoors in secured devices are closed, the software update system will be an increasingly appealing option to law enforcement.

The risk of these practices raises serious privacy concerns for individual users. And malicious software updates also raise substantial cybersecurity concerns that could have serious effects on all of us, whether we use a particular product or not. Fixing vulnerabilities around the internet requires that the public trust the software update channel so that fixes to security weaknesses are deployed as soon as they’re made available. But people will not regularly update software if they fear the government or bad actors will use the new code to exploit their systems.  

Governments may ask you for help, or they might seek to compel assistance. You have the right to say no to requests that are not backed up by a court order. By obtaining a court order demanding technical assistance, however, the government might attempt to compel you to install malware on a user’s machine as a software update that appears to be entirely ordinary, and that comes directly from you.

Law enforcement routinely seeks the assistance of software makers to obtain data in the course of criminal investigations. If you do not already hold the relevant data in a legible format, law enforcement might turn to “technical-assistance orders” that purport to require you to take novel and extraordinary steps to obtain unencrypted information about your users. These steps could include disclosing encryption keys, turning on microphones or cameras in consumer goods, circumventing password lock-out mechanisms, and more. One such novel form of technical assistance that might be attractive to law enforcement is the subversion of a software update mechanism. Such a technical assistance order might even include a secrecy clause purporting to prohibit you from telling anyone about the order.

You have the right to say no to requests that are not backed up by a court order. But by obtaining a court order demanding technical assistance, the government might try to compel you to install malware on a user’s machine as a software update that appears to be entirely ordinary, and that comes directly from you. You have a right to challenge these orders in court.

It is uncertain whether governments have already sought to obtain such orders or will do so in the future, but you should not wait to receive one before thinking through how to respond. We've compiled a guide with recommendations on how software developers and distributors like you can make informed decisions about protecting the integrity of your software update channels, both legally and technically. These pages seek to inform you on how to plan for and respond to such potential orders so that you are in a good position to resist, narrow, or defeat them — and protect your customers’ privacy and the security of the internet at large.

Here’s what you can do:

1. Understand the issue.
2. Implement privacy-minded designs and policies.
3. Plan responses to potential technical assistance orders ahead of time, rather than upon receipt.
4. Lawyer up!

---

 

If law enforcement were to ask you push a malicious software update to one or more of your users, it would likely do so by serving you with a court order for technical assistance similar to this one. 

However, there are no clear legal obligations requiring you to do something as dangerous as subverting your software update channels to push malware, which can endanger your end users as well as other developers who may rely on your product. 

Technical assistance orders may purport to be based on various legal provisions. If the underlying surveillance is based on a search warrant, the technical assistance order will likely be issued pursuant to the All Writs Act, a law that gives courts the authority to issue orders necessary to enforce other lawful orders, such as search warrants. Other sources of legal authority may be the Foreign Intelligence Surveillance Act, the Wiretap Act, or the Pen Register statute. Regardless of the legal authority for the technical assistance demand, the issues you will likely consider when deciding whether and how to comply will often be similar.

The pages on this website are not legal advice, but they are meant to help you think through how you might plan for and respond to a technical assistance order. If you receive a court order for technical assistance, you should talk to an attorney about your particular situation. There is a lot of uncertainty around the application of these laws, so your lawyer is in the best position to help you weigh your rights and options in deciding whether and how to comply with or challenge the order. You may consult your attorney even if the order comes with a non-disclosure provision.

Possible Scenarios

To help you plan, imagine situations in which you might receive a technical assistance order demanding subversion of a software update. Below are three potential scenarios that may arise under the All Writs Act:

1. Clever, a free web browser, receives a technical assistance order from federal law enforcement, which orders Clever to write an automatic update targeted at one user of the Clever browser. The investigators would like Clever to develop and incorporate malware into a software update sent to that user to enable them to monitor the user’s online activity to collect evidence of a crime. 
2. Arthur is a release manager at Penguin, a free operating system popular for its security features and analytics tools. Arthur has received a technical assistance order from the FBI, and an accompanying gag order says he is bound to secrecy. The FBI has demanded that Arthur use Penguin’s automatic update network to push code to an individual it suspects of computer hacking. The code Arthur is told to push includes an FBI tool to log keystrokes; record web browsing history, usernames, and passwords; and list all the internet-facing ports open on the user’s computer. Arthur has reservations about complying with the order but is afraid to talk to anyone about it because of the accompanying gag.
3. TheGraham is a social media app that is widely used by people to share pictures of their food. TheGraham users are a dedicated bunch and post photos with great frequency throughout the day in an effort to share and discover their next best meal. Usually, the location where the photo was taken is shared along with the image. Law enforcement has sent a technical assistance order to TheGraham ordering that they send a non-targeted update to all users containing a patch that would enable law enforcement to trigger at-will location tracking for specific users. This request is not pursuant to any specific investigation; law enforcement has stated it will only use this ability pursuant to a warrant in the future.

---

Implement privacy-minded designs and policies.

As someone who develops or supplies software, you and your organization have an obligation to consider the risk that your software update channel presents to your users. As a responsible software distributor, you should take steps to defend your users against misuse of this channel. Similarly, users should prefer software distributors like you—ones that offer these defenses.

The First Line of Defense: Cryptographic Signatures

The most basic protection that every software distributor should offer is strong cryptographic signatures over the updates. As a software distributor, you should have a secret key, and your users should know about the corresponding public key. They protect their systems during the update process by verifying that the proposed update is signed by your public key. The security depends on your retaining exclusive control over the corresponding secret key so that malicious updates will not verify.

However, your secret key can be compromised (i.e. used to sign malicious updates) by legal compulsion, technical attack, or economic coercion. So more accountability is needed for a trustworthy update system than signatures alone.

Defense Against Targeted Attack

One possible attack on a user via software updates is the targeted distribution of malware. That is, an attacker who gains control of a distributor’s secret key can sign a malicious update and ship it only to the targeted user. Other users will continue to see the typical, non-malicious update channel.

Mirrorable Distribution

To bolster user defense against targeted attack, you should adopt “mirrorable distribution.” This means enabling users to pass information about available updates between one another, or allowing any number of intermediaries to copy and re-distribute (“mirror”) that information. If the attacker cannot predict where the user will pull their update from and cannot identify the user when they do an update, it is significantly harder to target a specific user. Letting a user fetch software updates anonymously from a range of sources concretely demonstrates that it is likely to be difficult to subject them to targeted attack.

In contrast, it is problematic to cryptographically sign software updates that are specific to a particular user or group of users. Targeted software updates facilitate targeted malware. For example, Apple’s iOS will not update unless it verifies a signature over the specific identifier for an individual phone or other device. This is dangerous both because it facilitates and incentivizes targeted malware and because it means Apple can “fingerprint” devices. You should ensure there is no tradeoff between end-user privacy and secure software updates.

Binary Transparency

An additional form of defense against targeted attack is an accountability mechanism known as binary transparency.

A mechanism protected by binary transparency will refuse to install an update unless the software has been verifiably logged in a global, irrevocable, auditable log. Website certificates — the mathematical objects behind the lock symbol in your web browser’s address bar — are already logged in a similar way. There are many ways to do this technically, but the tools are still relatively new and not widely tested. Mozilla is leading the way on this, but more work is needed, including tools for auditors to review binary transparency logs to highlight unusual behavior.

Auditability and Verifiability

If you take software update safety seriously, you will want to hold yourself accountable to your users. Accountability starts with auditability and verifiability. Even when individual users do not do their own audits or verification, the fact that others can do so provides a level of "herd immunity” — flawed updates can be detected and corrected even if your company is prevented from doing so.

Free/Libre and Open Source Software, also known as “FLOSS,” has the benefit of being readily auditable. FLOSS is a contract between the software distributor and the software user — it says that the human-readable source code that corresponds to the software update will be available to the user for inspection and improvement. With inspectable source code, it becomes much harder to hide malware inside software updates and patches. And with modifiable source, it becomes easier to fix problems that may have been introduced by the initial supplier.  While some vendors will continue to sell proprietary software, visible and independently fixable code brings important security advantages.

You can also offer additional reassurance against malware with reproducible builds (a.k.a. deterministic compilation). Reproducible builds focus on how a developer translates software from code written by software engineers (source code) to machine code that computers can read and run (binaries).  If the build is reproducible, the set of tools that do that translation (compiler, assembler, etc., sometimes referred to collectively as the “toolchain”) can not be modified to inject malware into those binaries. A compromised toolchain is not a theoretical attack.

If an uncompromised toolchain acting on the same source code produces a binary completely identical, down to the last bit, to all the other binary versions compiled by the vendor, then we can be confident that the vendor’s toolchain was not compromised. If not, it suggests that something may be, but is not necessarily, wrong. Toolchains sometimes produce minor differences between binaries because the compiled version includes trivial varying elements such as timestamps, tags indicating the name of the compiling computer, or more esoteric harmless changes. But serious security flaws can also be tiny, which makes them difficult to distinguish from inconsequential variation. With a reproducible toolchain, anyone who compiles the source code themselves can verify that it exactly matches a binary offered by a vendor (which is what the vast majority of software users are going to download).

Reproducible builds provide the most meaningful protection for users of Free/Libre and Open Source Software, because FLOSS vendors have already committed to having meaningful, visible source code that other people can review, modify and, double-check for themselves. But even if you are a vendor of completely proprietary software, you can take advantage of reproducible builds internally, as a way of ensuring that an order can’t be placed on one part of the organization (e.g., the build/release team) without other parts (e.g., the team that verifies reproducibility) knowing about it.  If you have contractual agreements to provide copies of source code to auditors (even if you supply prorietary, non-FLOSS software), you can make reproducibility a part of their audits. And if you make a public commitment to reproducible builds, you will let your users know that a compromised toolchain can’t be used as an excuse for shipping malware.

Among other things, reproducible builds can protect individual software developers from extortion or personal attack because any malware slipped into a build through compromise of their build systems will be discovered as soon as someone else compiles the source code independently and discovers the mismatch. If you employ software developers, you can simultaneously reduce your organization’s attack surface, and reduce potential threats to your employees with this improvement.

Build reproducibility has in recent years become much more of a focus within the developer community, and non-reproducibility is increasingly considered a bug. Thanks to years of effort, the majority of widely used free and open source software packages, including the Apache webserver, the Linux kernel, and the Tor Browser are now reproducible, and work is ongoing across the ecosystem to establish this as a new norm in software development.

Providing Responsible Software Updates

Your users depend on you to provide them with updates that are concrete improvements and are not designed to harm them. While all software developers will make mistakes, you can provide your users with concrete defenses against potential malware and put in place systems that provide deterrence against attempts to use your update channels for stealthy injection of malware.

As a software user, if your preferred software distributor doesn’t apply the mechanisms and principles described above, consider encouraging them to adopt these approaches, or consider other software distributors that can offer you comparable functionality with better protections.

The technical protections above should of course be combined with legal and policy guidelines that make it clear that the software update channels should be inviolable as a critical piece of public infrastructure. But even if we get legal and policy constraints that keep some governments from trying to abuse these channels, less principled governments and non-governmental actors (organized crime, industrial espionage, etc.) may attempt to ship malware by these channels, so the technical guards described here are a necessary part of any healthy software ecosystem.

---

Plan for potential technical assistance orders ahead of time, rather than upon receipt.

Internal discussion among your executives, software designers and engineers, and lawyers is a helpful and important step to take prior to receiving a technical assistance order. If you are able to think through technical, legal, and business resources available to you in responding to such an order, you will be in a much better position to protect your software and your users should you receive one. You will also be more aware of how thoughtful, privacy-minded technical design can help protect your users by incorporating transparency and accountability in your software update channel.

Now that you understand the potential risks from government efforts to subvert your software update channels to distribute malware for surveillance purposes, here are some questions to consider to mitigate those risks.

Software Update Policies Generally

The policies and technology of your software update mechanism should be designed to guard against the risk of government ordered malware installation. Additionally, you may want to tell your users that you are taking steps to protect the software updates they depend on from compromise, including potential technical assistance orders. These honest representations will encourage your users to install updates, and may be relevant to whether you can raise First Amendment arguments against such orders later on.

1. Consider implementing technical measures to create more transparency in your software update channel, such as reproducible builds or binary transparency. See our review of technical defenses.
2. Consider how your software update channel operates. Should more than one person need to sign off on a software update before it can be sent out? Who should be in the loop?
3. What is your public commitment on the security of your software update channel? Does the internal process you have in place to address the possible receipt of a technical assistance order concerning your software update channel reflect that public commitment? Have you taken any steps to publicly communicate that when you cryptographically sign a software update or otherwise issue new software, you are confirming that it does not contain any malware, government-requested or otherwise? Have you articulated this position on your website or in the terms and conditions of your software?

Before You Receive a Technical Assistance Order

Make a plan for how to respond if you receive a technical assistance order using the following questions as a roadmap.

 

1. If you were to receive a technical assistance order, who would be the critical people on your team to inform?
2. Know that you may consult a lawyer even if the technical assistance order includes a nondisclosure order (also known as a “gag order”)—but depending on what such an order says, not all members of your team will be entitled to see, discuss, or respond to it at the time, and your lawyer will also be bound by the order. Do you have an employee who is designated to receive law enforcement requests? Do you have a general counsel, or other outside legal counsel already retained to handle law enforcement matters? Have you discussed with your counsel if they are familiar with various types of law enforcement requests, and whether they have ideas about potential ways to respond or resist such requests? If you do not presently have counsel, who could you reach out to in order to prepare for the event you received a technical assistance order?
3. Why might you resist a technical assistance order in a given case? How would complying with such an order affect the security of your software update channel? How would complying with such an order affect your users’ trust in your software? How would complying with such an order affect your resources, your project, or your business in the future should you have to comply with subsequent orders?
4. If you decide you would likely resist a technical assistance order that you believe is unlawful or dangerous, do you have the resources—financial and human—to resist such an order? Why or why not? If you feel you do not presently have the capacity to resist an order, what steps could you take to better position you to do so?

When You Receive a Technical Assistance Order

Contact a lawyer—and/or let the ACLU know.

---

If you receive a technical assistance order, you should first and foremost seek legal counsel. Your lawyer will be in the best position to help you understand the scope of the order, your legal obligations under the order, and whether you may want to challenge the order and in which ways.

While you should plan for receiving technical assistance orders with your regular legal counsel, if you receive one, please reach out to us at the ACLU for additional information and potential legal representation.

Orders for surveillance and technical assistance may be accompanied by a non-disclosure or protective order (sample), referred to colloquially as a gag order. Through a gag order, law enforcement will either request or require, or a court will order, that you not disclose even the fact that you received a demand for information or technical assistance to anyone, including the target(s) of the investigation.

Even if you receive a technical assistance order with such a gag order, you can consult a lawyer about the order’s contents.

Your lawyer can help you identify your legal obligations under any secrecy clauses. Gag orders may also purport to limit who you may discuss the matter with. These limits may cause serious problems with your ability to appropriately respond to the demand. The lawyer can help you work through these limitations and negotiate the scope of the gag order with the government.

You and your lawyer should also verify that the technical assistance order you have received is signed by a judge and that the scope of the court’s order aligns with what law enforcement is asking of you. You have no legal obligation to comply with requests for assistance that are not signed by a judge (though gag orders need not necessarily come from a court).

If you are considering complying with a technical assistance order, there are several best practices that you can incorporate to imbue the process with more transparency, accountability, and oversight.

First, as with ordinary law enforcement demands, you may be able to work with the agency to narrow the scope of any request. Agencies often use overbroad or boilerplate orders to facilitate their numerous investigations. In consultation with your legal counsel, you can thoughtfully review the request you have received to determine whether the request has been properly issued, whether you have the data sought, and whether the full scope of information requested is likely to be relevant to the government’s investigation. Before complying with the order, you can ask the agency if the full scope of the information requested is necessary for the investigation.

Second, you can go to court to challenge a request or court order for technical assistance. Whether or not to challenge a technical assistance order is a complicated business and legal question. In making this decision, you and your attorney may want to consider the legal risks, the financial and technical costs, the reputational impact of complying (or not), and the possibility of receiving similar orders in the future. 

In considering whether you wish to challenge the order in court, you might consider the following legal arguments:

• The order may be preempted by the Communications Assistance Law Enforcement Act (CALEA), which generally exempts providers of wire or electronic communication services, manufacturers of telecommunications equipment, and providers of telecommunications support services from having to design their products or services in specified ways, including to create the capability to wiretap or surveil consumers. Telecommunications carriers are not required to be able to decrypt encryption provided by another entity.
  • For an example of this argument, see pages 4-8 of Apple’s response to a 2015 request to unlock an iPhone.
     
• Requests must pertain to a specific investigation. The order is overbroad if it requests assistance prospectively. CALEA has not been interpreted to authorize the government to mandate changes to an organization’s program, software, or product that would generally enable surveillance or wiretapping by creating a backdoor.
  • For a more in-depth discussion of this argument, see this article, which discusses prior attempts by the FBI to pressure Microsoft into creating backdoors in their software.
     
• The All Writs Act generally has not been interpreted to authorize assistance in retrieving data that you neither own nor control.
  • For an example of this argument, see pages 5-7 of the ACLU’s amicus brief in the 2016 Apple v. FBI case, where law enforcement sought to compel Apple to create and install code onto a target iPhone that would undo the security feature whereby the phone’s contents would be erased after 10 failed passcode attempts.
• The All Writs Act may not apply to you or your company. Historically, the All Writs Act has been used to compel highly regulated public utilities with a duty to serve the public, though more recent cases have held otherwise. There are also limits to what technical assistance orders can compel, especially where you have a valid interest in protecting consumer privacy and security.
  • For an example of this argument, see pages 8-12 of a group of technology companies’ amicus brief from the Apple v. FBI case.
• The order may be too burdensome. The Wiretap Act and the Pen Register Act are explicit that technical assistance demands should cause a “minimum of interference” with the service being provided. If complying would involve diverting resources away from your existing workflowinterfere with the operation of your service, or cause other disruptions, it may be outside the scope of the law.
  • For an example of this argument, see Apple’s response to the 2015 request mentioned above. See also In the Matter of the Application of The United States for an Order Authorizing the Roving Interception of Oral Communications, The Company v. United States, 349 F.3d 1132 (9^th Cir. 2003).
     
• The order may not be necessary to obtain the information sought by the government’s legal process. You may be able to argue that law enforcement should exhaust other available means to collect information on the suspect’s online activity.
   
• An order may ask you to deliver malware that the government has already written, or it could ask you to actually write new code to implement its attack. Either way, the order’s requirement that you deliver a targeted malware update over your update channel may constitute compelled speech and violate your First Amendment rights. This argument may be stronger if you have publicly expressed and established that you will not use your software update channels maliciously or have said that, when you cryptographically sign software updates, the signature certifies that to the best of your knowledge, they are not malicious. Under such a circumstance, you could argue that the compelled speech is a lie to your software users and the public, and it violates your First Amendment rights.
  • For an example of this argument, see pages 32-34 of Apple’s motion to vacate in the Apple v. FBI case.