Discussion:
[tor-dev] Domain Fronting, Meek, Cloudflare, and Encrypted SNI...
Nathaniel Suchy
2018-09-24 17:46:10 UTC
Permalink
Hi everyone,

Cloudflare has added support to TLS 1.3 for encrypted server name
indication (SNI). This mailing list post is a high level overview of how
meek could take advantage of this in relation to Cloudflare who until just
now wasn’t an option for domain fronting.

What this means:
Effectively domain fronting works by sending a different SNI and host
header. CDN providers like Cloudflare started double checking to make
governments happy, scratch that line, I mean to protect their customers
from fraud and abuse. They seem to of backtracked now. Encrypted SNI means
that a firewall or coffee shop owner won’t be able to use SNI to see the
real origin of TLS traffic.

Why this matters:
With the right adjustments for TLS 1.3 and Encrypted SNI support,
Cloudflare may be a viable option for Meek.

Risks:
* Firewall products could always use DPI and block TLS 1.3 altogether.
* Firewall products could block all requests with encrypted SNI.

Thoughts anyone?

References:
* https://blog.cloudflare.com/encrypted-sni/
* https://blog.cloudflare.com/esni
Tom Ritter
2018-09-24 18:15:37 UTC
Permalink
Post by Nathaniel Suchy
Hi everyone,
Cloudflare has added support to TLS 1.3 for encrypted server name
indication (SNI). This mailing list post is a high level overview of how
meek could take advantage of this in relation to Cloudflare who until just
now wasn’t an option for domain fronting.
Effectively domain fronting works by sending a different SNI and host
header. CDN providers like Cloudflare started double checking to make
governments happy, scratch that line, I mean to protect their customers
from fraud and abuse. They seem to of backtracked now. Encrypted SNI means
that a firewall or coffee shop owner won’t be able to use SNI to see the
real origin of TLS traffic.
With the right adjustments for TLS 1.3 and Encrypted SNI support,
Cloudflare may be a viable option for Meek.
* Firewall products could always use DPI and block TLS 1.3 altogether.
* Firewall products could block all requests with encrypted SNI.
Thoughts anyone?
The latter concern seems real enough for me that we should consider not
front-running major adoption in browsers.

-tom
David Fifield
2018-09-24 17:57:48 UTC
Permalink
With the right adjustments for TLS 1.3 and Encrypted SNI support, Cloudflare
may be a viable option for Meek.
* Firewall products could always use DPI and block TLS 1.3 altogether.
* Firewall products could block all requests with encrypted SNI.
I wrote an essay on some of the implications of encrypted SNI here:
https://groups.google.com/d/msg/traffic-obf/UyaLc9jPNmY/ovNImK5HEQAJ
tl;dr: yes, encrypted SNI is a very good thing for censorship
circumvention and privacy more generally; a risk though is that it
further centralizes access to online resources.


----

2018-08-18

Efforts are underway to add SNI encryption as an extension in TLS 1.3.
* https://tools.ietf.org/html/draft-rescorla-tls-esni-00
* https://www.ietf.org/mail-archive/web/tls/current/msg26842.html
I find this a really hopeful development. I appreciate the work of
everyone who is helping to make it a reality (some of them are on this
list). Encrypted SNI will of course be a boon for online privacy
generally, but in our world of censorship circumvention it could be the
biggest thing since the ascendance of TLS. Along with its benefits, I
foresee that encrypted SNI will change the basic game in ways that we
need to be ready for. I expect that we'll need to reevaluate our
customary models and begin to consider new challenges.

At first glance, encrypted SNI—in whatever form it may eventually
take—is a silver bullet. It's domain fronting without the downsides. It
solves all our problems, up to traffic analysis: payload encryption
prevents blocking by content, and SNI encryption protects the
destination address. The censor cannot distinguish various connections
to a TLS server and faces the old dilemma: block all, or block none. And
experience shows that we can find servers with a sufficient degree of
co-hosting that a censor will hesitate to "block all."

So what's the catch? I don't really think there is one, at least not a
major one. SNI encryption is poised to put censorship circumvention on
substantially securer footing. As
https://tools.ietf.org/html/draft-ietf-tls-sni-encryption-03 (which I
encourage you to read) says, "Historically, adversaries have been able
to monitor the use of web services through three channels... These
channels are getting progressively closed." But what I do think SNI
encryption will do is force us to reconsider our threat models. Solving
our current batch of problems is going to uncover new problems—lesser
problems, to be sure—but ones that we have until now mostly ignored
because they were not the most pressing. Censors, too, will be forced
evolve, when they are finally deprived of their last easy traffic
distinguisher. I predict a displacement of the battleground, from the
land of firewalls to new arenas.

It's a credit to everyone's work on domain fronting, and the basic
soundness of the idea, that when it began to falter, it was not because
of the Great Firewalls of the world, but because of the Googles and
Amazons. This phenomenon is an example of what I mean when I say that
old challenges will give way to new ones. We beat the censors at their
game, and resisted direct blocking long enough for another weakness to
reveal itself; i.e., that network intermediaries don't reliably perform
the functions that we depended on. I mean that as an observation of
fact, not as implied judgement—personally I don't really blame Google
and Amazon for their policy change regarding domain fronting. While the
wisdom of the decision is debatable, and I suspect there is more to
their rationale than they have stated publicly, certainly they are under
no obligation to continue supporting an unintended feature, no matter
how useful we find it. But whatever the cause, the fact is that domain
fronting, while demonstrably robust against border-firewall censors, is
susceptible to the changing dispositions of intermediary services. We
reached this frontier of experiential knowledge only because we had
beaten the censor's usual tricks—we transcended the cat-and-mouse game.
I like to draw an analogy with human health: our caveman forebears
didn't worry about dying from heart disease, because it was almost
certain that they would be killed by something else first, a woolly
mammoth, say. It was only after the immediate threat of death by mammoth
had subsided, that humans had the comparative luxury of being concerned
about heart disease. So it is with us now: we built a cat-proof mouse,
and now we see what other worries a mouse has.

I can see something similar playing out with encrypted SNI, only on a
larger and more pervasive scale. Network intermediaries—CDNs, app
stores, hosting providers—are going to face more and more pressure: as
other links in the chain of communication are strengthened, those
intermediaries will become attractive targets of action by censors. We
already see examples of censors having to step out of their accustomed
comfort zones because of generally increasing Internet security, for
example when the government of China pressured Apple to yank VPN apps
from its app store. I contend that if the government had the ability to
block apps all by itself, without petitioning Apple, then that's what it
would have done. That the censor resorted to pressuring a third party
shows a certain weakness, but the fact that it succeeded shows it is
still strong enough for its purposes. It also highlights a shift in
moral responsibility. If the government were able to block apps without
asking, then Apple could just throw up its hands and say: "not my
fault." But because the censor has no choice but to ask, Apple must make
the deliberate choice of whether to become an agent—however unwilling—of
censorship.

We circumvention designers have customarily assumed that network
intermediaries are benevolent, or at least non-malicious—that they do
not collaborate with a censor. We assumed so, because the risk of direct
blocking by a censor overshadowed any other risk. In a world of
encrypted SNI, where the direct risk from the censor is greatly
diminished, we will need to reexamine this assumption. Intermediaries
will become de-facto gatekeepers of information, to an even greater
degree than they are now, and they'll be in the unenviable position of
being the logical place at which to implement censorship. As things
stand now, when a court in India orders a site blocked, it's Airtel's
problem to block it. But when encrypted SNI renders Airtel unable, it'll
be Cloudflare's problem. Now, if I had to choose between the good will
of Cloudflare et al. and that of the GFW, there's no comparison:
obviously you choose Cloudflare every time. And yet, we can't overlook
that Cloudflare once booted a site on a CEO's whim; nor that Google
secretly started building a censored search engine for China. The
operators of network intermediaries, and their commitment to human
rights, will be tested more than ever, and the population of Internet
users will increasingly rely on them to do the right thing.

As circumvention designers, one thing we can do to help those services
help themselves is not to proxy directly from the services themselves,
but to use at least one additional encrypted hop to an separate proxy
server. That way, it becomes technically harder for the services to do
surgical blocking.

I have to admit that I don't fully understand the apparent enthusiasm
for encrypted SNI from groups that formerly were not excited about
domain fronting. It's possible I misunderstand some subtlety, but to my
mind, they both should amount to about the same thing from the their
perspective. The primary difference is one of scale. The stated concerns
with domain fronting also apply to encrypted SNI; in particular that if
one customer gets blocked, it will have a collateral effect on other
customers. Maybe the difference in scale is really it: the cloud
companies are happier to bet against blocking when *all* their
customer's domains are potentially affected, rather than just one. It's
a rational enough viewpoint (greater potential collateral damage → lower
probability of blocking), but to my mind encrypted SNI doesn't
fundamentally alter the nature of the game, it just raises the stakes.
Don't get me wrong: I welcome the adoption of encrypted SNI for whatever
reason. It's better than domain fronting, it'll be nice to have it in a
standard, and once we have it we won't want to go back. But I hope that
operators understand what they're getting into. Will they get cold feet
when push comes to shove—when a future version of Telegram uses
encrypted SNI and Russia again blocks millions of IPs? Or malware adopts
it for C&C and infosec blue teamers get annoyed?

I said earlier that I didn't see any major catch with encrypted SNI. The
minor catch I see is the potential for increased centralization. TLS
with encrypted SNI is likely to be the most effective form of
circumvention, which means that unless you're a major player, or hosted
on one, you'll be at increased risk of being blocked. I've read some
criticism online of circumvention systems, like domain fronting, that
route traffic through the major cloud companies. One the one hand, I
find that kind of criticism annoying, because it's not that the use of
centralized services is a desired, designed-in feature; it's that we
don't yet know how to do it better. Circumvention is already hard
enough, and by demanding that is be simultaneously decentralized, these
critics are asking us not only to juggle, but to do so backwards on
roller skates. But on the other hand, I can sympathize with their point
of view. Despite the difficulty, we *should* aspire to better designs. I
dislike giving connection metadata to Amazon and Microsoft as much as
anyone. Unfortunately, encrypted SNI is likely to move us even farther
from the decentralized end of the scale. It will be so effective, and so
easy to use, that I predict there will be a convergence of systems using
it. We see something like that effect today, where there is a perception
that if you want to resist DoS, you have no choice but to be on one of
the big CDNs. The outcome I fear for the web is something like we have
today with SMTP, where the costs of setting up an independent server are
so great as to make the choice effectively impossible. But I don't want
to overblow my concern here. We should be thinking about ways to
decentralize, but encrypted SNI is worth pursuing even if we can't think
of any.

What are the risks to reaching a future of easy and effective
circumvention using encrypted SNI? The worst case is if the proposal
fails or is permanently stalled: we'll be stuck in a world that is
pretty much like the world of today, except more hostile to domain
fronting, waiting for something else to come along. As I understand it,
draft-rescorla-tls-esni-00 is subject to change before standardization,
and I suppose there's a chance it could morph into something so unwieldy
or undeployable that it fails despite standardization. Most of the
discussion that I've seen so far has been positive, but not all
stakeholders at the IETF love the idea; in particular I get the
impression that some people rely on plaintext SNI for monitoring or
regulatory compliance, and encrypted SNI will make their lives more
difficult. So we have to watch out for it being neutered in a way that
enables censorship. Crucially, the value of encrypted SNI for
circumvention depends on its adoption. If at first it's only
circumventors and malware authors using encrypted SNI, then censors and
security firewalls will start to block it, and then it's permanently
skunked, no use to anybody. What we need is for at least one of the
major browsers to implement encrypted SNI and (importantly) enable it by
default. It's browsers that have to lead the way here, just as they
effectively snuck TLS deployment past the censors' notice, until it was
too late to do anything about it.

And what about traffic analysis; that is, classification based on the
timing and size of packets in an SNI-encrypted TLS stream? My gut
feeling is that it still won't be the first tool that censors reach for.
I see pressure on third parties as a more likely threat. But it becomes
more likely with each passing day, and anyway, my instinct could be
wrong. So I think that research on traffic analysis obfuscation will
become more and more relevant.
Andreas Krey
2018-10-01 18:12:15 UTC
Permalink
Post by David Fifield
I have to admit that I don't fully understand the apparent enthusiasm
for encrypted SNI from groups that formerly were not excited about
domain fronting.
It's simply wrong to use different names in SNI and the host header. :-)
Post by David Fifield
customer's domains are potentially affected, rather than just one. It's
a rational enough viewpoint (greater potential collateral damage ??? lower
probability of blocking), but to my mind encrypted SNI doesn't
fundamentally alter the nature of the game, it just raises the stakes.
But in a game-changingly massive way. Remember the github blocking?

When you block one domain that is on cloudflare, almost noboby will care.
When you block all of cloudflare you will get an outcry of a lot of
people, and probably worse for the censors, businesses.

- Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800
David Fifield
2018-09-25 02:23:58 UTC
Permalink
Effectively domain fronting works by sending a different SNI and host header.
CDN providers like Cloudflare started double checking to make governments
happy, scratch that line, I mean to protect their customers from fraud and
abuse. They seem to of backtracked now. Encrypted SNI means that a firewall or
coffee shop owner won’t be able to use SNI to see the real origin of TLS
traffic.
What we would need in order for meek to used encrypted SNI would be
either:
1) support for encrypted SNI in Go's crypto/tls package; or
2) support for encrypted SNI in the Firefox that ships with Tor
Browser, which meek-client could use through its TLS camouflage
helper support.

IMO (2) is less desirable because I'd like to get rid of the TLS
camouflage helper support and replace it with a Go-level TLS camouflage
library: https://github.com/refraction-networking/utls. The TLS helper
works, but its complexity is a pain to deal with and leads to problems
like https://bugs.torproject.org/12774 https://bugs.torproject.org/25405.

Note, however, that a transport based on encrypted SNI doesn't have to
be meek per se. I use "meek" to refer to a specific combination of
domain fronting and a certain (fairly stupid and inefficient) HTTP/1.1
serialization of a bidirectional byte stream. It's conceptually
straightforward to simply replace the "domain fronting" part with an
"encrypted SNI" part. But it's possible to do better: if you're willing
to abandon HTTP/1.1 compatibility and require HTTP/2, you can use the
"server push" feature to implement a serialization that's much more
efficient than the current one in meek. It may even be better to start
over with a new codebase, it's not like meek's code is that large.

But tjr's point stands: I think it's best not to push anything like this
out until after encrypted SNI has seen some level of adoption by
browsers.
Andreas Krey
2018-10-01 17:55:31 UTC
Permalink
On Mon, 24 Sep 2018 20:23:58 +0000, David Fifield wrote:
...
Post by David Fifield
"encrypted SNI" part. But it's possible to do better: if you're willing
to abandon HTTP/1.1 compatibility and require HTTP/2, you can use the
"server push" feature to implement a serialization that's much more
efficient than the current one in meek.
How about websockets instead of trying to cram this into HTTP/2?

- Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800
David Fifield
2018-10-04 01:01:21 UTC
Permalink
Post by Andreas Krey
...
Post by David Fifield
"encrypted SNI" part. But it's possible to do better: if you're willing
to abandon HTTP/1.1 compatibility and require HTTP/2, you can use the
"server push" feature to implement a serialization that's much more
efficient than the current one in meek.
How about websockets instead of trying to cram this into HTTP/2?
And for that matter, why not a plain old HTTP CONNECT proxy? That would
be even more efficient. But we're limited to what the CDN supports. Most
CDNs only support basic methods like GET and POST, not CONNECT or the
special headers needed by WebSocket.

Cloudflare does support WebSocket, though:
https://www.cloudflare.com/website-optimization/web-sockets/
https://blog.cloudflare.com/cloudflare-now-supports-websockets/
So this, combined with encrypted SNI, could be a viable technique when
tunneling through Cloudflare--it just wouldn't be portable to other
services. We even already have an existing WebSocket-based pluggable
transport implementation--it would need changes to the client to support
encrypted SNI.
https://gitweb.torproject.org/pluggable-transports/websocket.git/
David Fifield
2018-10-04 01:19:02 UTC
Permalink
Post by David Fifield
And for that matter, why not a plain old HTTP CONNECT proxy? That would
be even more efficient.
I should add that--leaving out domain fronting/encrypted SNI--there's an
implementation of exactly this, a pluggable transport built on an HTTP
proxy, by Sergey Frolov et al. He has been trying to get some attention
or buy-in to get it integrated into Tor Browser, but hasn't had much
luck so far. In my opinion, it will make a great alternative to obfs4
and be effective in many situations.

There's a bit more to it than I've described above. It can work with any
HTTP proxy (with HTTPS encryption to hide the destination from the
censor, of course)--but they've also implemented a proxy plugin for the
Caddy web server, which supports authentication. The authentication is
to resist active probing like the GFW does: a genuine client who got the
password through BridgeDB will be able to use the proxy, while a censor
probing IP address will just get the web server's normal pages. Check
the links for more info.

https://bugs.torproject.org/26923
https://github.com/sergeyfrolov/httpsproxy
Andreas Krey
2018-10-04 07:37:18 UTC
Permalink
On Wed, 03 Oct 2018 19:01:21 +0000, David Fifield wrote:
...
Post by David Fifield
And for that matter, why not a plain old HTTP CONNECT proxy?
Because the typical load balancer/forwarder would have to
decide whether to forward that CONNECT or do it itself,
and some other. CONNECT with a Host: header - I'm not
sure there is such a thing.
Post by David Fifield
That would
be even more efficient. But we're limited to what the CDN supports. Most
CDNs only support basic methods like GET and POST, not CONNECT or the
special headers needed by WebSocket.
Yes. No. A quick search indicates that aws and azure are already
supporting it, although I'm unable to interpret whether that is
actually the respective product you are/were using.

But websockets are a relevant thing unlike CONNECT, so I do expect
all major players to implement that (and the components I know of
(haproxy, nginx, apache, golang) are there already).

- Andreas
--
"Totally trivial. Famous last words."
From: Linus Torvalds <torvalds@*.org>
Date: Fri, 22 Jan 2010 07:29:21 -0800
David Fifield
2018-10-04 15:14:06 UTC
Permalink
Post by Andreas Krey
A quick search indicates that aws and azure are already
supporting it, although I'm unable to interpret whether that is
actually the respective product you are/were using.
That's exactly it. Of course you can spin up a random EC2 machine or
Azure VM and connect to it with WebSocket. But a random AWS or Azure IP
address doesn't have the properties we want--it is easily blockable.
Why? Because there are no other valuable services running on the same IP
address, once it is discovered, the censor can block it with virtually
zero cost. (Not that it's an unworkable model--that's basically how
obfs4 works--it means you have to have to be careful about distributing
bridges, and keep replenishing the pool as they are inevitably blocked.)
Encrypted SNI does not help you here, not when there is only one service
running on the IP address.

We specifically use the CDN service, because the high degree of
co-hosting behind a CDN edge server makes it more expensive to block. In
this case, encrypted SNI (or formerly domain fronting) really does help,
because there are many and valuable sites hosted there, and connections
to different sites look the same from the censor's point of view. We're
not connecting via a cloud service just for fun, but because we depend
critically on that co-hosting to make blocking difficult. It's possible
that services other than the CDN services also have that property--it
would require some investigation to uncover--but then you would have to
ensure that the other service, too, supports WebSocket.

I'm not saying a WebSocket-in-TLS transport is a bad idea. It's just
that there's more to it than the protocol you use. You also have to go
through a proxy server that's expensive to block. As a proxy server,
most CDNs will support an HTTP/2 tunnel, but not WebSocket. And if you
don't have a CDN in the middle, you may as well use a plain CONNECT
proxy rather than WebSocket.

Related reading:
https://tools.ietf.org/html/draft-ietf-tls-sni-encryption-03#section-2.2
Many deployments still allocate different IP addresses to
different services, so that different services can be identified
by their IP addresses. However, content distribution networks
(CDN) commonly serve a large number of services through a small
number of addresses.
https://tools.ietf.org/html/draft-ietf-tls-esni-01#section-3.1
In principle, the provider might not be the origin for any
domains, but as a practical matter, it is probably the origin
for a large set of innocuous domains, but is also providing
protection for some private domains.
David Fifield
2018-10-23 22:32:41 UTC
Permalink
Post by David Fifield
What we would need in order for meek to used encrypted SNI would be
1) support for encrypted SNI in Go's crypto/tls package; or
2) support for encrypted SNI in the Firefox that ships with Tor
Browser, which meek-client could use through its TLS camouflage
helper support.
IMO (2) is less desirable because I'd like to get rid of the TLS
camouflage helper support and replace it with a Go-level TLS camouflage
library: https://github.com/refraction-networking/utls. The TLS helper
works, but its complexity is a pain to deal with and leads to problems
like https://bugs.torproject.org/12774 https://bugs.torproject.org/25405.
I wrote an untested overview of how to adapt meek to use ESNI, using an
external copy of Firefox Nightly rather than Tor Browser's built-in copy
of Firefox. Testing this out to see if it works would be a good task for
someone who wants to get involved with pluggable transports.

Use ESNI via Firefox HTTPS helper
https://bugs.torproject.org/28168

1. Download Tor Browser and Firefox Nightly.
2. Go to about:config in Firefox nightly and set
network.trr.mode=3
network.trr.uri=https://1.1.1.1/dns-query
network.security.esni.enabled=true
3. Copy the meek-http-***@bamsoftware.com.xpi from Tor Browser to
Firefox Nightly.
4. Hack meek-client-torbrowser/{mac,linux,windows}.go to point
firefoxPath at the copy of Firefox Nightly and disable the custom
profile. (Additional hacks to remove hardcoded Tor Browser
assumptions may be required.)
5. Set up a Cloudflare instance pointing to https://meek.bamsoftware.com/,
call it https://meek.example.com/.
6. Set up a custom bridge in Tor Browser, using url= without front=
(because we're no longer domain fronting).
bridge meek 0.0.2.0:3 url=https://meek.example.com/

The only slightly weird part I foresee is hacking
meek-client-torbrowser; it has some internal hardcoded paths and
profiles that are specific to the Tor Browser directory layout, and
you'll have to point those to an external Firefox Nightly. Of course,
once ESNI support makes its way into Tor Browser itself, there won't be
a need for another external copy of Firefox.

Loading...