[tor-dev] Help Wanted: Fallback Directory Update

Discussion:

teor

2017-12-11 03:51:17 UTC

(You might have already read this email on a closed list.
I'm re-posting it here, with an extra "step 0".)

Hi all,

Fallback directory mirrors help clients reach the Tor network.

We need to rebuild the list every so often when their details change.
(We have an automated check for 25% failure, and it just triggered.)

Does anyone want to help me do this?
It would be nice to get it done this month - but it's not urgent.

Here's what needs to be done in detail:

0. Ask operators to opt-in stable relays as fallback directory mirrors
If they are on the whitelist already, they don't need to do anything.
Ticket: https://trac.torproject.org/projects/tor/ticket/22527
File: https://gitweb.torproject.org/tor.git/tree/scripts/maint/fallback.whitelist

1. Update the fallback whitelist with new and changed relay details
Tickets: https://trac.torproject.org/projects/tor/ticket/22321
https://trac.torproject.org/projects/tor/ticket/22527
File: tor/scripts/maint/fallback.whitelist

2. Update updateFallbackDirs.py to have a CONSENSUS_EXPIRY_TOLERANCE of 0.
Ticket: https://trac.torproject.org/projects/tor/ticket/20942
File: tor/scripts/maint/updateFallbackDirs.py

3. Run updateFallbackDirs.py with the new whitelist and CONSENSUS_EXPIRY_TOLERANCE.
You'll need a decent network connection for this.
Please attach the script logs to the trac ticket.
Command: tor/scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc

4. Write a changes file
Ticket: https://trac.torproject.org/projects/tor/ticket/22271
Draft:

o Minor features (fallback directory list):
- Replace the 151 fallbacks originally introduced in Tor 0.3.1.3-alpha in
May 2017 (of which ~111 were still functional) with a list of
NNN fallbacks (NNN new, NNN unchanged, NNN removed) generated in December
2017. Resolves ticket 22271.

There are some commands that count fallback changes here:
https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors#FormattingChangesFiles

5. Announce the new list on tor-relays@

6. Tell the relay search maintainer that the list has changed

Next Time:

I'd like to make the process quicker and easier next time.
But I don't want to make any big changes to it at the moment.
Let's do that when we have time to try different things.

T

40/151 (26%) fallback directories have become slow or unresponsive...
(snip)

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------

teor

2017-12-13 01:58:11 UTC

Permalink

Hi All,

Dominique, pastly, and Hello71 have offered to help out with this.
(I don't have Hello71's email, I'll ping them on IRC.)

Let's start by fixing up some of the code, and asking relay operators
if they want to be on the list.

Post by teor
...
Fallback directory mirrors help clients reach the Tor network.
We need to rebuild the list every so often when their details change.
(We have an automated check for 25% failure, and it just triggered.)
...

pastly, I drafted an email for step 0, can you check it makes sense?
(I used to send out way too much info. I'm trying to be brief.)

https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors#ATypicalRelease

Post by teor
0. Ask operators to opt-in stable relays as fallback directory mirrors
If they are on the whitelist already, they don't need to do anything.
Ticket: https://trac.torproject.org/projects/tor/ticket/22527
File: https://gitweb.torproject.org/tor.git/tree/scripts/maint/fallback.whitelist
1. Update the fallback whitelist with new and changed relay details
Tickets: https://trac.torproject.org/projects/tor/ticket/22321
https://trac.torproject.org/projects/tor/ticket/22527
File: tor/scripts/maint/fallback.whitelist

Dominique, do you want to do Step 2?
It's a one-line patch that would be a good intro to Tor's bug tracker and git.
Please base it on the "master" branch.

Post by teor
2. Update updateFallbackDirs.py to have a CONSENSUS_EXPIRY_TOLERANCE of 0.
Ticket: https://trac.torproject.org/projects/tor/ticket/20942
File: tor/scripts/maint/updateFallbackDirs.py

There are two more optional tasks if someone likes python:

Mark the fallbacks that have extra-info descriptors, so stem can use them
https://trac.torproject.org/projects/tor/ticket/22759

Add fallback nicknames to the file, so stem can use them
https://trac.torproject.org/projects/tor/ticket/24600

Post by teor
3. Run updateFallbackDirs.py with the new whitelist and CONSENSUS_EXPIRY_TOLERANCE.
You'll need a decent network connection for this.
Please attach the script logs to the trac ticket.
Command: tor/scripts/maint/updateFallbackDirs.py > src/or/fallback_dirs.inc

And there's another step here I forgot:

3a. Look at the warnings from the script to see which addresses have changed.
Contact operators to see if the new addresses are permanent.
Then repeat steps 1 and 3 with the new details.
Ticket: https://trac.torproject.org/projects/tor/ticket/22321

Post by teor
4. Write a changes file
Ticket: https://trac.torproject.org/projects/tor/ticket/22271
- Replace the 151 fallbacks originally introduced in Tor 0.3.1.3-alpha in
May 2017 (of which ~111 were still functional) with a list of
NNN fallbacks (NNN new, NNN unchanged, NNN removed) generated in December
2017. Resolves ticket 22271.
https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors#FormattingChangesFiles

Once this is done, the list gets backported to 0.2.8 and later.
But the rest of the changes go into master.
Example: https://lists.torproject.org/pipermail/tor-relays/2017-May/012285.html

Post by teor
6. Tell the relay search maintainer that the list has changed

T

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------

teor

2017-12-19 00:42:38 UTC

Permalink

Hi All,

We're going to meet in about 22 hours (2300 UTC Tuesday 19 December)
in #tor-dev to work on this. (That's the network team patch party
time.)

I haven't done much, because I've been busy with work deadlines.

Post by teor
Hi All,
Dominique, pastly, and Hello71 have offered to help out with this.
(I don't have Hello71's email, I'll ping them on IRC.)
Let's start by fixing up some of the code, and asking relay operators
if they want to be on the list.

pastly, I drafted an email for step 0, can you check it makes sense?
(I used to send out way too much info. I'm trying to be brief.)
https://trac.torproject.org/projects/tor/wiki/doc/UpdatingFallbackDirectoryMirrors#ATypicalRelease

I am going to send this out today or tomorrow when I get time.

I want to add a note here about the ongoing DoS attacks.
Maybe this means adjusting the uptime requirement.

Post by teor

Dominique, do you want to do Step 2?
It's a one-line patch that would be a good intro to Tor's bug tracker and git.
Please base it on the "master" branch.

This is done, I have to review it.

Post by teor

Post by teor
2. Update updateFallbackDirs.py to have a CONSENSUS_EXPIRY_TOLERANCE of 0.
Ticket: https://trac.torproject.org/projects/tor/ticket/20942
File: tor/scripts/maint/updateFallbackDirs.py

Mark the fallbacks that have extra-info descriptors, so stem can use them
https://trac.torproject.org/projects/tor/ticket/22759
Add fallback nicknames to the file, so stem can use them
https://trac.torproject.org/projects/tor/ticket/24600

3a. Look at the warnings from the script to see which addresses have changed.
Contact operators to see if the new addresses are permanent.
Then repeat steps 1 and 3 with the new details.
Ticket: https://trac.torproject.org/projects/tor/ticket/22321

Once this is done, the list gets backported to 0.2.8 and later.
But the rest of the changes go into master.
Example: https://lists.torproject.org/pipermail/tor-relays/2017-May/012285.html

Post by teor
6. Tell the relay search maintainer that the list has changed

T

--
Tim / teor

PGP C855 6CED 5D90 A0C5 29F6 4D43 450C BA7F 968F 094B
ricochet:ekmygaiu4rzgsk6n
------------------------------------------------------------------------