[tor-dev] Brief state of sbws thoughts

[Matt Traudt]

Teor, Juga

There's a lot of things fighting for my attention right now, so you
might have noticed I've slowed way down on attending to sbws
tickets/PRs/etc. I think time will free up in the next few days.

I think sbws is in a very good place code-wise right now. I don't think
much more **has** to be done to the code. Even though I enjoy adding
things like the state file (GHPR#236 [2]), I don't think that was a good
use of my time.

It looks like there's a lot of check boxes Juga has made regarding
making a Debian package[0]. Those should get checked. These are important.

However, I think the absolute most important thing for us to be spending
our time on right now is deciding what "good" results are and verifying
sbws produces "good" results.

To accomplish this, I think one of the two suggestions I made in a
comment on GH#182 [1] (quoted here) is what we should be doing.

1. Run torflow and sbws side-by-side (but not at the same time) to
remove more variables. This has the added benefit of us having access to
the raw scanner results from torflow before it does whatever magic
scaling it does. OR

2. Ask for access to raw scanner results from someone running torflow.

I fear sbws is doomed to die the death of the new bandwidth scanners
before it if we don't start seriously verifying sbws is "good" or if I
personally slowly stop working/coordinating work on it.

Thanks

Matt

[0]: https://trac.torproject.org/projects/tor/ticket/26848
[1]:
https://github.com/pastly/simple-bw-scanner/issues/182#issuecomment-404250053
[2]: https://github.com/pastly/simple-bw-scanner/pull/236

[juga]

Post by Matt Traudt
Teor, Juga
There's a lot of things fighting for my attention right now, so you
might have noticed I've slowed way down on attending to sbws
tickets/PRs/etc. I think time will free up in the next few days.
I think sbws is in a very good place code-wise right now. I don't think
much more **has** to be done to the code. Even though I enjoy adding
things like the state file (GHPR#236 [2]), I don't think that was a good
use of my time.
It looks like there's a lot of check boxes Juga has made regarding
making a Debian package[0]. Those should get checked. These are important.
However, I think the absolute most important thing for us to be spending
our time on right now is deciding what "good" results are and verifying
sbws produces "good" results.
To accomplish this, I think one of the two suggestions I made in a
comment on GH#182 [1] (quoted here) is what we should be doing.
1. Run torflow and sbws side-by-side (but not at the same time) to
remove more variables. This has the added benefit of us having access to
the raw scanner results from torflow before it does whatever magic
scaling it does. OR

In that ticket you also mentioned that someone that already runs torflow
should also run sbws.
I said i can run both, and still the case if needed.

Post by Matt Traudt
2. Ask for access to raw scanner results from someone running torflow.
I fear sbws is doomed to die the death of the new bandwidth scanners
before it if we don't start seriously verifying sbws is "good" or if I
personally slowly stop working/coordinating work on it.

I don't think that's the case. I've not forget it... and i'm sure teor
neither.
Some of the last work we have done is regarding getting the bandwidth
files archived, what will also help to determine whether sbws results
are "good".

If 1. would be run by someone else, getting [0] done is indeed important
and i'm currently working on it.

And maybe we aren't able to determine how "good" sbws results are until
it actually starts being run by dirauths, for which [0] is still important.

Thanks for sharing your thoughts,
juga.

Post by Matt Traudt
Thanks
Matt
[0]: https://trac.torproject.org/projects/tor/ticket/26848
https://github.com/pastly/simple-bw-scanner/issues/182#issuecomment-404250053
[2]: https://github.com/pastly/simple-bw-scanner/pull/236
_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[Tom Ritter]

I'm happy and prepared to run sbws and torflow side by side. I'm a
little less swamped than I was a month ago. I don't need a debian
package; I'd rather run it from a git clone.

I think the only things I can't do are
a) give you access to the box directly (but I can make whatever
files/logs/raw results that you want available to you over HTTP)
b) stop running torflow. (Unless we're ready to switch a live bwauth
over to sbws.)

FWIW, I have the advantage of having archived my (semi-)raw bwauth
data for a while: https://bwauth.ritter.vg/bwauth/

-tom

Post by juga

Post by Matt Traudt
Teor, Juga
There's a lot of things fighting for my attention right now, so you
might have noticed I've slowed way down on attending to sbws
tickets/PRs/etc. I think time will free up in the next few days.
I think sbws is in a very good place code-wise right now. I don't think
much more **has** to be done to the code. Even though I enjoy adding
things like the state file (GHPR#236 [2]), I don't think that was a good
use of my time.
It looks like there's a lot of check boxes Juga has made regarding
making a Debian package[0]. Those should get checked. These are important.
However, I think the absolute most important thing for us to be spending
our time on right now is deciding what "good" results are and verifying
sbws produces "good" results.
To accomplish this, I think one of the two suggestions I made in a
comment on GH#182 [1] (quoted here) is what we should be doing.
1. Run torflow and sbws side-by-side (but not at the same time) to
remove more variables. This has the added benefit of us having access to
the raw scanner results from torflow before it does whatever magic
scaling it does. OR

In that ticket you also mentioned that someone that already runs torflow
should also run sbws.
I said i can run both, and still the case if needed.

Post by Matt Traudt
2. Ask for access to raw scanner results from someone running torflow.
I fear sbws is doomed to die the death of the new bandwidth scanners
before it if we don't start seriously verifying sbws is "good" or if I
personally slowly stop working/coordinating work on it.

I don't think that's the case. I've not forget it... and i'm sure teor
neither.
Some of the last work we have done is regarding getting the bandwidth
files archived, what will also help to determine whether sbws results
are "good".
If 1. would be run by someone else, getting [0] done is indeed important
and i'm currently working on it.
And maybe we aren't able to determine how "good" sbws results are until
it actually starts being run by dirauths, for which [0] is still important.
Thanks for sharing your thoughts,
juga.

Post by Matt Traudt
Thanks
Matt
[0]: https://trac.torproject.org/projects/tor/ticket/26848
https://github.com/pastly/simple-bw-scanner/issues/182#issuecomment-404250053
[2]: https://github.com/pastly/simple-bw-scanner/pull/236
_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

[teor]

Hi,

Post by juga

Post by Matt Traudt
Teor, Juga
There's a lot of things fighting for my attention right now, so you
might have noticed I've slowed way down on attending to sbws
tickets/PRs/etc. I think time will free up in the next few days.
I think sbws is in a very good place code-wise right now. I don't think
much more **has** to be done to the code. Even though I enjoy adding
things like the state file (GHPR#236 [2]), I don't think that was a good
use of my time.
It looks like there's a lot of check boxes Juga has made regarding
making a Debian package[0]. Those should get checked. These are important.
However, I think the absolute most important thing for us to be spending
our time on right now is deciding what "good" results are and verifying
sbws produces "good" results.

You’re right - we need to know if we can switch to sbws, and we can’t use
sbws unless it has reasonable results.

If the results aren’t reasonable, we might need to:
* do further processing on the sbws results (like scaling)
* change the sbws measurement design

The good news is that sbws ranks are approximately the same as torflow ranks.
So the measurement design is probably ok.

But torflow weights are larger (max 100,000) than sbws weights (max 4000),
so we will need to scale the sbws results.

torflow results are also steeper than sbws results: the ratio between high
and low ranked relays is 1000:1 in torflow, but 10:1 in sbws.

If we want to, we can make sbws match torflow by defining a scaling algorithm
that scales large relays more than small relays. But we could also decide that
the flatter sbws curve is better for the network, because high-weight relays
are overloaded.

Let’s do a few more experiments before we decide.

Post by juga

Post by Matt Traudt
To accomplish this, I think one of the two suggestions I made in a
comment on GH#182 [1] (quoted here) is what we should be doing.
1. Run torflow and sbws side-by-side (but not at the same time) to
remove more variables. This has the added benefit of us having access to
the raw scanner results from torflow before it does whatever magic
scaling it does. OR

In that ticket you also mentioned that someone that already runs torflow
should also run sbws.
I said i can run both, and still the case if needed.

Ok, so juga can run sbws and torflow at different times on the same machine.

Post by juga
I'm happy and prepared to run sbws and torflow side by side. I'm a
little less swamped than I was a month ago. I don't need a debian
package; I'd rather run it from a git clone.
I think the only things I can't do are
a) give you access to the box directly (but I can make whatever
files/logs/raw results that you want available to you over HTTP)
b) stop running torflow. (Unless we're ready to switch a live bwauth
over to sbws.)

And tom can run sbws and torflow at the same time on the same machine.

I think we should run both comparisons, wait a week so they are in a stable
state, and then check the results for a few weeks.

T

Post by juga

Post by Matt Traudt
[0]: https://trac.torproject.org/projects/tor/ticket/26848
https://github.com/pastly/simple-bw-scanner/issues/182#issuecomment-404250053
[2]: https://github.com/pastly/simple-bw-scanner/pull/236