[Whonix-devel] Whonix mirror system issues

[mick]

On Thu, 09 Apr 2015 18:31:28 +0000
Patrick Schleizer <adrelanos at riseup.net> allegedly wrote:

> Hi Mick!
> 
> The current mirror system is somehow maintenance, time intensive.
> 
> Every now and then, some mirrors is offline. Then I need to mail the
> mirror operator and wait until they fixed the issue. If I expect the
> mirror operattor to take longer than it would take until that IP has
> been removed from global DNS caches, I need to go to gandi
> webinterface and manually remove the IP from round robin. Then wait
> until the global DNS has been updated. Until then x of y users fail
> to download Whonix images, Whonix packages or Whonix News. This
> generates support requests. Once the mirror operator fixed the issue,
> I need to re-add the IP to round robin.
> 
> The more mirrors added, the more issues of that kind.
> 
> In meanwhile, what could users be told to at least be able to
> download? Other than "try again later, maybe you're lucky by then".
> For Firefox there is only a now somewhat broken DNS flusher add-on.
> How else could users flush DNS and force use some other IP from round
> robin? I know how to do it with curl, but how with browsers and
> apt-get?
> 
> I am wondering if a php script on whonix.org would do better. One that
> randomly [or round robin alike] redirects to a mirror IP from a list,
> where temporarily defunct ones can be easily commented out from a
> simple text file. A php script that in essence instructs the
> requester to do something like this:
> curl -H 'Host: mirror.whonix.de' -k http://86.59.119.84
> 
> Does that sound possible, sane?
> 
> By answering to this e-mail, your reply will be posted on the
> whonix-devel public mailing list.

Hi Patrick

Firstly my apologies for the laggardly response. My attention has been
elsewhere for a while now. I am only just catching up on a bunch of
stuff I have parked. 

So: Firstly the DNS issues can be (partly) addressed by winding down
the TTLs on the mirror records. A low TTL is always a trade off
between keeping cached records as fresh as possible and preventing
an unnecessarily high load of requests from time outs. Personally
I wouldn't set a TTL lower than an hour (3600). 

But you are right. That doesn't address either the user's browser
problem or the administrative load on you maintaining the DNS. So
I think your idea of controlling the end mirror list from
whonix.org is a good one.

If, as you suggest, you maintain a list of mirrors on
whonix.org (of the form:

mirror0.whonix.de
mirror1.whonix.de
mirror2.whonix.de
etc.

then you can use a php script to read a random line entry from that
list to return to the user. If you find that one of the mirrors
is off-line for some reason, you can simply remove it from the
list until it is fixed. A suggested form of such a script is
below.

Of course, you will have to let the mirror owners (including
me) know what you want our mirrors to be called (at the moment
they are all called mirror.whonix.de) and amend the central DNS
accordingly.

Does that help?

Mick

------------------- suggested script -------------------------

<?php

function random_mirror($max = 1) 
	{
	$n = 100;
	return ((mt_rand(1, $n * $max) -1)/$n);
	}

$line_num = 0;

$file = fopen('mirrors.txt', 'r') or die("can't open file");
while (!feof($file))
	{
	if ($string = fgets($file))
		{
		$line_num++;
		if (random_mirror($line_num) < 1) 
			{
			$line = $string;
			}
		}
	}
/** print "$line\n"; **/
fclose($file) or die();

?>
        
------------------- end ---------------

---------------------------------------------------------------------

 Mick Morgan
 gpg fingerprint: FC23 3338 F664 5E66 876B  72C0 0A1F E60B 5BAD D312
 http://baldric.net

---------------------------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.whonix.org/pipermail/whonix-devel/attachments/20150419/93ffe771/attachment.sig>