[Whonix-devel] Whonix mirror system issues

[Patrick Schleizer]

mick:
> On Thu, 09 Apr 2015 18:31:28 +0000
> Patrick Schleizer <adrelanos at riseup.net> allegedly wrote:
> 
>> Hi Mick!
>>
>> The current mirror system is somehow maintenance, time intensive.
>>
>> Every now and then, some mirrors is offline. Then I need to mail the
>> mirror operator and wait until they fixed the issue. If I expect the
>> mirror operattor to take longer than it would take until that IP has
>> been removed from global DNS caches, I need to go to gandi
>> webinterface and manually remove the IP from round robin. Then wait
>> until the global DNS has been updated. Until then x of y users fail
>> to download Whonix images, Whonix packages or Whonix News. This
>> generates support requests. Once the mirror operator fixed the issue,
>> I need to re-add the IP to round robin.
>>
>> The more mirrors added, the more issues of that kind.
>>
>> In meanwhile, what could users be told to at least be able to
>> download? Other than "try again later, maybe you're lucky by then".
>> For Firefox there is only a now somewhat broken DNS flusher add-on.
>> How else could users flush DNS and force use some other IP from round
>> robin? I know how to do it with curl, but how with browsers and
>> apt-get?
>>
>> I am wondering if a php script on whonix.org would do better. One that
>> randomly [or round robin alike] redirects to a mirror IP from a list,
>> where temporarily defunct ones can be easily commented out from a
>> simple text file. A php script that in essence instructs the
>> requester to do something like this:
>> curl -H 'Host: mirror.whonix.de' -k http://86.59.119.84
>>
>> Does that sound possible, sane?
>>
>> By answering to this e-mail, your reply will be posted on the
>> whonix-devel public mailing list.
> 
> Hi Patrick
> 
> Firstly my apologies for the laggardly response. My attention has been
> elsewhere for a while now. I am only just catching up on a bunch of
> stuff I have parked. 
> 
> So: Firstly the DNS issues can be (partly) addressed by winding down
> the TTLs on the mirror records. A low TTL is always a trade off
> between keeping cached records as fresh as possible and preventing
> an unnecessarily high load of requests from time outs. Personally
> I wouldn't set a TTL lower than an hour (3600). 
> 
> But you are right. That doesn't address either the user's browser
> problem or the administrative load on you maintaining the DNS. So
> I think your idea of controlling the end mirror list from
> whonix.org is a good one.
> 
> If, as you suggest, you maintain a list of mirrors on
> whonix.org (of the form:
> 
> mirror0.whonix.de
> mirror1.whonix.de
> mirror2.whonix.de
> etc.
> 
> then you can use a php script to read a random line entry from that
> list to return to the user. If you find that one of the mirrors
> is off-line for some reason, you can simply remove it from the
> list until it is fixed. A suggested form of such a script is
> below.
> 
> Of course, you will have to let the mirror owners (including
> me) know what you want our mirrors to be called (at the moment
> they are all called mirror.whonix.de) and amend the central DNS
> accordingly.
> 
> Does that help?
> 
> Mick
> 
> ------------------- suggested script -------------------------
> 
> <?php
> 
> function random_mirror($max = 1) 
> 	{
> 	$n = 100;
> 	return ((mt_rand(1, $n * $max) -1)/$n);
> 	}
> 
> $line_num = 0;
> 
> $file = fopen('mirrors.txt', 'r') or die("can't open file");
> while (!feof($file))
> 	{
> 	if ($string = fgets($file))
> 		{
> 		$line_num++;
> 		if (random_mirror($line_num) < 1) 
> 			{
> 			$line = $string;
> 			}
> 		}
> 	}
> /** print "$line\n"; **/
> fclose($file) or die();
> 
> ?>
>         
> ------------------- end ---------------
> 
> ---------------------------------------------------------------------
> 
>  Mick Morgan
>  gpg fingerprint: FC23 3338 F664 5E66 876B  72C0 0A1F E60B 5BAD D312
>  http://baldric.net
> 
> ---------------------------------------------------------------------
> 

Yes, that helps.

Thanks,
Patrick