On Aug 22, 2011, at 9:35 AM, Martijn Grooten wrote:
URL shorteners (bit.ly, goo.gl, tinyurl.com etc.) have become popular in
recent years for rather obvious reasons. They are being used by spammers for
equally obvious reasons - both in email and on other platforms (e.g. Twitter).
A filter that checks URLs/domains against a blacklist will either miss the
bad domains hidden behind the shorteners or, if they blacklist the shortener,
find itself blocking legitimate messages. Do Not Use (third-party) URL
Shorteners is sound advice to those sending email, but it's not going to stop
random users from copying shortened URLs from Twitter or Facebook and pasting
them into emails and shortened URLs are unlikely to stop featuring on Twitter.
Tell those providing shorteners to check URLs against blacklists is also a
good idea - and probably necessary for them to stop ending up on blacklists
themselves - but if a filter happens to prefer a different blacklist it
doesn't help much. (I also don't know if checks are made every time someone
clicks on the link or just when the shortened URL is generated.)
So I was wondering if it would help if shorteners published the URLs in a DNS
txt record. As the path of a shortened URL usually consists of lowercase,
uppercase letters and numbers, the uppercase letters need to be encoded, e.g.
by preceding them with an underscore. So for instance to look up the URL
behind
http://bit.ly/gkP0H
would require a lookup of the TXT record for
gk_p0_h._short.bit.ly
Now I don't know if this is something that would actually help those
developing spam-/content-filters. Doing a HTTP lookup to determine the URL
isn't exactly rocket science - though intuitively, it seems more 'natural' to
use DNS, especially if that's what is used for the URL blacklist lookup.
Nor do I know if this would be something that would interest those providing
shortening services. As it would allow browsers to avoid making a HTTP
request to their services, it would mean they would stop having reliable
click through statistics which, I guess, are a source of revenue to them.
It's not going to happen.
To do this would require a fairly complex, database backed "stunt" DNS server,
hooked fairly intimately into the URL shorteners core application.
URL shorteners are deployed by people who understand webservers and HTTP, not
DNS servers. Getting them to deploy anything beyond the basic DNS required to
serve a website is a non-starter.
The only thing I'd expect to get any level of traction for would be something
that can be implemented in a webserver, in much the same way as the URL
shortener itself is implented (and, probably, running in the same webapp as the
shortener itself).
There are several obvious ways to do that - http queries to a related hostname
(http://destination.wttw.me/q8Qzth), a related path (http://wttw.me/d/q8Qzth),
a suffix to the URL, a different MIME type, etc, etc.
The cleanest way would be to use a different request type. Conveniently, that's
already implemented. If you send a HEAD request for the URL, you'll get a 301
Moved response, and a Location: header containing the target URL.
And, at least for the URL shortener I use (bit.ly), that HEAD request will not
count as a click for statistics tracking.
platter:steve$ telnet wttw.me 80
Trying 168.143.174.97...
Connected to wttw.me.
Escape character is '^]'.
HEAD /q8Qzth HTTP/1.1
Host: wttw.me
Connection: close
HTTP/1.1 301 Moved
Server: nginx
Date: Mon, 22 Aug 2011 16:52:38 GMT
Content-Type: text/html; charset=utf-8
Connection: close
Set-Cookie: _bit=4eXXXXXXXXXXXXXXa8;domain=.wttw.me;expires=Sat Feb 18
11:52:38 2012;path=/; HttpOnly
Cache-control: private; max-age=90
Location: http://irtf.org/
MIME-Version: 1.0
Content-Length: 108
Connection closed by foreign host.
Cheers,
Steve
_______________________________________________
Asrg mailing list
Asrg(_at_)irtf(_dot_)org
http://www.irtf.org/mailman/listinfo/asrg