Serguey Asael Shinder: URL.equals resolves host names, so a HashSet of URLs does DNS lookups
A set of addresses looks like the simplest way to remember which pages a crawler has already seen:
Set<URL> seen = new HashSet<>();
for (String link : links) {
URL url = URI.create(link).toURL();
if (seen.add(url)) { // hashCode and equals run here
fetch(url);
}
}
Nothing in that loop looks like network code. The javadoc of java.net.URL says otherwise. For equals, two URLs are equal if they have the same protocol, port, file and fragment and "reference equivalent hosts", and two hosts are equivalent "if both host names can be resolved into the same IP addresses". Only if a name cannot be resolved does it fall back to comparing the names without regard to case. The next sentence draws the consequence: "Since hosts comparison requires name resolution, this operation is a blocking operation."
hashCode has the same property. It is "based upon all the URL components relevant for URL comparison. As such, this operation is a blocking operation." So seen.add(url) can wait on a name lookup, and a HashMap<URL, ...> used as a cache can turn a slow resolver into a slow application.
The result also depends on the network rather than on the strings. Two different host names served from the same IP address compare equal, and the javadoc warns that this definition "is known to be inconsistent with virtual hosting in HTTP", where many sites share one address. The same pair of URLs can compare differently on a machine where one name does not resolve, because then the fallback is a plain name comparison.

Use URI for identity
java.net.URI defines equality on the components themselves. For two server-based URIs, the hosts "must be equal without regard to case", the ports and user information must be equal, and the path and query are compared as text. No lookup is involved:
Set<URI> seen = new HashSet<>();
for (String link : links) {
URI uri = URI.create(link);
if (seen.add(uri)) { // pure string comparison
fetch(uri.toURL()); // convert only when a connection is needed
}
}
This matches what the URL javadoc already recommends elsewhere. It suggests using URI to parse or construct the address and calling URI.toURL() only when a URL is needed to open a connection, and the URL(String) constructor is deprecated with the advice to use URI.toURL() instead.
URI equality has its own rules worth knowing: it compares raw forms rather than decoded ones, so hello%20world and hello world are not equal, and hexadecimal digits in escapes are compared without regard to case. But its answer depends only on the text in front of it, which is what a key in a map or a set needs.