CRAWLER
Ilial, Inc. is a Los Angeles based Internet startup backed by
blue chip Northern and Southern California Venture Capitalists. We have
developed technologies that will redefine the $20B online advertising
industry. After several years of stealth development, the company is
poised to launch its service. Additional information regarding our
privacy policy can be found on the following page Privacy Policy.
You may be wondering about "ilial"¯ agent and be curious about why
it is
visiting your site. The Ilial crawler (robot), which identifies itself
as ilial in the HTTP "User-agent" header field, uses a web-wide crawl
strategy. Basically, it starts with a list of known URLs from across
the entire Internet, and then it fetches local links found as it goes.
We will not crawl anything you would like to remain private. By using
the Standard for Robot Exclusion (SRE) you can let us know not to crawl
your site. Based on www.robotstxt.org regulations, we honor both
robots.txt and HTML META TAG format. According to this regulation, if
none of these standards are set, by default it means we can crawl those
pages.
There are two ways to avoid Ilial¯ crawl your site or your page:
Using HTML META TAG
Place the following meta tag in the head of your HTML document:
To learn more about meta tags, please refer to http://www.robotstxt.org/wc/exclusion.html#meta; you can also read what the HTML standard has to say about these tags. Remember, changes to your site won't be immediately reflected; they'll be discovered and propagate when ilial next crawls your site.
Using "robots.txt" file
To exclude all robots, the robots.txt file should look like this:
Disallow: /
To exclude just one directory (and its subdirectories), say, the /images/ directory, the file should look like this:
Disallow: /images/
Disallow:
To prevent ilial from visiting (while allowing all others), your robots.txt file should look like this:
Disallow: /