Ubuntu Guides
Latest
After visiting Tartous, Hamah, Homs, Maloula and Lattakia, Russian delegation, as well International one, f
Among remote control solutions, there is an absolute leader – german TeamViewer. Works everywhere, connects thru firewalls (eg. by using 3rd party strapping) and does it quickly.
"Violence is not an answer" is most famous phrase on the subject. But how do you deal with someone so violent and ruthless, that they beat their enemy to death and then multilate their bodies, shouting on camera "Allahu Akbar" (God is great)?
Taken from InfoWARS.com.Mahdi Darius NazemroayaVoltairenet.org June 17, 2011
Washington and its allies, Israel and the Al-Sauds, are taking advantage of the upheavals in the Arab World. They are now working to dismantle the Resistance Bloc and weaken any drive for democracy in the Arab World. The geo-political chessboard is now being prepared for a broader confrontation that will target Tehran and include Syria, Lebanon, Iraq, and the Palestinians.
President Bashar Al Assad once again showed infinite wisdom in his truthful fight against foreign conspiracy and Islamic fundamentalism.
One would say “Social economy is where you’re satiated, but you have no choice; while open market economy is when you are hungry, but it was your choice only”.
Taken from Russia Today.
Taken from PressTV.
Interview with Dr. Paul Craig Roberts, former assistant secretary to US Treasury, Panama City.
It is not a secret that US government with their allies – Israel, Qatar, Saudi Arabia and some particular movements in Lebanon and Egypt are trying to do everything they can to flood Syria in innocent blood.
Quick Tip #5: How to write Robots.txt
Print this article
When hosting your own site, there might be necessity to configure behavior of bots – programs that crawl your website collecting information about its structure and content, and feed it to appropriate search engine.
For that sake, Robot.txt was created – a file that defines the way bots will behave on your site. Its configuration is pretty straight forward.
Create a text file (in Nano, Vim, Mousepad; Notepad), type in directives and upload it to root folder of your website.
Directives
User-agent: – name of botAllow: – path allowed to crawl. By default, equals to any path (defined as ‘*’).Disallow: – path denied from crawling.
For example, this can be defined as:
User-agent: *Disallow: /admin
To make sure than none bot is accessing, or indexing, your administrative area.
Or like this:
User-agent: GooglebotDisallow: /mail
To deny ONLY Google bot to crawl your /mail directory, but allow all other bots to do so.
NOTE: By default, most crawlers only support “User-agent” and “Disallow” directives.
You can define multiple rules for each bot. For example:
User-agent: GooglebotAllow: /indexable-contentDisallow: /adminUser-agent: BaiduSpiderDisallow: /indexable-contentDisallow: /mailDisallow: /admin
You can also prevent all bots from spidering your site, by writing in Robots.txt:
User-agent: *Disallow: /
Or, define crawl rate using “Crawl-delay” directive:
User-agent: GooglebotCrawl-delay: 5
This will limit Googlebot requests rate to 1 per 5 seconds.
NOTE: Some crawlers do ignore Robots.txt completely, so it cannot be considered as absolute protection from content and structure indexing, and also cannot be considered an effective crawler rate limitation technique.
There is also an unofficial “Sitemap” directive, allowing you to define sitemap in Robots.txt. For example:
Sitemap: http://www.your-site.com/sitemap2.xml
NOTE: This cannot be used to limit access to a site by defining browser string in “User-Agent” directive, because browsers ignore Robots.txt.
Common Bots List
Googlebot – Google's crawler.
Baiduspider – Baidu's crawler (Japanese Chinese search engine)
MSNBOT – Bing's crawler
Gigabot – Gigablast's crawler.
Yahoo! Slurp – Yahoo's crawler
Full list of crawlers can be found at UserAgentString.com.
Add new comment