Ifayile ye robots.txt egcinwe kwingcambu yewebhusayithi yakho iya kuxela iirobhothi zewebhu njengeentsimbi ze- injini ze- injini zokufumana izikhokelo kunye neefayili ezivunyelwe ukukhasa. Kulula ukusebenzisa ifayile ye robots.txt, kodwa kukho ezinye izinto omele uzikhumbule:
- Iifowuni zeebhanti zeMnyama ziza kuhoywa iifayile zakho ze-robots.txt. Iindidi eziqhelekileyo ziyi-bots kunye neerobhothi zikhangela iidilesi ze-imeyile ukuvuna.
- Abanye abatsha beprogram baza kubhala iirobhothi ezingayihoyi ifilothi ye-robots.txt. Oku kufuthi kwenziwe ngephutha.
- Nabani na onokubona ifayile yakho yeerobhothi.txt. Zisoloko zibizwa ngokuthi i-robots.txt kwaye zihlala zigcinwe kwingcambu yewebhusayithi.
- Ekugqibeleni, ukuba umntu udibanisa nefayili okanye ulawulo olukhutshwa ngaphandle kwefayile yakho ye robots.txt ukusuka kwiphepha elingabandakanyiweyo ngefayile yabo ye-robots.txt, iinjine zokukhangela ziyakuthi zifumane.
Ungasebenzisi iifayile ze robots.txt ukufihla into ebalulekileyo. Kunoko, kufuneka ubeke ulwazi olubalulekileyo emva kweephasiwedi ezikhuselekileyo okanye ushiye kwiwebhu ngokupheleleyo.
Indlela yokusebenzisa ezi Fayile zeSampula
Khuphela isicatshulwa kwisampuli esondeleyo kwinto ofuna ukuyenza, kwaye unamathisele kwifayile yakho ye-robots.txt. Shintsha i-robot, i-directory, kunye namagama efayile ukufanisa ukucwangciswa kwakho okukhethiweyo.
Iifayile ezimbini ezisisiseko zeRobots.txt
Umsebenzisi-nkonzo: *
Vumeli: /
Le fayili ithi nayiphi na i-robot (I-agent-agent: *) efikelela kuyo kufuneka ihoxise yonke iphepha kwisayithi (Vumela: /).
Umsebenzisi-nkonzo: *
Vumela:
Le fayile ithi nayiphi na i-robot (i-user-agent: *) efikelela kuyo ivumelekile ukujonga zonke iifayili kwisayithi (Vumela:).
Unokwenza oku ngokushiya ifayile yakho yeerobot.txt ingenanto okanye ungenayo enye kwisayithi yakho.
Khusela iileta ezicacileyo ezivela kwiiRobhothi
Umsebenzisi-nkonzo: *
Ungavumeli: / cgi-bin /
Ungavumeli: / i-temp /
Le fayile ithi nayiphi na i-robot (i-user-agent: *) efikelela kuyo kufuneka ihoxise iirejista / cgi-bin / kunye / temp / (Vumela: / cgi-bin / Disallow: / temp /).
Khusela Amaphepha athile avela kumarobhothi
Umsebenzisi-nkonzo: *
Ungavumeli: /jenns-stuff.htm
Ungavumeli: /private.php
Le fayile ithi nayiphi na i-robot (I-agent-agent: *) efikelela kuyo kufuneka ihoxise iifayile /jenns-stuff.htm kunye /private.php (Vumela: /jenns-stuff.htm Vumela: /private.php).
Ukuthintela iRobot ethile kwi-Access Site yakho
Umsebenzisi-nkonzo: i-Lycos / xx
Vumeli: /
Le fayili ithi i-Lycos bot (I-user-agent: i-Lycos / xx) ayivunyelwe ukufikelela nakuphi na indawo kwisiza (Vumela: /).
Vumela Ukuphela kweeRobot Access eyodwa
Umsebenzisi-nkonzo: *
Vumeli: /
Umsebenzisi-agent: Googlebot
Vumela:
Le fayile kuqala ichithe zonke iirobhothi njengathi senza ngentla, kwaye ngokucacileyo ivumela iGooglebot (i-Agent-agent: Googlebot) inokufikelela kuyo yonke into (Vumela:).
Hlanganisa iMida eMininzi ukuze ufumane ngokuchanekileyo i-Exclusions You Want
Ngelixa kungcono ukusebenzisa umgca we-Agent-agent-inclusive, njenge-Agent-agent: *, unokuba ngumntu othe wathanda. Khumbula ukuba ii-robots zifunde ifayile ngokulandelelana. Ngoko ukuba imida yokuqala ithi zonke iiprobhothi zivaliwe kuyo yonke into, kwaye kamva kwifayili ithi zonke iirobhothi zivunyelwe ukufikelela kuyo yonke into, iirobhothi ziya kuba nokufikelela kuyo yonke into.
Ukuba awuqinisekanga ukuba ubhale ifayile yakho i-robots.txt ngokuchanekileyo, unokusebenzisa i-Google Webmaster Tools ukujonga iifayile zakho ze-robots.txt okanye ubhale entsha.