List of supported Spiders and Crawlers ordered by country

Web site optimization | search engine optimization | Web ite indexation

The database of Editor and Analyzer has more than 150 spiders and it is regularly updated. This data is used in both the standard exclusion file editor for spider management and the log analyzer for spider detection. You can easily edit existing spiders and add new crawlers to your own database on the Spiders List tab. If you know a popular spider and would like to see it in the next database update, please let us know.

View the list of spiders ordered by primary language

Database update date: 16 January, 2004
Spiders in the database: 184

Australia (2) Austria (1) Belgium (1) Canada (3) China (2) Denmark (2) Finland (1) France (10) Germany (12) Hong Kong (1) Hungary (3) Ireland (1) Italy (2) Japan (11) Korea, Republic of (2) Netherlands, The (1) New Zealand (2) Norway (2) Poland (1) Portugal (1) Russian Federation (3) Spain (1) Sweden (3) Switzerland (3) Taiwan (1) Thailand (1) USA (100) Ukraine (3) United Kingdom (6) Viet Nam (1) Yugoslavia (1)

• AusLII (User-agent: Gromit)English
• WebWombat (User-agent: www.WebWombat.com.au)English

• Suchknecht (User-agent: Suchknecht.at-Robot)German

• RoadHouse (User-agent: RHCS)English

• Portal Juice (User-agent: Pjspide)English
• RoboCrawl (User-agent: robocrawl)English
• spider_monkey (User-agent: spider_monkey)English

• BaiDu (User-agent: BaiDuSpider)Chinese
• Felix IDE (User-agent: felix)Chinese

• Denmex (User-agent: Denmex)English
• Jubii (User-agent: jubii)Danish

• Hamahakki (User-agent: finnish)Finnish

• Antisearch (User-agent: antibot)French
• DeepIndex (User-agent: DeepIndex)French
• Dir.com (User-agent: Pompos)French
• Exalead (User-agent: Exalead)French
• Inria (User-agent: xyro xyro_)French
• NG (User-agent: NG/1.0 NG/2.0)English
• Synomia (User-agent: SynoBot)French
• Voila (User-agent: VoilaBot)French
• WanadooRechereche (User-agent: SurferF3)French
• XYLEME (User-agent: cosmos)English

• Abacho (User-agent: AbachoBOT)German
• Acoon (User-agent: Acoon)German
• CyberCon (User-agent: motor)German
• Die Blinde Kuh (User-agent: blindekuh)German
• Fireball.de (User-agent: Firefly KIT-Fireball)German
• Nathan (User-agent: Tarantula)German
• SpeedFind (User-agent: speedfind ramBot xtreme)German
• Suchbot (User-agent: suchbot)German
• Suchmaschine21 (User-agent: CoolBot)German
• Tigersuche.de (User-agent: Tigersuche)German
• WebSearchBench (User-agent: WebSearchBench)German
• Webspinne.de (User-agent: Webspinne)German

• Suntek search engine (User-agent: suntek)Chinese

• Goliat (User-agent: Goliat goliatspider263)Hungarian
• Let Find It Now! (User-agent: ELFINBOT)English
• Sztaki.hu (User-agent: Computer_and_Automation_Research_Institute_Crawler)English

• ADSA (User-agent: ADSARobot)English

• Arianna (User-agent: arianna)Italian
• Iltrovatore (User-agent: IlTrovatore-Setaccio)Italian

• @nifty (User-agent: InfoNaviRobot(F109))Japanese
• Goo (User-agent: moget)Japanese
• Kototoi (User-agent: Kototoi)Japanese
• Lisa (User-agent: Voyager)Japanese
• Navi (User-agent: griffon nttdirectory_robot)Japanese
• NEC-MeshExplorer (User-agent: meshexplorer)Japanese
• NTT Directory (User-agent: nttdirectory_robot)Japanese
• Steeler (User-agent: Steeler)English
• Verno (User-agent: iron33)Javanese
• Yappo (User-agent: ko_yappo_robot)Japanese
• YokogaoSearch (User-agent: suzuran swbot/0.9c libwww/5.3.1)English

• Daum (User-agent: RaBot)Korean
• Naver.com (User-agent: NaverBot dloader dumrobo NaverRobot Cowbot)Korean

• Ingrid (User-agent: INGRID)Dutch

• Katipo (User-agent: kapito)English
• nzexplorer (User-agent: nzexplorer)English

• Boitho (User-agent: boitho)Norwegian
• Kvasir (User-agent: Solbot)Norwegian

• Szukacz (User-agent: Szukacz)Polish

• Cusco (User-agent: cusco)Portuguese

• Rambler (User-agent: StackRambler)Russian
• UDMSearch (User-agent: mnoGoSearch)English
• Yandex (User-agent: Yandex)Russian

• Abcdatos (User-agent: abcdatos_botlink)Spanish

• EntireWeb (User-agent: speedy)English
• Euroseek (User-agent: Freecrawl Arachnoidea)English
• The NWI Robot (User-agent: w3index)English

• Alkaline (User-agent: AlkalineBOT)English
• Augurfind (User-agent: augurfind)English
• Search.ch (User-agent: search.ch)German

• Openfind (User-agent: Openbot Openfind)English

• SpiderKU (User-agent: SpiderKU)Thai

• ABCsearch (User-agent: ABCsearch)English
• AESOP (User-agent: AESOP_com_SpiderMan)English
• Ah-ha (User-agent: ah-ha.com)English
• Alexa (User-agent: ia_archiver)English
• AllTheWeb (User-agent: FAST-WebCrawler FASTCrawler FAST-RealWebCrawler)English
• AltaVista (User-agent: Scooter AVFetch AltaVista Mercator roach.smo.av.com Tv33_Mercator)English
• Anzwers (User-agent: AnzwersCrawl)English
• Araneo (User-agent: araneo)Esperanto
• ARIADNE (User-agent: ariadne)English
• AskJeeves (User-agent: Jeeves askjeeves)English
• ASPseek (User-agent: ASPseek)English
• ASTRAFIND (User-agent: AstraSpider)English
• Atomz.com (User-agent: Atomz)English
• BBot (User-agent: BBot)English
• Boito (User-agent: boito)English
• Borg-Bot (User-agent: borg-bot)English
• ChristCENTRAL.com (User-agent: christcrawler)English
• ComputingSite Robi (User-agent: robi)English
• Ditto (User-agent: DittoSpyder)English
• Eureka (User-agent: EZResult)English
• Excite (User-agent: ArchitextSpider)English
• FastHealth (User-agent: lachesis)English
• FastSearch (User-agent: Fast)English
• FirstGov (User-agent: FirstGov.gov)English
• Fluid Dynamics (User-agent: FDSE)English
• FusionBOT (User-agent: galaxy)English
• GAIS (User-agent: Gaisbot)English
• GeckoBot (User-agent: geckobot)English
• GenDoor (User-agent: GenCrawler)English
• Gigablast (User-agent: Gigabot)English
• Google (User-agent: Googlebot)English
• Greenpac.com (User-agent: inspectorwww Greenpac)English
• Grub (User-agent: grub-client)English
• Hometown Spider Pro (User-agent: hometown)English
• Hoppa (User-agent: Toutatis)English
• ht://Dig (User-agent: htdig)English
• Hubat (User-agent: Hubater)English
• IBM_Planetwide (User-agent: IBM_Planetwide almaden)English
• IncyWincy (User-agent: incywincy NetResearchServer)English
• Infoseek.com (User-agent: infoseek infoseeksidewinder UltraSeek)English
• Inktomi (User-agent: Slurp LG_catalog moget)English
• Intelliseek (User-agent: Intelliseek)English
• InternetSeer.com (User-agent: internetseer)English
• IP3000 (User-agent: ip3000.com-crawler)English
• Larbin (User-agent: larbin larbin_devel)English
• Lexis-Nexis (User-agent: LNSpiderguy)English
• Look (User-agent: Look.com)English
• Looksmart (User-agent: MantraAgent WISENutBot WiseNut)English
• Lycos (User-agent: Lycos funnelweb WhoWhere)English
• MagPortal (User-agent: legs)English
• Maxbot (User-agent: maxbot)English
• MerzScope (User-agent: MerzScope)English
• Mopilot.com (User-agent: Wapspider)English
• MSNBOT (User-agent: MSNBOT)English
• NationalDirectory (User-agent: NationalDirectory-SuperSpider NationalDirectory-WebSpider)English
• NetNose (User-agent: NetNose)English
• Northern Light (User-agent: Gulliver)English
• NPBot (User-agent: NPBot)English
• Nutch (User-agent: NutchOrg NutchCVS nutch)English
• Only.com (User-agent: Spida)English
• PerlCrawler (User-agent: perlcrawler)English
• PhpDig (User-agent: phpdig)English
• PicSearch (User-agent: psbot)English
• Pinpoint.com (User-agent: CrawlerBoy)English
• PlanetSearch (User-agent: Fido)English
• Poppi (User-agent: poppi)English
• PortalB Spider (User-agent: portalb)English
• QuePasa (User-agent: QuepasaCreep)Spanish
• Raven Search (User-agent: Raven)English
• Robozilla (User-agent: robozilla)English
• Roverbot (User-agent: roverbot)English
• ScrubTheWeb (User-agent: Scrubby)English
• SearchHippo (User-agent: Fluffy searchhippo.com)English
• SearchSpider.com (User-agent: SearchSpider)English
• SightQuest (User-agent: SightQuestBot)English
• SingingFish (User-agent: asterias)English
• Skymob.com (User-agent: skymob)English
• Sleek (User-agent: Sleek)English
• Slider (User-agent: Slider_Search_v1-de)English
• StackRambler (User-agent: StackRambler)English
• SurfNoMore (User-agent: Surfnomore)English
• Survey (User-agent: SurveyBot)English
• Teoma (User-agent: teomaagent teomaagent1 directhit)English
• Teradex Mapper (User-agent: Teradex)English
• The Web Moose (User-agent: webmoose)English
• Thunderstone (User-agent: T-H-U-N-D-E-R-S-T-O-N-E)English
• TopicLink (User-agent: Topiclink)English
• TutorDig (User-agent: tutorgig)English
• Unitek (User-agent: Unitek)English
• VeryGoodSearch.com (User-agent: VeryGoodSearch)English
• Walhello (User-agent: appie)English
• WebTop (User-agent: MuscatFerret)English
• WhatUSeek / Chubba (User-agent: whatUseek_winona)English
• whatUseek Winona (User-agent: whatuseek)English
• Whizbanglabs (User-agent: WhizBangLab)English
• WorldSearchCenter (User-agent: WSCbot)English
• Wotbox (User-agent: wotbox)English
• Yahoo (User-agent: YahooBot YahooSeeker Yahoo-MMCrawler)English
• Yuntis (User-agent: gulperbot)English
• Zippyfinder (User-agent: Zippy)English

• SEARCH.COM.UA crawler (User-agent: SEARCH.COM.UA)Russian
• Uaportal crawler (User-agent: Uaportal)Russian
• UASEARCH.KIEV.UA (User-agent: UASEARCH.KIEV.UA)Russian

• Altavista.UK (User-agent: AltaVista-Intranet)English
• Mirago (User-agent: HenryTheMiragoRobot)English
• Search-UK (User-agent: SearchUK MegaSheep Superewe)English
• UKSearcher (User-agent: UKSearcher)English
• Wire.UK (User-agent: WIRE)English
• YellowPet (User-agent: Yellopet-Spider)English

• VietGATE (User-agent: jcrawler)Vietnamese

• Internet Cruiser (User-agent: cruiser)English