Apple held talks with DuckDuckGo to replace Alphabet's Google as the default search engine for the private mode on Apple's Safari browser, the Bloomberg News reported on Wednesday, citing people familiar with the discussions.
  • loki@lemmy.mlEnglish
    2·
    9 months ago

    I can’t find any information about their search engine crawler. Isn’t it standard for search engines to label their crawlers or something?

    • darth_helmet@sh.itjust.worksEnglish
      2·
      9 months ago

      It’s conventional to do this, but a user agent string is entirely up to the client, and robots.txt is just a suggestion.

      So for the best results, you probably want to mock Google’s crawler because it’s suicidal to block that if you want search traffic.