Package xsmeral.semnet.crawler.util

Auxiliary classes used by the crawler providing, for example, robots policy handling and charset detection.

See:
          Description

Class Summary
ConnectionManager Provides consistent network settings across classes using HttpURLConnections.
CharsetDetector Provides method for detection of character set of HTML content.
RobotsPolicy Represents site crawling policy defined in Robots Exclusion Protocol for one host.
 

Package xsmeral.semnet.crawler.util Description

Auxiliary classes used by the crawler providing, for example, robots policy handling and charset detection.