Python: module Cr0wl3r

Cr0wl3r (version 1.0.1)

Modules

logging

Classes



abc.ABC(builtins.object)

_Crawler

CrawlerRawPrinter
CrawlerRawUrlOnlyPrinter

builtins.Exception(builtins.BaseException)

CriticalUrllibError

builtins.ValueError(builtins.Exception)

ContentTypeError

class ContentTypeError(builtins.ValueError)

    Exception raised when the Content-Type is not valid for web page.

Method resolution order:

ContentTypeError

builtins.ValueError

builtins.Exception

builtins.BaseException

builtins.object

Data descriptors defined here:

__weakref__

list of weak references to the object (if defined)

Methods inherited from builtins.ValueError:

__init__(self, /, *args, **kwargs)
Initialize self.  See help(type(self)) for accurate signature.

Static methods inherited from builtins.ValueError:

__new__(*args, **kwargs) from builtins.type
Create and return a new object.  See help(type) for accurate signature.

Methods inherited from builtins.BaseException:

__delattr__(self, name, /)
Implement delattr(self, name).

__getattribute__(self, name, /)
Return getattr(self, name).

__reduce__(...)
Helper for pickle.

__repr__(self, /)
Return repr(self).

__setattr__(self, name, value, /)
Implement setattr(self, name, value).

__setstate__(...)

__str__(self, /)
Return str(self).

with_traceback(...)
Exception.with_traceback(tb) -- set self.__traceback__ to tb and return self.

Data descriptors inherited from builtins.BaseException:

__cause__

exception cause

__context__

exception context

__dict__

__suppress_context__

__traceback__

args

class CrawlerRawPrinter(_Crawler)

    CrawlerRawPrinter(url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None) This class prints all URLs.

Method resolution order:

CrawlerRawPrinter

_Crawler

abc.ABC

builtins.object

Methods defined here:

handle_resource(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

handle_static(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

handle_web_page(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

Data and other attributes defined here:

__abstractmethods__ = frozenset()

Methods inherited from _Crawler:

__init__(self, url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None)
Initialize self.  See help(type(self)) for accurate signature.

add(self, parsed_url: urllib.parse.ParseResult, url: str, request: bool) -> None
This method adds an URL if not already added.

add_domain_new_url(self, url: str) -> None
This function add new url to parse.

ask_crossdomain(self, url: urllib.parse.ParseResult) -> None
This method send request for crossdomain.xml

ask_robots(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

ask_sitemap(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

crawl(self) -> None
This function starts crawler.

decode(self, data: bytes) -> str
This method try to decode data using utf-8 and latin-1.

get_complete_url(self, url: urllib.parse.ParseResult) -> Tuple[str, urllib.parse.ParseResult]
This function build a complete url.

get_data(self, url_parsed: urllib.parse.ParseResult, first: bool) -> Tuple[Cr0wl3r.HttpResponse, str]
This function sends the HTTP request and returns the HTTP response.

get_local_filepath(self, url: urllib.parse.ParseResult) -> str
This method returns the local file path to store the response content.

get_parse_html(self, urls: List[str], urls_done: Set[str], first: bool) -> urllib.parse.ParseResult
This function gets HTML data and parses it.

get_url_and_data(self, urls: List[str], urls_done: Set[str], first: bool) -> Tuple[http.client.HTTPResponse, urllib.parse.ParseResult, str]
This method gets url, parses it and gets data from it.

parse_robots_line(self, from_url: str, line: str, type_: str) -> None
This method parses a robots.txt line.

parse_sitemap(self, url: Union[str, urllib.parse.ParseResult]) -> bool
This method requests a sitemap URL and/or parse it.

parse_sitemap_text(self, data: http.client.HTTPResponse) -> None
This method parses TEXT sitemaps.

parse_sitemap_xml(self, data: http.client.HTTPResponse) -> None
This method parses XML sitemaps.

prepare_request(self, url_parsed: urllib.parse.ParseResult) -> Tuple[Cr0wl3r.UrlReport, str, bool]
This method prepares the request and response parsing.

read_and_parse(self, response: http.client.HTTPResponse, url: str, first: bool) -> None
This method reads the response and parses it.

reader(self, response: http.client.HTTPResponse) -> bytes
This method excepts the IncompleteRead error.

write_error(self, url: urllib.parse.ParseResult, error: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> None
This method writes the response content on HTTP error.

write_response(self, url: urllib.parse.ParseResult, response: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> Optional[http.client.HTTPResponse]
This method writes the response content for a URL.

Data descriptors inherited from _Crawler:

__dict__

dictionary for instance variables (if defined)

__weakref__

list of weak references to the object (if defined)

class CrawlerRawUrlOnlyPrinter(_Crawler)

    CrawlerRawUrlOnlyPrinter(url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None) This class prints all URLs.

Method resolution order:

CrawlerRawUrlOnlyPrinter

_Crawler

abc.ABC

builtins.object

Methods defined here:

handle_resource(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

handle_static(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

handle_web_page(self, from_url: str, url: str, tag: str, attribute: str) -> Optional[bool]
This method prints the URL found.

Data and other attributes defined here:

__abstractmethods__ = frozenset()

Methods inherited from _Crawler:

__init__(self, url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None)
Initialize self.  See help(type(self)) for accurate signature.

add(self, parsed_url: urllib.parse.ParseResult, url: str, request: bool) -> None
This method adds an URL if not already added.

add_domain_new_url(self, url: str) -> None
This function add new url to parse.

ask_crossdomain(self, url: urllib.parse.ParseResult) -> None
This method send request for crossdomain.xml

ask_robots(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

ask_sitemap(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

crawl(self) -> None
This function starts crawler.

decode(self, data: bytes) -> str
This method try to decode data using utf-8 and latin-1.

get_complete_url(self, url: urllib.parse.ParseResult) -> Tuple[str, urllib.parse.ParseResult]
This function build a complete url.

get_data(self, url_parsed: urllib.parse.ParseResult, first: bool) -> Tuple[Cr0wl3r.HttpResponse, str]
This function sends the HTTP request and returns the HTTP response.

get_local_filepath(self, url: urllib.parse.ParseResult) -> str
This method returns the local file path to store the response content.

get_parse_html(self, urls: List[str], urls_done: Set[str], first: bool) -> urllib.parse.ParseResult
This function gets HTML data and parses it.

get_url_and_data(self, urls: List[str], urls_done: Set[str], first: bool) -> Tuple[http.client.HTTPResponse, urllib.parse.ParseResult, str]
This method gets url, parses it and gets data from it.

parse_robots_line(self, from_url: str, line: str, type_: str) -> None
This method parses a robots.txt line.

parse_sitemap(self, url: Union[str, urllib.parse.ParseResult]) -> bool
This method requests a sitemap URL and/or parse it.

parse_sitemap_text(self, data: http.client.HTTPResponse) -> None
This method parses TEXT sitemaps.

parse_sitemap_xml(self, data: http.client.HTTPResponse) -> None
This method parses XML sitemaps.

prepare_request(self, url_parsed: urllib.parse.ParseResult) -> Tuple[Cr0wl3r.UrlReport, str, bool]
This method prepares the request and response parsing.

read_and_parse(self, response: http.client.HTTPResponse, url: str, first: bool) -> None
This method reads the response and parses it.

reader(self, response: http.client.HTTPResponse) -> bytes
This method excepts the IncompleteRead error.

write_error(self, url: urllib.parse.ParseResult, error: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> None
This method writes the response content on HTTP error.

write_response(self, url: urllib.parse.ParseResult, response: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> Optional[http.client.HTTPResponse]
This method writes the response content for a URL.

Data descriptors inherited from _Crawler:

__dict__

dictionary for instance variables (if defined)

__weakref__

list of weak references to the object (if defined)

class CriticalUrllibError(builtins.Exception)

    Exception raised after urllib exception when is critical.

Method resolution order:

CriticalUrllibError

builtins.Exception

builtins.BaseException

builtins.object

Data descriptors defined here:

__weakref__

list of weak references to the object (if defined)

Methods inherited from builtins.Exception:

__init__(self, /, *args, **kwargs)
Initialize self.  See help(type(self)) for accurate signature.

Static methods inherited from builtins.Exception:

__new__(*args, **kwargs) from builtins.type
Create and return a new object.  See help(type) for accurate signature.

Methods inherited from builtins.BaseException:

__delattr__(self, name, /)
Implement delattr(self, name).

__getattribute__(self, name, /)
Return getattr(self, name).

__reduce__(...)
Helper for pickle.

__repr__(self, /)
Return repr(self).

__setattr__(self, name, value, /)
Implement setattr(self, name, value).

__setstate__(...)

__str__(self, /)
Return str(self).

with_traceback(...)
Exception.with_traceback(tb) -- set self.__traceback__ to tb and return self.

Data descriptors inherited from builtins.BaseException:

__cause__

exception cause

__context__

exception context

__dict__

__suppress_context__

__traceback__

args

class _Crawler(abc.ABC)

    _Crawler(url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None) This class crawls a web page to get URLs and resources paths.

Method resolution order:

_Crawler

abc.ABC

builtins.object

Methods defined here:

__init__(self, url: str, recursive: bool = True, update: bool = False, max_request: int = None, only_domain: bool = True, headers: Dict[str, str] = {}, robots: bool = True, sitemap: bool = True, crossdomain: bool = True, context: ssl.SSLContext = <ssl.SSLContext object at 0x0000017B7FD7FA40>, interval: float = 0, download_policy: str = None)
Initialize self.  See help(type(self)) for accurate signature.

add(self, parsed_url: urllib.parse.ParseResult, url: str, request: bool) -> None
This method adds an URL if not already added.

add_domain_new_url(self, url: str) -> None
This function add new url to parse.

ask_crossdomain(self, url: urllib.parse.ParseResult) -> None
This method send request for crossdomain.xml

ask_robots(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

ask_sitemap(self, url: urllib.parse.ParseResult) -> None
This method send request for robots.txt.

crawl(self) -> None
This function starts crawler.

decode(self, data: bytes) -> str
This method try to decode data using utf-8 and latin-1.

get_complete_url(self, url: urllib.parse.ParseResult) -> Tuple[str, urllib.parse.ParseResult]
This function build a complete url.

get_data(self, url_parsed: urllib.parse.ParseResult, first: bool) -> Tuple[Cr0wl3r.HttpResponse, str]
This function sends the HTTP request and returns the HTTP response.

get_local_filepath(self, url: urllib.parse.ParseResult) -> str
This method returns the local file path to store the response content.

get_parse_html(self, urls: List[str], urls_done: Set[str], first: bool) -> urllib.parse.ParseResult
This function gets HTML data and parses it.

get_url_and_data(self, urls: List[str], urls_done: Set[str], first: bool) -> Tuple[http.client.HTTPResponse, urllib.parse.ParseResult, str]
This method gets url, parses it and gets data from it.

handle_resource(self, url: str, tag: str, attribute: str) -> Optional[bool]

handle_static(self, url: str, tag: str, attribute: str) -> Optional[bool]

handle_web_page(self, url: str, tag: str, attribute: str) -> Optional[bool]

parse_robots_line(self, from_url: str, line: str, type_: str) -> None
This method parses a robots.txt line.

parse_sitemap(self, url: Union[str, urllib.parse.ParseResult]) -> bool
This method requests a sitemap URL and/or parse it.

parse_sitemap_text(self, data: http.client.HTTPResponse) -> None
This method parses TEXT sitemaps.

parse_sitemap_xml(self, data: http.client.HTTPResponse) -> None
This method parses XML sitemaps.

prepare_request(self, url_parsed: urllib.parse.ParseResult) -> Tuple[Cr0wl3r.UrlReport, str, bool]
This method prepares the request and response parsing.

read_and_parse(self, response: http.client.HTTPResponse, url: str, first: bool) -> None
This method reads the response and parses it.

reader(self, response: http.client.HTTPResponse) -> bytes
This method excepts the IncompleteRead error.

write_error(self, url: urllib.parse.ParseResult, error: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> None
This method writes the response content on HTTP error.

write_response(self, url: urllib.parse.ParseResult, response: http.client.HTTPResponse, report: Cr0wl3r.UrlReport) -> Optional[http.client.HTTPResponse]
This method writes the response content for a URL.

Data descriptors defined here:

__dict__

dictionary for instance variables (if defined)

__weakref__

list of weak references to the object (if defined)

Data and other attributes defined here:

__abstractmethods__ = frozenset({'handle_resource', 'handle_static', 'handle_web_page'})

Functions

main() -> int
This function starts the script from the command line.

Data

__all__ = ['_Crawler', 'CrawlerRawPrinter', 'CrawlerRawUrlOnlyPrinter', 'CrawlerColoredPrinter', 'main', 'CriticalUrllibError', 'ContentTypeError']
__annotations__ = {'extensions': typing.FrozenSet[str], 'format_output': <class 'bool'>, 'html_url_tags_attributes': typing.FrozenSet[typing.Tuple[str]], 'input_types': typing.Dict[str, int], 'interaction_tags': typing.Dict[str, int], 'reports': typing.Dict[str, Cr0wl3r.UrlReport], 'static_url_tags_attributes': typing.FrozenSet[typing.Tuple[str]], 'url_attributes': typing.FrozenSet[str]}
__author_email__ = 'mauricelambert434@gmail.com'
__copyright__ = '\nCr0wl3r Copyright (C) 2023 Maurice Lambert\nTh...ome to redistribute it\nunder certain conditions.\n'
__description__ = '\nThis module implements a crawler to find all links and resources\non the target web site.\n'
__license__ = 'GPL-3.0 License'
__maintainer__ = 'Maurice Lambert'
__maintainer_email__ = 'mauricelambert434@gmail.com'
__url__ = 'https://github.com/mauricelambert/Cr0wl3r'

Author

Maurice Lambert

Functions
		main() -> int `This function starts the script from the command line.`

Data
		__all__ = ['_Crawler', 'CrawlerRawPrinter', 'CrawlerRawUrlOnlyPrinter', 'CrawlerColoredPrinter', 'main', 'CriticalUrllibError', 'ContentTypeError'] __annotations__ = {'extensions': typing.FrozenSet[str], 'format_output': <class 'bool'>, 'html_url_tags_attributes': typing.FrozenSet[typing.Tuple[str]], 'input_types': typing.Dict[str, int], 'interaction_tags': typing.Dict[str, int], 'reports': typing.Dict[str, Cr0wl3r.UrlReport], 'static_url_tags_attributes': typing.FrozenSet[typing.Tuple[str]], 'url_attributes': typing.FrozenSet[str]} __author_email__ = 'mauricelambert434@gmail.com' __copyright__ = '\nCr0wl3r Copyright (C) 2023 Maurice Lambert\nTh...ome to redistribute it\nunder certain conditions.\n' __description__ = '\nThis module implements a crawler to find all links and resources\non the target web site.\n' __license__ = 'GPL-3.0 License' __maintainer__ = 'Maurice Lambert' __maintainer_email__ = 'mauricelambert434@gmail.com' __url__ = 'https://github.com/mauricelambert/Cr0wl3r'