The parts of a Source

Overview

This section describes how text is structured in chunks, that are SourceEntity objects, that are part of a SourceFile, that is managed by a SourceMaker, all of which are SourceNodes that share an index tree. Everything is kept in a Source, which is an Agentic and also has a cache and a chromadb database for the chunks. We usually use the word chunk to refer to a SourceEntity that has no children, implying that it is a leaf in the tree: a short text, a link of the cell in a table.

Known limitations:

For production environments, this module is not yet ready. It works well enough to be used as a PoC. Everything is built on top of OSS components. This should be seen as a starting point that will evolve into a more robust and production-ready implementation.

mercury.graph.evidence.Source(schema, extra_args, endpoint=None, logger=None)

Bases: Agentic

The Source is an Agentic interface to a corpus of documents.

Overview

The Source can contain different types of documents and possibly call format conversion tools. Documents can be text or code.

The Source provides:

• A "chunking" interface to break documents into smaller pieces for easier processing.
• A hierarchy that divides a corpus into: a collection (that manages many files), a document (a file) a section (which can be nested).
• An indexing system that provides unique identifiers for each component.
• A persistence backend that possibly includes vectorization and embedding of the chunks for later retrieval.
• An Agentic interface to everything above.

Source Components

The Source uses the following components to manage file conversion, chunking, indexing and to represent the parts of a document:

• SourceNode: The base class to manage the index logic of all components.
• SourceMaker: The root SourceNode responsible for managing a tree of markdown files.
• SourceFile: Each individual file as a SourceNode.
• SourceEntity: Each section, subsection, paragraph, table, figure, text, table cell or link in a markdown file as a SourceNode.

Known Limitations

See

• Warning
• Limitations
• The SourceNode tree (all the SourceEntity objects inside each SourceFile, all the SourceFiles inside the SourceMaker created on demand) can become very large and the Source has yet not mechanisms to limit the size of the tree. We postpone until a later release exploring how to do that to make Sources production ready. For now, Sources typically load very fast since everything is stored in files and can be re-loaded when they grow too much.
• The vector database is configured, created by the Source, and not used. This requires exposing new capabilities by the Source. It is postponed until a later release, since it is not part of the MVP.

Parameters:

Name	Type	Description	Default
schema	str	a schema (a unique name) to use for the Source's ID.	required
endpoint	Agentic	an optional Endpoint. It becomes part of the Source's ID and is available via self.endpoint. If not provided, the Source becomes its own Endpoint.	None
logger	list	an optional logger to use for logging events. It must provide an append() method to add new events.	None
extra_args	dict	the configuration for the Source.	required

Source code in mercury/graph/evidence/source.py

def __init__(self, schema, extra_args, endpoint = None, logger = None):
	super().__init__(my_class = 'source', schema = schema, endpoint = endpoint, logger = logger)

	if schema is None:			# This allows finding out the class name (to link tools) without actually instantiating a full object.
		return

	self.states = SourceState

	self.conf = extra_args
	self.name = schema

	self._maker	 = None
	self._chroma = None

	self._meta_	 = self._meta()	# Just to make .meta reflect the initial state.

_capabilities()

Returns the capabilities of the Source.

Returns:

Type

Description

list

A list of capabilities, each represented as a dictionary with the following keys: 'type': The type of capability (e.g., 'function'). 'function': A dictionary containing details about the function The value of 'function' is: 'name': The name of the function. 'description': A brief description of what the function does. 'parameters': A dictionary with 'type', 'properties', and 'required' 'returns': A dictionary with 'type' and 'items'

Source code in mercury/graph/evidence/source.py

def _capabilities(self):
	""" Returns the capabilities of the Source.

	Returns:
		(list): A list of capabilities, each represented as a dictionary with the following keys:

			- 'type': The type of capability (e.g., 'function').
			- 'function': A dictionary containing details about the function

			The value of 'function' is:

			* 'name': The name of the function.
			* 'description': A brief description of what the function does.
			* 'parameters': A dictionary with 'type', 'properties', and 'required'
			* 'returns': A dictionary with 'type' and 'items'
	"""

	name_get_children_idx = 'children_by_idx_%s' % self.name
	name_child			  = 'object_by_idx_%s' % self.name

	self.call = {name_get_children_idx: self.get_children_idx, name_child: self.child}

	return [
		{
			'type': 'function',
			'function': {
				'name': name_get_children_idx,
				'description': 'Get indices of the children of an index. Indices are either sources, files, sections or chunks.',
				'parameters': {
					'type': 'object',
					'properties': {
						'index': {
							'type': 'string',
							'description': 'Index whose children indices are required.'
						}
					},
					'required': ['index']
				},
				'returns': {
					'type': 'array',
					'items': {
						'type': 'string'
					}
				}
			}
		},
		{
			'type': 'function',
			'function': {
				'name': name_child,
				'description': 'Get the text at a given index.',
				'parameters': {
					'type': 'object',
					'properties': {
						'index': {
							'type': 'string',
							'description': 'Index of the text component.'
						}
					},
					'required': ['index']
				},
				'returns': {
					'type': 'dict'
				}
			}
		}
	]

_dry_run(request)

Simulates running the Source with the given request.

(See Agentic.dry_run().)

NOTE:

The Endpoint takes care of validating the request according to the capabilities exposed by the Source. It is not necessary to validate again here and the Endpoint does not forward the dry_run() request to the Source. This method is provided as a requirement of the Agentic interface, but it is only used when you use Sources directly outside of an Endpoint.

Source code in mercury/graph/evidence/source.py

def _dry_run(self, request):
	""" Simulates running the Source with the given request.

	(See [`Agentic.dry_run()`][mercury.graph.evidence.Agentic.dry_run].)

	## NOTE:

	The Endpoint takes care of validating the request according to the capabilities exposed by the Source. It is not necessary to
	validate again here and the Endpoint does not forward the dry_run() request to the Source. This method is provided as a
	requirement of the Agentic interface, but it is only used when you use Sources directly outside of an Endpoint.
	"""

	return {'status': 0, 'description': 'Valid request.'}

_meta()

Returns the metadata of the Source.

(See Agentic.meta().)

Source code in mercury/graph/evidence/source.py

def _meta(self):
	""" Returns the metadata of the Source.

	(See [`Agentic.meta()`][mercury.graph.evidence.Agentic.meta].)
	"""

	meta = {}
	meta['state'] = SourceState.INITIAL.value

	meta['description'] = self.conf.get('description', '')
	if type(meta['description']) is list:
		meta['description'] = '\n'.join(meta['description'])

	meta['capabilities'] = self._capabilities()

	return meta

_run(request)

Runs the Source with the given request.

(See Agentic.run().)

Source code in mercury/graph/evidence/source.py

def _run(self, request):
	""" Runs the Source with the given request.

	(See [`Agentic.run()`][mercury.graph.evidence.Agentic.run].)
	"""

	call = self.call.get(request['name'], None)

	if call is None:
		self.log_error('Source does not have a function named "%s".' % request['name'])
		raise AgenticRunInvalidRequest

	index = request['arguments'].get('index', None)

	if index is None:
		self.log_error('Source function "%s" requires an "index" argument.' % request['name'])
		raise AgenticRunInvalidRequest

	ret = {'finish_reason': 'stop', 'message': call(index)}

	return ret

_setup_chroma_db()

Sets up the Chroma vector database for the Source.

The Chroma vector database is used to store embeddings of chunks for later retrieval. It is configured in the Source's configuration.

Returns:

Type	Description
bool	True if the Chroma vector database was set up successfully or is not used, False if setup failed.

Source code in mercury/graph/evidence/source.py

def _setup_chroma_db(self):
	""" Sets up the Chroma vector database for the Source.

	The Chroma vector database is used to store embeddings of chunks for later retrieval. It is configured in the Source's
	configuration.

	Returns:
		(bool): True if the Chroma vector database was set up successfully or is not used, False if setup failed.
	"""

	self._chroma = None

	chroma_path = self.conf.get('chroma_path', None)

	if chroma_path is None:
		return True			# This is the neat way to disable ChromaDB.

	try:
		self._chroma = chroma.PersistentClient(path = chroma_path)

	except:
		self.log_error('Source failed to create Chroma client at path: "%s".' % (chroma_path))

		return False

	try:
		name = self.conf['chroma_descriptions_collection_name']

		self._chroma_descr = self._chroma.get_or_create_collection(name)

		name = self.conf['chroma_chunks_collection_name']

		self._chroma_chunks = self._chroma.get_or_create_collection(name)

	except:
		self.log_error('Source failed to create Chroma collections at path: "%s".' % (chroma_path))

		return False

	return True

child(index)

Returns the corresponding SourceNode object following the SourceNode interface and serializes it to a dictionary.

(See SourceNode.child().)

Source code in mercury/graph/evidence/source.py

def child(self, index):
	""" Returns the corresponding SourceNode object following the SourceNode interface and serializes it to a dictionary.

	(See [`SourceNode.child()`][mercury.graph.evidence.source_parts.SourceNode.child].)
	"""

	if (   self._maker is None
		or self._maker.state != self.states.MAKER_READY_OK.value
		or self._meta_['state'] < self.states.READY.value):

		self.log_error('Source is not ready for child("%s").' % index)

		return None

	child = self._maker.child(index)

	if child is None:
		return None

	if type(child) is str:					# The SourceMaker returned the index of a SourceFile that understands the index.
		child = self._maker.child(child)
		child = child.child(index)

		if child is None:
			return None

	if type(child) is not SourceEntity:
		return {'type': 'object', 'class': str(type(child)), 'description': child.description}

	if child._children is None:
		return {'type': str(child.entity_type), 'content': child.content}

	return {'type': 'SourceEntity: %s' % child.entity_type, 'description': child.description}

close(endpoint_locked)

Closes the Source and releases any resources it holds.

(See Agentic.close().)

Source code in mercury/graph/evidence/source.py

def close(self, endpoint_locked):
	""" Closes the Source and releases any resources it holds.

	(See [`Agentic.close()`][mercury.graph.evidence.Agentic.close].)
	"""

	self._chroma = None		# There is no need to explicitly .close(), .flush() ... That persists changes.
	self._maker	 = None

get_children_idx(index=None)

Returns the children indices following the SourceNode interface.

(See SourceNode.get_children_idx().)

Source code in mercury/graph/evidence/source.py

def get_children_idx(self, index = None):
	""" Returns the children indices following the SourceNode interface.

	(See [`SourceNode.get_children_idx()`][mercury.graph.evidence.source_parts.SourceNode.get_children_idx].)
	"""

	if (   self._maker is None
		or self._maker.state != self.states.MAKER_READY_OK.value
		or self._meta_['state'] < self.states.READY.value):

		self.log_error('Source is not ready for get_children_idx("%s").' % index)

		return None

	if index is None or index == '':
		index = self._maker.index

	ret = self._maker.get_children_idx(index)

	if type(ret) is list or ret is None:		# The SourceMaker provided the children indices or an error.
		return ret

	# Now, ret is a string that is the index of the SourceFile.

	file = self._maker.child(ret)
	if type(file) is not SourceFile:
		self.log_error('SourceMaker could not find a SourceFile for index "%s".' % index)
		return None

	return file.get_children_idx(index)

pilot(intent, just_once=False)

Pilots the Source to a new state based on the given intent.

(See Agentic.pilot().)

Source code in mercury/graph/evidence/source.py

def pilot(self, intent, just_once = False):
	""" Pilots the Source to a new state based on the given intent.

	(See [`Agentic.pilot()`][mercury.graph.evidence.Agentic.pilot].)
	"""

	if self.meta['state'] < 0:
		self.log_error('Source is in error state %d' % self._meta_['state'])
		return

	while self._meta_['state'] < intent:
		if self._meta_['state'] == self.states.INITIAL.value:
			try:
				typ = self.conf['type']
				src = self.conf['src_path']
				dst = self.conf['dst_path']
				siz = self.conf.get('cluster_size', 256)
				ext = self.conf.get('extensions', None)
				pdf = self.conf.get('pdf_to_markdown', None)
				self._maker = SourceMaker(self.name, typ, src, dst, siz, ext, pdf)

			except:
				self.log_error('SourceMaker could not be created and initialized for Source "%s".' % self.name)
				self._meta_['state'] = self.states.ERR_MAKER_INIT.value
				break

			self._meta_['state'] = self.states.MAKER_INIT_OK.value

			if just_once:
				break

		if self._meta_['state'] == self.states.MAKER_INIT_OK.value:
			if self._maker.build_indices():
				self._meta_['state'] = self.states.MAKER_READY_OK.value
			else:
				self.log_error('SourceMaker could not build indices for Source "%s".' % self.name)
				self._meta_['state'] = self.states.ERR_MAKER_INDEX.value
				break

			if just_once:
				break

		if self._meta_['state'] == self.states.MAKER_READY_OK.value:
			if self._setup_chroma_db():
				self._meta_['state'] = self.states.READY.value

			else:
				self.log_error('Source could not setup the vector database for Source "%s".' % self.name)
				self._meta_['state'] = self.states.ERR_DB_SETUP.value

			break

mercury.graph.evidence.source_parts.SourceNode(index, parent=None)

Bases: ABC

Everything in the Source is a tree of SourceNodes: A SourceMaker, SourceFile or SourceEntity.

Overview

This class has the common interface to manage indices in the tree. It provides:

• Attributes: type, index, description and state
• A mechanism to locate the SourceNodes higher in the tree via the parent attribute.
• A mechanism to index its children in the tree via the get_children_idx() method.
• A mechanism to access its children in the tree via the child() method.

API:

Parameters:

Name	Type	Description	Default
index	str	the index of this SourceNode in the tree.	required
parent	SourceNode	the parent of this SourceNode in the tree.	None

Source code in mercury/graph/evidence/source_parts/source_node.py

def __init__(self, index, parent = None):
	self._index  = index
	self._parent = parent
	self._type	 = None
	self._descr	 = None

	self.state = SourceState.INITIAL

description property

Returns the description of the SourceNode. It is a string that describes each SourceNode. If there is a title or section title, it will the title with some numbering. The final SourceEntityType with no children does not have a description, The description is mechanism to provide titles, sub-titles, table names, figure names, etc. to make them searchable independently of the text.

index property

Returns the index of this SourceNode in the tree. Indices are separated by | and can include relative paths and sections, sub-sections, sub-sub-sections as different SourceNodes. E.g., "maker|a/b/file.md|sec3|sec3.1|sec3.1.2|para5"

This allows, on top of any other caching mechanisms (the Source has both a cache and a vector database with a key/value store), any SourceNode can be retrieved from its index by just parsing it left to right, starting from the SourceMaker, then the SourceFile, ... This is implemented in the Source. The SourceNode can retrieve any valid SourceNode by its index via its Agentic interface.

type property

Returns the type of this SourceNode. A type is more specific than the class name. E.g. A SourceMaker can have 'pdf_mirror', 'xml_stream', 'markdown_tree'. A SourceEntity can be anything in SourceEntityType.

child(index) abstractmethod

Returns the child of the SourceNode with the given index.

Parameters:

Name	Type	Description	Default
index	str	the index of the child to return.	required

Returns:

Type	Description
(SourceNode, str, None)	The child SourceNode with the given index. None for an invalid index. The index of the deepest SourceNode that exists in this SourceNode when the given index goes beyond the tree depth.

Source code in mercury/graph/evidence/source_parts/source_node.py

@abstractmethod
def child(self, index):
	""" Returns the child of the SourceNode with the given index.

	Args:
		index (str): the index of the child to return.

	Returns:
		(SourceNode, str, None): The child SourceNode with the given index. None for an invalid index. The index of the deepest
			SourceNode that exists in this SourceNode when the given index goes beyond the tree depth.
	"""

	pass

get_children_idx(index=None) abstractmethod

Returns the children of a SourceNode.

All SourceNodes are in a large tree of SourceNodes. Each node owns at least the node in the tree whose index is its own. Since some nodes can have a large number of children, they can divide the index tree into clusters, returning themselves as the SourceNode pointed to by the "cluster part" of the index (see example below).

Example:

A SourceMaker with id 'corpus' can hold millions of files and divide them into clusters, of say 100 files per cluster. So the final index to a file can be 'corpus|2:41|1:84|file_39.md'

Calling this method with index == 'corpus' will return [.., 'corpus|2:41', ..], calling it with index == 'corpus|2:41' will return [.., 'corpus|2:41|1:84', ..]. The next depth will return list of files in that cluster.

In the same example, calling child('corpus|2:41') will return the same SourceMaker object issuing the call. Calling child('corpus|2:41|1:84') will again return the same SourceMaker object, but calling child('corpus|2:41|1:84|file_39.md') will return the SourceFile object for that file.

About the index argument:

Not all SourceNodes should support this complexity. It makes sense for a SourceMaker since it can hold millions of files. It is used in a SourceEntity for efficiency, to avoid making copies of parts of itself that make copies of parts of themselves. The rest just ignore the index argument and return all their children in just one list.

Parameters:

Name	Type	Description	Default
index	str	An optional index to clarify which part of the SourceNode tree should be returned. (See the example above.)	None

Returns:

Type	Description
(list, str, None)	A list of children indices when the entire index belongs to the SourceNode. None for an invalid index. The index of the deepest SourceNode that exists in this SourceNode when the given index goes beyond the tree depth.

Source code in mercury/graph/evidence/source_parts/source_node.py

@abstractmethod
def get_children_idx(self, index = None):
	""" Returns the children of a SourceNode.

	All SourceNodes are in a large tree of SourceNodes. Each node owns at least the node in the tree whose index is its own.
	Since some nodes can have a large number of children, they can divide the index tree into clusters, returning themselves
	as the SourceNode pointed to by the "cluster part" of the index (see example below).

	## Example:

	A SourceMaker with id 'corpus' can hold millions of files and divide them into clusters, of say 100 files per cluster.
	So the final index to a file can be 'corpus|2:41|1:84|file_39.md'

	Calling this method with index == 'corpus' will return [.., 'corpus|2:41', ..], calling it with index == 'corpus|2:41'
	will return [.., 'corpus|2:41|1:84', ..]. The next depth will return list of files in that cluster.

	In the same example, calling `child('corpus|2:41')` will return the same SourceMaker object issuing the call. Calling
	`child('corpus|2:41|1:84')` will again return the same SourceMaker object, but calling `child('corpus|2:41|1:84|file_39.md')`
	will return the SourceFile object for that file.

	## About the index argument:

	Not all SourceNodes should support this complexity. It makes sense for a SourceMaker since it can hold millions of files.
	It is used in a SourceEntity for efficiency, to avoid making copies of parts of itself that make copies of parts of themselves.
	The rest just ignore the index argument and return all their children in just one list.

	Args:
		index (str): An optional index to clarify which part of the SourceNode tree should be returned. (See the example above.)

	Returns:
		(list, str, None): A list of children indices when the entire index belongs to the SourceNode. None for an invalid index. The
			index of the deepest SourceNode that exists in this SourceNode when the given index goes beyond the tree depth.
	"""

	pass

mercury.graph.evidence.source_parts.SourceMaker(index, typ, src_path, dst_path, cluster_size, extensions, pdf_to_md)

Bases: SourceNode

The SourceMaker is the root SourceNode that is responsible for creating a tree of SourceFile objects.

A Source only has one SourceMaker. An Endpoint can have as many Sources as required.

It may do nothing, when the Source is already a tree markdown files on disk, or it can mirror a tree of PDF files as their corresponding markdown files, or it may dump large XML files into a tree of markdown files.

Since it may potentially manage a large number of files, it will not provide the full list via get_children_idx(), but it will cluster them. A cluster defines a section in the index tree that is managed by the same SourceMaker. So, instead of having an index 'maker|file25000000.md', pointing to some SourceFile, it will have an index 'maker|@023|@19|file17.md', where '@023' and '@19' are handled by the SourceMaker. Meaning: maker.child('maker|@023') will return the same SourceMaker, maker.child('maker|@023|@19') too, but maker.child('maker|@023|@19|file17.md') will return a SourceFile.

Parameters:

Name	Type	Description	Default
index	str	the index of this SourceMaker in the tree.	required
typ	str	the type of this SourceMaker. It can be one of: "pdf_mirror", "xml_stream" or "markdown_tree".	required
src_path	str	the path to the source files. (None for "markdown_tree" type.)	required
dst_path	str	the path to the destination markdown files.	required
cluster_size	int	the maximum number of files per cluster. It is used to create clusters as explained above.	required
extensions	list of str	If given, only files with these extensions will be indexed. (A filtering mechanism for "markdown_tree".)	required
pdf_to_md	dict	If given, it is a dictionary with the configuration to load a custom PdfToMarkdown descendant.	required

Source code in mercury/graph/evidence/source_parts/source_maker.py

def __init__(self, index, typ, src_path, dst_path, cluster_size, extensions, pdf_to_md):
	super().__init__(index)

	if typ not in ['pdf_mirror', 'xml_stream', 'markdown_tree']:
		raise ValueError('Invalid type: %s' % typ)

	if typ == 'pdf_mirror':
		if pdf_to_md is None:
			self._pdf_to_md = MDbyPDFoxide
			self._pdf_extra = None
		else:
			class_name	= pdf_to_md['class_name']
			module_path	= pdf_to_md['path']
			extra_args	= pdf_to_md.get('extra_args', None)

			module_name = os.path.splitext(os.path.basename(module_path))[0]

			spec = importlib.util.spec_from_file_location(module_name, module_path)

			if spec is None or spec.loader is None:
				raise ValueError('Could not load module from path: %s' % module_path)

			module = importlib.util.module_from_spec(spec)
			spec.loader.exec_module(module)
			custom_conv = getattr(module, class_name, None)

			if custom_conv is None or not issubclass(custom_conv, PdfToMarkdown):
				raise ValueError('Invalid class: %s in module: %s' % (class_name, module_path))

			self._pdf_to_md = custom_conv
			self._pdf_extra = extra_args

	self._type = typ

	if src_path is not None:
		self._src = src_path.rstrip('/')

	self._dst	  = dst_path.rstrip('/')
	self._cl_size = cluster_size

	self._descr	= 'SourceMaker: %s, type: %s, output: %s' % (self._index, self._type, self._dst)

	# Note the ':' is %-encoded in file names by _safe_filename(), therefore it is used in cluster indices to make collisions with
	self.rex_kwap = re.compile('(^ |[<>:"/\\\\|?*\\x00-\\x1f])')	# actual file names impossible.

	self._ext = extensions
	if self._ext is None:
		return

	if type(extensions) is str:
		extensions = [extensions]

	if len(extensions) == 0:
		self._ext = None

		return

	self._ext = set()

	for e in extensions:
		e = e.lstrip('.').lower()
		self._ext.add(e)

_build_child_at(index)

Builds a SourceFile object for the given index. It is used by child() to create the SourceFile objects on demand.

Parameters:

Name	Type	Description	Default
index	str	the index of the SourceFile to create.	required

Returns:

Type	Description
SourceFile	The SourceFile object for the given index.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def _build_child_at(self, index):
	""" Builds a SourceFile object for the given index. It is used by child() to create the SourceFile objects on demand.

	Args:
		index (str): the index of the SourceFile to create.

	Returns:
		(SourceFile): The SourceFile object for the given index.
	"""

	fn_dst = '%s/%s' % (self._dst, index.split('|')[-1])

	if self._type != 'pdf_mirror':	# The file must exist in self._dst
		if os.path.isfile(fn_dst):
			return SourceFile(index, self, fn_dst)

		return SourceState.FILE_NEEDS_UPDATE.value

	fn_src = '%s/%s' % (self._src, index.split('|')[-1])
	if not os.path.isfile(fn_src):
		return SourceState.FILE_NEEDS_UPDATE.value

	if os.path.isfile(fn_dst):
		tim_src = int(os.path.getmtime(fn_src))
		tim_dst = int(os.path.getmtime(fn_dst))

		if tim_dst >= tim_src:
			return SourceFile(index, self, fn_dst)

	cnv = self._pdf_to_md(fn_src, fn_dst, self._pdf_extra)

	if cnv.ready:
		cnv.run()

		if os.path.isfile(fn_dst):
			return SourceFile(index, self, fn_dst)

	return SourceState.FILE_NEEDS_UPDATE.value

_create_markdown_from_xml()

Creates the markdown files from the source XML file. It is used by build_indices() to create the output files for the SourceMaker.

Returns:

Type	Description
bool	True if the markdown files were successfully created, False otherwise.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def _create_markdown_from_xml(self):
	""" Creates the markdown files from the source XML file. It is used by build_indices() to create the output files for the
	SourceMaker.

	Returns:
		(bool): True if the markdown files were successfully created, False otherwise.
	"""

	if not os.path.isdir(self._dst):
		os.makedirs(self._dst, exist_ok = True)

	# The file has 25,275,933 pages.

	PAGE_TAG = '%shttp://www.mediawiki.org/xml/export-0.11/}page' % '{'
	NS		 = {'mw': 'http://www.mediawiki.org/xml/export-0.11/'}

	for event, elem in etree.iterparse(self._src, events = ('end',), tag = PAGE_TAG):
		title = elem.findtext('mw:title', namespaces = NS)

		idx = self._write_xml_page_as_md(title, elem)

		elem.clear()	# Clear the element to free memory.

		while elem.getprevious() is not None:	# Also clear the previous siblings of the element to free memory.
			del elem.getparent()[0]

	return True

_recurse_tree(root=None, abort_if_before=None)

Recursively builds a dictionary of indices for the SourceMaker. It is used by build_indices() to build the indices of the SourceFile objects.

Parameters:

Name	Type	Description	Default
root	str	the path to the source files. If None, it uses self._dst.	None
abort_if_before	int	If given, it will abort the recursion returning None if it finds a file with a modification time older than this value. This is used to check if the source XML file is more recent than the output files.	None

Returns:

Type	Description
dict	A dictionary with the indices of the SourceFile objects. The keys are the indices and the values None.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def _recurse_tree(self, root = None, abort_if_before = None):
	""" Recursively builds a dictionary of indices for the SourceMaker. It is used by build_indices() to build the indices of the
	SourceFile objects.

	Args:
		root (str): the path to the source files. If None, it uses self._dst.
		abort_if_before (int): If given, it will abort the recursion returning None if it finds a file with a modification time
			older than this value. This is used to check if the source XML file is more recent than the output files.

	Returns:
		(dict): A dictionary with the indices of the SourceFile objects. The keys are the indices and the values None.
	"""

	if root is None:
		root = self._dst

	if not os.path.isdir(root):
		return None

	children = {}

	def recurse(root, folder_file_idx):
		for name in os.listdir(root):
			fn		 = '%s/%s' % (root, name)
			file_idx = '%s/%s' % (folder_file_idx, name) if folder_file_idx is not None else name

			if os.path.isdir(fn):
				if recurse(fn, file_idx) is None:
					return None

				continue

			if not os.path.isfile(fn):
				continue

			if self._ext is not None:
				ext = str(name).lower().split('.')[-1]
				if ext not in self._ext:
					continue

			if abort_if_before is not None:
				tim = os.path.getmtime(fn)
				if tim < abort_if_before:
					return None

			children[file_idx] = SourceState.FILE_NEEDS_UPDATE.value

		return True

	if recurse(root, None) is None:
		return None

	return children

_safe_filename(path, name)

This %-encodes the characters that are not allowed in file names (defined by self.rex_kwap). It also %-encodes leading spaces and removes trailing spaces. It has been tested to avoid collisions in wikipedia dumps using titles and page file names.

Parameters:

Name	Type	Description	Default
path	str	The absolute path some storage tree. (Typically self.src or self.dst, but any name without a trailing / is valid.)	required
name	str	The relative name of the file within path. Only the last part of the path is encoded, path is assumed to be valid.	required

Returns:

Type	Description
str	Absolute path to the file with a safe name.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def _safe_filename(self, path, name):
	""" This %-encodes the characters that are not allowed in file names (defined by self.rex_kwap). It also %-encodes leading spaces
	and removes trailing spaces. It has been tested to avoid collisions in wikipedia dumps using titles and page file names.

	Args:
		path (str): The absolute path some storage tree. (Typically self.src or self.dst, but any name without a trailing / is valid.)
		name (str): The relative name of the file within path. Only the last part of the path is encoded, path is assumed to be valid.

	Returns:
		(str): Absolute path to the file with a safe name.
	"""

	return '%s/%s.md' % (path, self.rex_kwap.sub(lambda m: '%%%02X' % ord(m.group()), name).rstrip(' '))

_write_xml_page_as_md(title, elem)

Writes a Wikipedia XML page as a Markdown file.

The XML export keeps the article contents as MediaWiki wikitext in the revision/text element. This method extracts that text, renders the common document structures to Markdown, and adds the article title as the level-one heading.

Parameters:

Name	Type	Description	Default
title	str	Article title from the XML page element.	required
elem	_Element	Completed MediaWiki page element.	required

Returns:

Type	Description
str	Path of the written Markdown file.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def _write_xml_page_as_md(self, title, elem):
	""" Writes a Wikipedia XML page as a Markdown file.

	The XML export keeps the article contents as MediaWiki wikitext in the
	``revision/text`` element. This method extracts that text, renders the
	common document structures to Markdown, and adds the article title as the
	level-one heading.

	Args:
		title (str): Article title from the XML page element.
		elem (lxml.etree._Element): Completed MediaWiki ``page`` element.

	Returns:
		(str): Path of the written Markdown file.
	"""

	fn = self._safe_filename(self._dst, title)

	page_text = ''
	for child in elem.iter():
		if type(child.tag) is str and etree.QName(child).localname == 'text':
			page_text = child.text or ''
			break

	body = WikiMarkdownWriter(page_text).render()

	with open(fn, 'w', encoding = 'utf-8') as f:
		f.write('# %s\n\n' % title)
		f.write(body)

	return fn

build_indices()

Builds the indices of the SourceMaker. It builds a dictionary with all the indices of the SourceFile objects, but without creating the object. (That is done on demand by child().)

There is different behavior depending on the type of the SourceMaker:

• markdown_tree: Does nothing, just exposes the files with appropriate extensions in the destination.
• pdf_mirror: Mirrors the source tree of PDF files into the destination tree of markdown files, creating the directories. It returns the indices of (non existing) markdown files via get_children_idx(). When the files is requested via child(), it is created on demand by calling the PDF to markdown conversion tool.
• xml_stream: Creates the destination tree of markdown files from the source XML file. Since it doesn't have an efficient way to access the XML file randomly, it creates all the markdown files in the destination. Those files will not be created again unless the source XML file is updated.

Returns:

Type	Description
bool	True if the indices were successfully built, False otherwise.

Source code in mercury/graph/evidence/source_parts/source_maker.py

def build_indices(self):
	""" Builds the indices of the SourceMaker. It builds a dictionary with all the indices of the SourceFile objects, but without
	creating the object. (That is done on demand by child().)

	There is different behavior depending on the type of the SourceMaker:

	  * **markdown_tree**: Does nothing, just exposes the files with appropriate extensions in the destination.
	  * **pdf_mirror**: Mirrors the source tree of PDF files into the destination tree of markdown files, creating the directories.
	  	It returns the indices of (non existing) markdown files via get_children_idx(). When the files is requested via child(), it
		is created on demand by calling the PDF to markdown conversion tool.
	  * **xml_stream**: Creates the destination tree of markdown files from the source XML file. Since it doesn't have an efficient
	  	way to access the XML file randomly, it creates all the markdown files in the destination. Those files will not be created
		again unless the source XML file is updated.

	Returns:
		(bool): True if the indices were successfully built, False otherwise.
	"""

	if self._type == 'markdown_tree':
		if not os.path.isdir(self._dst):
			return False

		self._children = self._recurse_tree()

		if type(self._children) is not dict:
			return False

	elif self._type == 'pdf_mirror':
		if not os.path.isdir(self._src):
			return False

		if not os.path.isdir(self._dst):
			os.makedirs(self._dst, exist_ok = True)

		self._children = self._recurse_tree(self._src)

		if type(self._children) is not dict:
			return False

	else:	# self._type == 'xml_stream'
		if not os.path.isfile(self._src):
			return False

		src_time = int(os.path.getmtime(self._src))

		if os.path.isdir(self._dst):
			self._children = self._recurse_tree(abort_if_before = src_time)

		if type(self._children) is not dict:	# The destination files are not up to date and need to be created again.
			if not self._create_markdown_from_xml():
				return False

			self._children = self._recurse_tree()

			if type(self._children) is not dict:
				return False

	depth = 0
	while len(self._children) > self._cl_size:
		depth += 1

		old_keys = list(self._children.keys())

		clust_num = None
		clust_items = 0
		for o_key in old_keys:
			if clust_num is None or clust_items >= self._cl_size:
				clust_num = 1 if clust_num is None else clust_num + 1
				clust_key = '%d:%d' % (depth, clust_num)

				self._children[clust_key] = {}
				clust_items = 0

			# Move the old key to the new cluster.
			self._children[clust_key][o_key] = self._children[o_key]
			del self._children[o_key]

			clust_items += 1

	self.state = SourceState.MAKER_READY_OK.value

	return True

child(index)

Returns the corresponding SourceFile object following the SourceNode interface.

(See SourceNode.child().)

Source code in mercury/graph/evidence/source_parts/source_maker.py

def child(self, index):
	""" Returns the corresponding SourceFile object following the SourceNode interface.

	(See [`SourceNode.child()`][mercury.graph.evidence.source_parts.SourceNode.child].)
	"""

	idx_stack = index.split('|')

	if idx_stack.pop(0) != self._index:
		return None

	ret = self._children
	while len(idx_stack) > 0:
		ky = idx_stack.pop(0)

		if ky not in ret:
			return None

		dic = ret
		ret = ret[ky]

		if type(ret) is not dict:
			break

	if type(ret) is dict:
		return self				# Index returns the SourceMaker itself, calling get_children_idx() will explore further down the tree.

	if len(idx_stack) > 0:		# The index is longer than the tree depth, we return the part that exists in this SourceNode.
		return '|'.join(index.split('|')[0:-len(idx_stack)])

	if type(ret) is SourceFile:
		return ret

	ret = self._build_child_at(index)

	if type(ret) is not SourceFile:
		self.state = SourceState.ERR_MAKER_ACCESS.value

	dic[ky] = ret

	return ret

get_children_idx(index=None)

Returns the children indices following the SourceNode interface.

(See SourceNode.get_children_idx().)

Source code in mercury/graph/evidence/source_parts/source_maker.py

def get_children_idx(self, index = None):
	""" Returns the children indices following the SourceNode interface.

	(See [`SourceNode.get_children_idx()`][mercury.graph.evidence.source_parts.SourceNode.get_children_idx].)
	"""

	if index is None:	# The index is mandatory for the SourceMaker.
		return None

	idx_stack = index.split('|')

	if idx_stack.pop(0) != self._index:
		return None

	ret = self._children
	while len(idx_stack) > 0:
		ky = idx_stack.pop(0)
		if ky not in ret:
			return None

		ret = ret[ky]

		if type(ret) is not dict:
			if len(idx_stack) > 0:		# The index is longer than the tree depth, we return the part that exists in this SourceNode.
				return '|'.join(index.split('|')[0:-len(idx_stack)])
			else:
				return index

	return ['%s|%s' % (index, k) for k in ret.keys()]

mercury.graph.evidence.source_parts.SourceFile(index, parent, path)

Bases: SourceNode

The SourceFile is a SourceNode represents a single markdown file in the Source. It serves the file as a tree of SourceEntity objects, one for each section, subsection, paragraph, table, figure, etc. in the markdown file.

Parameters:

Name	Type	Description	Default
index	str	the index of this SourceFile in the tree.	required
parent	SourceMaker	The SourceMaker that owns this SourceFile.	required
path	str	the path to the markdown file.	required

Source code in mercury/graph/evidence/source_parts/source_file.py

def __init__(self, index, parent, path):
	super().__init__(index, parent)

	if not os.path.isfile(path):
		raise ValueError('Invalid file path: %s' % path)

	self._type	= 'file'
	self._descr	= 'SourceFile: %s' % self._index

	self._content = None

	self.path = path

_load_and_parse()

This method has all the internal logic of the class. It starts by loading the file into memory (self._content, which is a list of str). The coordinates in terms of lines and character ranges cannot be modified. Markdown parsing is very line-oriented, so even if pathological paragraphs are found, they will live in one line and be broken by characters. Every division is either multiline with no character range or single-line with a character range. This is enforced by this method.

This method uses numpy (as np) to build integer indices to define header levels, table rows, etc. The Markdown interpretation is done by the class MarkdownParser to keep this class simple.

Hierarchy Example

HEADER_1 Title_1
    └── HEADER_2 Subtitle_1_1
        └── PARAGRAPH
            └── Chunk_1, Chunk_2, Chunk_3

Markdown has an inherent hierarchy, like in the example above. In that case, Header 1 becomes an entity with two children: Header 2 and the Title. The Title has content (the title itself) and no children. Header 2 has two children: the Subtitle and the Paragraph. This becomes:

entity	content	description	children
HEADER_1		"Title: The life of birds"	Title_1, HEADER_2
Title_1	"The life of birds"
HEADER_2		"Section 1: Overview"	Subtitle_1_1, PARAGRAPH
Subtitle_1_1	"Overview"
PARAGRAPH		"Content of 1.1"	Chunk_1, Chunk_2, Chunk_3
Chunk_1	"Bla, bla, bla"
Chunk_2	"Pio, pio, pio"
Chunk_3	"Trust me."

Note that, range-wise, HEADER_1 covers all the lines in the file from itself to the line before the next HEADER_1 (possibly the whole file), but Title_1 is only the slice of the line that contains the title. The same applies to HEADER_2, etc.

Note that all the text content in the file becomes the content of some SourceEntity, so when the Source/EvidenceGraph/etc. use it, everything is there. The descriptions are as informative as possible, using the titles of the sections to make a smaller database of descriptions possible. The numbering is created automatically by the parser.

Source code in mercury/graph/evidence/source_parts/source_file.py

def _load_and_parse(self):
	""" This method has all the internal logic of the class. It starts by loading the file into memory (self._content, which is a
	list of str). The coordinates in terms of lines and character ranges cannot be modified. Markdown parsing is very line-oriented,
	so even if pathological paragraphs are found, they will live in one line and be broken by characters. Every division is either
	multiline with no character range or single-line with a character range. This is enforced by this method.

	This method uses numpy (as np) to build integer indices to define header levels, table rows, etc. The Markdown interpretation
	is done by the class MarkdownParser to keep this class simple.

	## Hierarchy Example

	```text
	HEADER_1 Title_1
	    └── HEADER_2 Subtitle_1_1
	        └── PARAGRAPH
	            └── Chunk_1, Chunk_2, Chunk_3
	```

	Markdown has an inherent hierarchy, like in the example above. In that case, Header 1 becomes an entity with two children:
	Header 2 and the Title. The Title has content (the title itself) and no children. Header 2 has two children: the Subtitle and
	the Paragraph. This becomes:

	| entity       | content             | description                | children                  |
	| ------------ | ------------------- | -------------------------- | ------------------------- |
	| HEADER_1     |                     | "Title: The life of birds" | Title_1, HEADER_2         |
	| Title_1      | "The life of birds" |                            |                           |
	| HEADER_2     |                     | "Section 1: Overview"      | Subtitle_1_1, PARAGRAPH   |
	| Subtitle_1_1 | "Overview"          |                            |                           |
	| PARAGRAPH    |                     | "Content of 1.1"           | Chunk_1, Chunk_2, Chunk_3 |
	| Chunk_1      | "Bla, bla, bla"     |                            |                           |
	| Chunk_2      | "Pio, pio, pio"     |                            |                           |
	| Chunk_3      | "Trust me."         |                            |                           |

	Note that, range-wise, HEADER_1 covers all the lines in the file from itself to the line before the next HEADER_1 (possibly the
	whole file), but Title_1 is only the slice of the line that contains the title. The same applies to HEADER_2, etc.

	Note that all the text content in the file becomes the content of some SourceEntity, so when the Source/EvidenceGraph/etc. use it,
	everything is there. The descriptions are as informative as possible, using the titles of the sections to make a smaller database
	of descriptions possible. The numbering is created automatically by the parser.
	"""

	self._children = {}

	with open(self.path, 'r', encoding = 'utf-8') as f:
		self._content = f.read().splitlines()

	parser = MarkdownParser(self._content)
	entities = {}

	for part in parser.parse():
		parent = self if part['parent'] is None else entities[part['parent']]
		idx	   = '%s|%s' % (parent.index, part['index'])
		entity = SourceEntity(idx, self, part['ent_type'], part['line'], part['span'], part['description'])

		entities[part['index']] = entity

		if parent is self:
			self._children[entity.index] = entity
		else:
			parent.add_child(entity)

	self.state = SourceState.READY.value

child(index)

Returns the corresponding SourceEntity object following the SourceNode interface.

(See SourceNode.child().)

Source code in mercury/graph/evidence/source_parts/source_file.py

def child(self, index):
	""" Returns the corresponding SourceEntity object following the SourceNode interface.

	(See [`SourceNode.child()`][mercury.graph.evidence.source_parts.SourceNode.child].)
	"""

	if self.state != SourceState.READY.value:
		self._load_and_parse()

		if self.state != SourceState.READY.value:
			return None

	idx_stack = index.split('|')
	ky 		  = idx_stack.pop(0)
	while len(idx_stack) > 0:
		if ky == self._index:
			break

		if not self._index.startswith(ky):
			return None

		ky = '%s|%s' % (ky, idx_stack.pop(0))

	obj = self
	while len(idx_stack) > 0:
		ky	= '%s|%s' % (ky, idx_stack.pop(0))
		obj = obj._children.get(ky, None)
		if obj is None:
			return None

	return obj

get_children_idx(index=None)

Returns the children indices following the SourceNode interface.

(See SourceNode.get_children_idx().)

Source code in mercury/graph/evidence/source_parts/source_file.py

def get_children_idx(self, index = None):
	""" Returns the children indices following the SourceNode interface.

	(See [`SourceNode.get_children_idx()`][mercury.graph.evidence.source_parts.SourceNode.get_children_idx].)
	"""

	if self.state != SourceState.READY.value:
		self._load_and_parse()

		if self.state != SourceState.READY.value:
			return None

	idx_stack = index.split('|')
	ky 		  = idx_stack.pop(0)
	while len(idx_stack) > 0:
		if ky == self._index:
			break

		if not self._index.startswith(ky):
			return None

		ky = '%s|%s' % (ky, idx_stack.pop(0))

	ret = self._children
	while len(idx_stack) > 0:
		ky = '%s|%s' % (ky, idx_stack.pop(0))
		if ky not in ret:
			return None

		ret = ret[ky]._children
		if type(ret) is not dict:
			return None

	return list(ret.keys())

line_slice(line, span)

This is how every SourceEntity gets access to the text.

The SourceNode objects in a SourceFile keep only ranges, never text and call this lines() or line_slice() to get the text.

Parameters:

Name	Type	Description	Default
line	int	The line number to return.	required
span	slice	A slice object that defines the range of characters to return from the line.	required

Returns:

Type	Description
str	The characters in the given range from the specified line, or None if the range is invalid.

Source code in mercury/graph/evidence/source_parts/source_file.py

def line_slice(self, line, span):
	""" This is how every SourceEntity gets access to the text.

	The SourceNode objects in a SourceFile keep only ranges, never text and call this lines() or line_slice() to get the text.

	Args:
		line (int): The line number to return.
		span (slice): A slice object that defines the range of characters to return from the line.

	Returns:
		(str): The characters in the given range from the specified line, or None if the range is invalid.
	"""

	try:
		ret = self._content[line][span]

	except IndexError:
		return None

	return ret

lines(span)

This is how every SourceEntity gets access to the text.

The SourceNode objects in a SourceFile keep only ranges, never text and call this lines() or line_slice() to get the text.

Parameters:

Name	Type	Description	Default
span	slice	A slice object that defines the range of lines to return.	required

Returns:

Type	Description
list of str	The lines of text in the given range, or None if the range is invalid.

Source code in mercury/graph/evidence/source_parts/source_file.py

def lines(self, span):
	""" This is how every SourceEntity gets access to the text.

	The SourceNode objects in a SourceFile keep only ranges, never text and call this lines() or line_slice() to get the text.

	Args:
		span (slice): A slice object that defines the range of lines to return.

	Returns:
		(list of str): The lines of text in the given range, or None if the range is invalid.
	"""

	try:
		ret = self._content[span]

	except IndexError:
		return None

	return ret

mercury.graph.evidence.source_parts.SourceEntity(index, parent, ent_type, line, span=None, description=None)

Bases: SourceNode

The SourceEntity is a SourceNode that represents a single section, subsection, paragraph, chunk, table, cell, figure, etc. in a file. It does not contain the text itself, just its span in the SourceFile. SourceEntity objects are created by the SourceFile and live inside it

The class SourceEntityType is an Enum that defines all possible types of SourceEntity objects.

Parameters:

Name	Type	Description	Default
index	str	the index of this SourceEntity in the tree.	required
parent	SourceFile	The parent SourceFile that contains created this SourceEntity and has the content.	required
ent_type	SourceEntityType	The type of this SourceEntity.	required
line	slice or int	The line range to pass to it's parent's lines() if multiline or the only line (combined with span) for single line entities..	required
span	slice	The range of characters in the only line (line must be an int when this is used) that this SourceEntity covers.	None
description	str	An optional description of this SourceEntity. It is used for titles, sub-titles, given to it by the SourceFile.	None

Source code in mercury/graph/evidence/source_parts/source_entity.py

def __init__(self, index, parent, ent_type, line, span = None, description = None):
	super().__init__(index, parent)

	self._type	= ent_type
	self._line	= line
	self._span	= span
	self._descr = description if description is not None else ''

	self._children = None

	self.state = SourceState.READY.value

content property

Returns the content of this SourceEntity. Only SourceEntity without children have content.

entity_type property

Returns the type of this SourceEntity. It is one of the values in the SourceEntityType enumeration.

add_child(child)

Adds a child SourceEntity to this SourceEntity.

This is called by the SourceFile when parsing the markdown file and creating the SourceEntity tree.

Parameters:

Name	Type	Description	Default
child	SourceEntity	The child SourceEntity to add.	required

Source code in mercury/graph/evidence/source_parts/source_entity.py

def add_child(self, child):
	""" Adds a child SourceEntity to this SourceEntity.

	This is called by the SourceFile when parsing the markdown file and creating the SourceEntity tree.

	Args:
		child (SourceEntity): The child SourceEntity to add.
	"""

	if self._children is None:
		self._children = {}

	self._children[child.index] = child

child(index)

Returns the child of this SourceEntity with the given index.

The child can only be a deeper SourceEntity or None.

(See SourceNode.child().)

Source code in mercury/graph/evidence/source_parts/source_entity.py

def child(self, index):
	""" Returns the child of this SourceEntity with the given index.

	The child can only be a deeper SourceEntity or None.

	(See [`SourceNode.child()`][mercury.graph.evidence.source_parts.SourceNode.child].)
	"""

	if self._children is None:
		return None

	return self._children.get(index, None)

get_children_idx(index=None)

Returns the children indices of this SourceEntity. Only SourceEntity with children have children indices.

(See SourceNode.get_children_idx().)

Source code in mercury/graph/evidence/source_parts/source_entity.py

def get_children_idx(self, index = None):
	""" Returns the children indices of this SourceEntity. Only SourceEntity with children have children indices.

	(See [`SourceNode.get_children_idx()`][mercury.graph.evidence.source_parts.SourceNode.get_children_idx].)
	"""

	if self._children is not None:
		return list(self._children.keys())