Tools in mercury.graph.evidence.formats

Intention

In order to provide a framework that can seriously (i.e., handle a wikipedia dump with 25 million pages) manage sources, we need many tools. The tools in this sub-sub-module are PoC implementations that allow us to build on top of the whole idea.

Warning:

These implementations are vibe-coded. Ideally, if enough people want to build a community around this, we will have owners of different parts and make these tools as good as they should be. For now, they are just a starting point pushing the problem of file conversion to a precise place while still providing something that works well enough to be used as a PoC.

Reference

mercury.graph.evidence.formats.WikiMarkdownWriter(text)

Converts the useful MediaWiki wikitext in a Wikipedia page to Markdown.

(See Warning about the limitations of this conversion.)

Parameters:

Name	Type	Description	Default
text	str	Wikitext to convert. None is treated as an empty page.	required

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def __init__(self, text):
	self.text = text or ''

_normalize_lines(lines)

Collapses excess blank lines while retaining Markdown block separation.

Parameters:

Name	Type	Description	Default
lines	list of str	Rendered Markdown lines.	required

Returns:

Type	Description
str	Normalized Markdown ending in one newline, or an empty string.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _normalize_lines(self, lines):
	""" Collapses excess blank lines while retaining Markdown block separation.

	Args:
		lines (list of str): Rendered Markdown lines.

	Returns:
		(str): Normalized Markdown ending in one newline, or an empty string.
	"""

	result = []
	for line in lines:
		if not line.strip() and (not result or not result[-1]):
			continue

		result.append(line.rstrip())

	while result and not result[-1]:
		result.pop()

	return '\n'.join(result) + ('\n' if result else '')

_remove_templates(text)

Removes nested templates and parser functions, which do not have Markdown equivalents.

Parameters:

Name	Type	Description	Default
text	str	Wikitext that may contain {{...}} fragments.	required

Returns:

Type	Description
str	Wikitext with templates removed.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _remove_templates(self, text):
	""" Removes nested templates and parser functions, which do not have Markdown equivalents.

	Args:
		text (str): Wikitext that may contain ``{{...}}`` fragments.

	Returns:
		(str): Wikitext with templates removed.
	"""

	result = []
	depth  = 0
	pos	   = 0
	while pos < len(text):
		pair = text[pos:pos + 2]

		if pair == '{{':
			depth += 1
			pos += 2
			continue

		if pair == '}}' and depth:
			depth -= 1
			pos += 2
			continue

		if not depth:
			result.append(text[pos])

		pos += 1

	return ''.join(result)

_render_inline(text)

Converts inline MediaWiki markup to Markdown.

Parameters:

Name	Type	Description	Default
text	str	Inline wikitext.	required

Returns:

Type	Description
str	Markdown inline content.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _render_inline(self, text):
	""" Converts inline MediaWiki markup to Markdown.

	Args:
		text (str): Inline wikitext.

	Returns:
		(str): Markdown inline content.
	"""

	flags = regex.IGNORECASE

	text = regex.sub('<(?:br|br /|br/)\\s*>', '<br>', text, flags = flags)
	text = regex.sub('<(?:nowiki|pre|code)(?:\\s[^>]*)?>([\\s\\S]*?)</(?:nowiki|pre|code)\\s*>', '`\\1`', text, flags = flags)
	text = regex.sub('<[^>]+>', '', text)
	text = regex.sub('\\[\\[(?:File|Image|Category):[^\\]]+\\]\\]', '', text, flags = flags)
	text = regex.sub('\\[\\[([^\\]|]+)\\|([^\\]]+)\\]\\]', r'\\2', text)
	text = regex.sub('\\[\\[([^\\]]+)\\]\\]', r'\\1', text)
	text = regex.sub('\\[(https?://[^\\s\\]]+)\\s+([^\\]]+)\\]', r'[\\2](\\1)', text)
	text = regex.sub("'''(.*?)'''", r'**\\1**', text)
	text = regex.sub("''(.*?)''", r'*\\1*', text)

	return regex.sub('[ \\t]+', ' ', text).strip()

_render_line(line)

Converts headings, lists, links, emphasis, and HTML-like markup in one line.

Parameters:

Name	Type	Description	Default
line	str	One wikitext line.	required

Returns:

Type	Description
str	The corresponding Markdown line.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _render_line(self, line):
	""" Converts headings, lists, links, emphasis, and HTML-like markup in one line.

	Args:
		line (str): One wikitext line.

	Returns:
		(str): The corresponding Markdown line.
	"""

	match = regex.match('^\\s*(={2,6})\\s*(.*?)\\s*\\1\\s*$', line)
	if match:
		return '%s %s' % ('#' * len(match.group(1)), self._render_inline(match.group(2)))

	match = regex.match('^([*#]+)\\s*(.*)$', line)
	if match:
		marks = match.group(1)
		indent = '  ' * (len(marks) - 1)
		marker = '1.' if marks[-1] == '#' else '-'
		return '%s%s %s' % (indent, marker, self._render_inline(match.group(2)))

	match = regex.match('^;\\s*([^:]+)\\s*:\\s*(.*)$', line)
	if match:
		return '**%s:** %s' % (self._render_inline(match.group(1)), self._render_inline(match.group(2)))

	return self._render_inline(line)

_render_table(lines)

Renders the contents of one MediaWiki table block.

Parameters:

Name	Type	Description	Default
lines	list of str	Lines between {| and |}.	required

Returns:

Type	Description
list of str	A Markdown table, or readable fallback lines when no cells exist.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _render_table(self, lines):
	""" Renders the contents of one MediaWiki table block.

	Args:
		lines (list of str): Lines between ``{|`` and ``|}``.

	Returns:
		(list of str): A Markdown table, or readable fallback lines when no cells exist.
	"""

	rows	= []
	current	= []
	for line in lines:
		stripped = line.strip()
		if not stripped or stripped.startswith('|-'):
			if current:
				rows.append(current)
				current = []
			continue

		if stripped.startswith('|+'):
			continue

		if stripped.startswith('!'):
			cells = stripped[1:].split('!!')
		elif stripped.startswith('|'):
			cells = stripped[1:].split('||')
		else:
			continue

		for cell in cells:
			current.append(self._table_cell(cell))

	if current:
		rows.append(current)

	if not rows:
		return []

	width = max([len(row) for row in rows])
	for row in rows:
		row.extend([''] * (width - len(row)))

	header = rows[0]

	return ['| %s |' % ' | '.join(header), '| %s |' % ' | '.join(['---'] * width)] + ['| %s |' % ' | '.join(row) for row in rows[1:]]

_render_tables(lines)

Converts MediaWiki table blocks in lines to GitHub-flavored Markdown tables.

Parameters:

Name	Type	Description	Default
lines	list of str	Source lines.	required

Returns:

Type	Description
list of str	Lines with each complete table replaced by Markdown lines.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _render_tables(self, lines):
	""" Converts MediaWiki table blocks in lines to GitHub-flavored Markdown tables.

	Args:
		lines (list of str): Source lines.

	Returns:
		(list of str): Lines with each complete table replaced by Markdown lines.
	"""

	result = []
	pos	   = 0
	while pos < len(lines):
		if not lines[pos].lstrip().startswith('{|'):
			result.append(lines[pos])
			pos += 1
			continue

		end = pos + 1
		while end < len(lines) and not lines[end].lstrip().startswith('|}'):
			end += 1

		if end == len(lines):
			result.append(lines[pos])
			pos += 1
			continue

		result.extend(self._render_table(lines[pos + 1:end]))
		pos = end + 1

	return result

_table_cell(cell)

Removes MediaWiki cell attributes and renders one table cell.

Parameters:

Name	Type	Description	Default
cell	str	A source table cell, optionally prefixed by attributes and |.	required

Returns:

Type	Description
str	Escaped Markdown cell content.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def _table_cell(self, cell):
	""" Removes MediaWiki cell attributes and renders one table cell.

	Args:
		cell (str): A source table cell, optionally prefixed by attributes and ``|``.

	Returns:
		(str): Escaped Markdown cell content.
	"""

	if '|' in cell:
		attributes, value = cell.split('|', 1)

		if '=' in attributes or attributes.strip().startswith(('style', 'class', 'colspan', 'rowspan')):
			cell = value

	return self._render_line(cell).strip().replace('|', '\\|').replace('\n', '<br>')

render()

Returns the page wikitext as a clean Markdown document body.

Returns:

Type	Description
str	Markdown without an article title heading.

Source code in mercury/graph/evidence/formats/wiki_markdown_writer.py

def render(self):
	""" Returns the page wikitext as a clean Markdown document body.

	Returns:
		(str): Markdown without an article title heading.
	"""

	text = self.text.replace('\r\n', '\n').replace('\r', '\n')
	text = regex.sub('<!--[\\s\\S]*?-->', '', text)
	text = regex.sub('<ref(?:\\s[^>]*)?\\s*/\\s*>', '', text, flags = regex.IGNORECASE)
	text = regex.sub('<ref(?:\\s[^>]*)?>[\\s\\S]*?</ref\\s*>', '', text, flags = regex.IGNORECASE)
	text = self._remove_templates(text)

	lines = self._render_tables(text.split('\n'))
	lines = [self._render_line(line) for line in lines]

	return self._normalize_lines(lines)

mercury.graph.evidence.formats.PdfToMarkdown(src, dst, extra_args=None)

Bases: ABC

The abstract base class for converting a PDF file to Markdown.

To support every possible technology, (oss, proprietary, cloud, etc.) this is built around custom descendants of this class.

There are multiple classes supporting different technologies already implemented. Build a new Endpoint using the mge client to see the different implementations: MDbyDocling, MDbyPyMuPDF4LLM, MDbyUnlimitedOCR, ...

The simple baseline is MDbyPdfOxide and is part of the mercury-graph library (the rest are just tutorials). It is built around pdf-oxide. To use it, you need to install pdf-oxide since it is not a requirement.

Run pip install pdf-oxide to install it.

This baseline is extremely fast and good with text-based PDFs, but other converters may cover a wider range of PDFs and capture more details especially with graphic content.

To make the Source use custom PdfToMarkdown descendants

1. Write a new class just like in the .py examples you will find when creating a new Endpoint with the mge client.
2. Include it in the configuration of your Source in a similar way as this

   "pdf_to_markdown": {
           "class_name": "MDbyDocling",
           "$path": "md_by_docling.py",
           "extra_args": {"do_ocr": false}
   },
   

Using converters

The usage is done in three steps:

1. Constructing the PdfToMarkdown descendant object. This just stores the paths and extra arguments.
2. Verifying that the converter is ready by reading the ready property. There will be no conversion if this is False. This checks, files, dependencies, and server availability, whatever is needed for running the converter.
3. Calling the run() method to perform the conversion. This actually writes the output file.

Using other document formats than PDF as the source

The SourceMaker type 'pdf_mirror' can actually work with whatever document format you wish since it does not read the file. It just maps the file to a corresponding Markdown file. The PdfToMarkdown descendant does the conversion and since it is a custom class, there is no restriction on the source file format. We just use the word "PDF" out of habit since it is the lingua franca of document formats. Some of the libraries we give as examples already include support for Word, Excel, PowerPoint, and HWP/HWPX (E.g., pymupdfpro instead pymupdf4llm from https://github.com/pymupdf/pymupdf4llm) Only the destination file is format is important, which is Markdown.

Parameters:

Name	Type	Description	Default
src	str	The path to the source PDF file.	required
dst	str	The path to the destination Markdown file.	required
extra_args	dict	Any extra arguments needed for the conversion. Defaults to None.	None

Source code in mercury/graph/evidence/formats/pdf_to_markdown.py

def __init__(self, src, dst, extra_args = None):
	self.src   = os.path.abspath(src)
	self.dst   = os.path.abspath(dst)
	self.extra = extra_args

ready property

Check if the converter is ready to run.

_ready() abstractmethod

Check if the converter is ready to run, meaning: the dependencies can be imported, the server (if required) is running, and the source file exists.

Returns:

Type	Description
bool	True if the converter is ready to run, False otherwise.

Source code in mercury/graph/evidence/formats/pdf_to_markdown.py

@abstractmethod
def _ready(self):
	""" Check if the converter is ready to run, meaning: the dependencies can be imported, the server (if required) is running, and
	the source file exists.

	Returns:
		(bool): True if the converter is ready to run, False otherwise.
	"""

	pass

run() abstractmethod

Convert the PDF file to Markdown.

This is the actual file conversion method. It reads self.src and writes to self.dst. It may use self.extra for any extra arguments.

Returns:

Type	Description
bool	True if the conversion was successful, False otherwise.

Source code in mercury/graph/evidence/formats/pdf_to_markdown.py

@abstractmethod
def run(self):
	""" Convert the PDF file to Markdown.

	This is the actual file conversion method. It reads self.src and writes to self.dst. It may use self.extra for any extra arguments.

	Returns:
		(bool): True if the conversion was successful, False otherwise.
	"""

	pass