Open Access Mining Guide

A skill for systematically mining open access full-text repositories to extract structured research data at scale. Covers legal frameworks for text and data mining (TDM), major open access repositories and their APIs, full-text retrieval and parsing, section-level extraction, entity recognition in scientific text, and building reproducible mining pipelines.

Legal Framework for Text and Data Mining

Rights and Regulations

Text and data mining of published literature operates within a specific legal framework that varies by jurisdiction. Understanding these rules is essential before starting any mining project.

Legal landscape for TDM:

EU Directive 2019/790 (DSM Directive):
  - Article 3: TDM exception for research organizations
    - Lawful access required (institutional subscription counts)
    - Must be for scientific research purposes
    - No opt-out possible for publishers
    - Applies to EU/EEA research institutions