Skip to content

Hiyori CLI for maintaining the relational Anime & Manga Database — HiyoriDB

Notifications You must be signed in to change notification settings

Hiyori-API/hiyori-cli

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

56 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hiyori CLI

stable Discord Server

Hiyori CLI is being developed to create and maintain HiyoriDB — a relational Anime and Manga Database.

Warning

This project is a work-in-progress and is not production ready.

Process

Hiyori CLI has two main primary functions:

  1. Ingesting transformed data based on the Hiyori Schema ingested from available sources
  2. Combining the ingested data into HiyoriDB

Hiyori also collects relational mappings from other sources (if available) and uses that to populate references in the schema. Supported references have parsers and are able to extract the ID of the entry for that source and store them in reference_ids.

WIP. 🚧


Installation

  1. Install PHP8.2+, MongoDB, MongoDB PHP Driver, Composer
  2. git clone https://github.com/Hiyori-API/hiyori-cli.git
  3. cd hiyori-cli && composer install
  4. chmod +x hiyori
  5. Run commands

Ingestion

You can use the cli tool as ./hiyori, php hiyori or php src/run.php.

Note

It's recommended to keep a 1-second delay between requests for sources to prevent rate-limiting.

MyAnimeList Ingestion

php hiyori ingest myanimelist --delay 1

Kitsu Ingestion

php hiyori ingest kitsu --delay 1

AniList Ingestion

php hiyori ingest anilist --delay 1

Note

General spoiler or Media spoiler tags are not ingested.


Combiner

Note

Run the combiner after the ingestion of multiple sources are complete.

php hiyori combine {base} --strategy {strategy}
  • {base} the name of the source to dataset to use
  • There are multiple combining strategies available. The default one is relational_mapping.
// example
php hiyori combine myanimelist
// specify a stratgegy
php hiyori combine myanimelist --strategy relational_mapping

Available Combining Strategies

Relational Mapping --strategy relational_mapping

This strategy simply checks for matching source IDs across available source datasets and combines values based off of those.

Relational Fuzzy --strategy relational_fuzzy

This strategy cross-checks all reference_ids across available source datasets in hopes to find a matching ID between sources.

Metadata Fuzzy

Warning

This is not yet implemented.

This strategy compares other metadata across available source datasets. The metadata properties are given weights and are cross checked using various algorithms. Based on this, a score is given. If the score is above a certain threshold, it is merged.


Hiyori Schema

Common data with different representations like status (MAL: "Finished Airing", Anilist: "Finished") are transformed into a common value via Hiyori's Enums.

Anime

Property Data Type Remarks Nullable
title String Entry's main title
synonyms Array of String All other titles (combined from sources)
type String Entry type [tv, movie, ova, ona, special, music]
episodes Integer Number of episodes (if any mentioned)
status String Status of entry [finished, airing, to_be_aired, cancelled]
season Nullable String Release Season [winter, summer, spring, fall]
year Nullable Integer Release Year
images Array of String Default Image URLs (combined from sources)
reference_ids Object Parsed IDs from Supported References
references Array of String Reference URLs (combined from sources)
tags Array of String Genres/Tags (combined from sources)

Manga

TBD.


Supported References

The following URL type, if detected in an entry's references, will be parsed.

Reference Shorthand (as returned)
Amazon Prime ap
AniDB adb
AniList al
AnimeNewsNetwork ann
Bangumi b
Crunchyroll cr
Funimation fm
Kitsu k
MyAnimeList mal
Netflix nf
Syoboi sb
Trakt t
TubiTV ttv
TVTokyo tvt
Twitter tx
Wikipedia English wpen
Wikipedia Japanese wpjp
YouTube yt

Roadmap

Right now the main focus is to integrate the initial 3 sources and build a combined relational Anime metadata DB.

Quality of Life

  • Tests
  • Sweepers - Remove dead entries (or create a separate collection of them)

Feature

Sources

  • MyAnimeList Integration via REST API
  • Kitsu Integration via REST API
  • AniList Integration via REST API

FAQ

Why does Hiyori Schema consist of metadata?

In the event an entry from a source does not contain relational IDs or mapping provided by sources, the metadata will be used to find the relation instead.

Important

Hiyori is not affiliated with any sources. You are responsible for the usage of this tool. Please be respectful towards the terms and conditions set by these sources.