Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for HuggingFace to httpfs #11831

Merged
merged 16 commits into from
May 13, 2024
Merged

Add support for HuggingFace to httpfs #11831

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
b518b46
initial hffs implementation
samansmink Apr 9, 2024
4d34a3c
further implement hffs
samansmink Apr 9, 2024
7d56c73
add tests, fix authentication
samansmink Apr 24, 2024
03cc46e
format
samansmink Apr 24, 2024
66a6dcc
add autoload support for hffs, some cleanup
samansmink Apr 25, 2024
f9f8e36
fix pagination, add tests for it. Add retry mechanism
samansmink Apr 25, 2024
dedd294
format
samansmink Apr 25, 2024
d43f9d2
fix incorrect forward declare
samansmink Apr 25, 2024
f27d4d1
remove s in comment
samansmink Apr 25, 2024
280f91e
improve huggingface secret provider, more testing
samansmink May 1, 2024
522179e
fix hf secret test
samansmink May 1, 2024
db624d6
allow spaces urls, add tests, fix ci
samansmink May 2, 2024
b4d4a2f
update error message and tests
samansmink May 2, 2024
802dd4c
format
samansmink May 2, 2024
c0a9ea0
Merge branch 'main' into hugging-face-fs
samansmink May 6, 2024
f73854b
fix hffs issue from merge
samansmink May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
allow spaces urls, add tests, fix ci
  • Loading branch information
@samansmink
samansmink committed May 2, 2024
commit db624d6cbf5d7a379d26d9657fd623f160e4e73d
2 changes: 1 addition & 1 deletion extension/httpfs/hffs.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@ ParsedHFUrl HuggingFaceFileSystem::HFUrlParse(const string &url) {
ThrowParseError(url);
}
result.repo_type = url.substr(last_delim, curr_delim - last_delim);
samansmink marked this conversation as resolved.
Show resolved Hide resolved
if (result.repo_type != "datasets") {
if (result.repo_type != "datasets" && result.repo_type != "spaces") {
throw IOException("Failed to parse: '%s'. Currently DuckDB only supports querying datasets, so the url should "
"start with 'hf://datasets'",
samansmink marked this conversation as resolved.
Show resolved Hide resolved
url);
Expand Down
1 change: 1 addition & 0 deletions src/include/duckdb/main/extension_entries.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ static constexpr ExtensionEntry EXTENSION_SETTINGS[] = {
{"calendar", "icu"},
{"enable_server_cert_verification", "httpfs"},
{"force_download", "httpfs"},
{"hf_max_per_page", "httpfs"},
{"http_keep_alive", "httpfs"},
{"http_retries", "httpfs"},
{"http_retry_backoff", "httpfs"},
Expand Down
6 changes: 6 additions & 0 deletions test/sql/httpfs/hffs.test_slow
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,12 @@ FROM parquet_scan('hf://datasets/samansmink/duckdb_ci_private/hive_data/**/*.par
----
401

# Ensure spaces work too
query I
select size from read_text('hf://spaces/samansmink/duckdb_ci_tests/README.md');
----
199

# FIXME: push auth key into CI for this to ensure it is tested in CI properly
require-env HUGGING_FACE_TOKEN

Expand Down
Loading