Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extension Updating #11677

Merged
merged 62 commits into from
May 21, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
d783f00
add syntax to install a versioned extension
samansmink Jan 2, 2024
867547a
fixes to extension versioned install
samansmink Feb 2, 2024
b8657dc
duplicate header
samansmink Feb 2, 2024
d55942b
basic idea of extension updating working
samansmink Mar 6, 2024
445d9b1
extension updating + small refactor
samansmink Mar 7, 2024
308bc10
Merge branch 'main' into install-extension-version
samansmink Apr 3, 2024
22a0b33
Merge branch 'extension_metadata' into install-extension-version-merged
samansmink Apr 10, 2024
0d22196
add repository aliasing
samansmink Apr 10, 2024
c6e0337
rework extension repository mechanism slightly
samansmink Apr 10, 2024
1f0386b
more extension updating improvements
samansmink Apr 15, 2024
5059c5b
switch to dedicated opeator for update extensions
samansmink Apr 15, 2024
228ebc2
add CI test for succesful update extensions statement
samansmink Apr 15, 2024
1de4405
format
samansmink Apr 15, 2024
f00322b
some minor extension updating fixes
samansmink Apr 15, 2024
f775594
remove old test
samansmink Apr 15, 2024
2f294b1
add errors to incorrect version installation as well
samansmink Apr 16, 2024
5cd6002
format
samansmink Apr 16, 2024
6f1bde6
improve extension versioning/updating CI
samansmink Apr 16, 2024
70d416d
disable broken test for now
samansmink Apr 16, 2024
c6082bc
fix broken test
samansmink Apr 16, 2024
96e5ff8
remove old code
samansmink Apr 16, 2024
36299a8
Merge branch 'main' into install-extension-version-merged
samansmink Apr 16, 2024
96f8922
Extension API: Add optional member Version()
carlopi Apr 16, 2024
2a55929
Use ext.Version() to handle versioning info also for built-in extensions
carlopi Apr 16, 2024
c32e54e
CMake: Propagate EXT_VERSION_NAME to hold the CMake computed version …
carlopi Apr 16, 2024
ae5d3eb
In-tree extensions to be passed the versioning info
carlopi Apr 16, 2024
7099214
forward define for python
samansmink Apr 17, 2024
f8a4dac
re-enable test assertion
samansmink Apr 17, 2024
21f5e7e
small CI fixes
samansmink Apr 17, 2024
4f0088a
wip
samansmink Apr 19, 2024
f4d0597
Merge branch 'main' into install-extension-version-merged
samansmink May 10, 2024
5f12217
fix merge conflict issue
samansmink May 10, 2024
df7c860
add accidentally removed cast
samansmink May 13, 2024
4cf2ccb
Merge branch 'main' into install-extension-version-merged
samansmink May 13, 2024
4bcc31e
cleanup changes to extension updating
samansmink May 13, 2024
7d684db
fixing ci for extension update PR
samansmink May 13, 2024
99f87da
add extension version detection
samansmink May 13, 2024
7ba67d3
fix out-of-tree version-detection
samansmink May 14, 2024
5bb8b50
dont set EXT_VERSION_ define if no version is available
samansmink May 14, 2024
09a19cd
add auto-loading for extension install over httpfs
samansmink May 14, 2024
8efd4a5
format
samansmink May 14, 2024
1e8ebdd
fix install gzipped from https urls
samansmink May 15, 2024
9c26dc2
better error handling for malformed or missing info files
samansmink May 15, 2024
9422b49
small tweak to extension install write order
samansmink May 16, 2024
18d0edb
switch to explicit difference between repo and repo url
samansmink May 16, 2024
ee1c21e
add test cases for malformed and missing info files
samansmink May 16, 2024
4aefceb
add gzipped extension install to non-http install
samansmink May 16, 2024
b3c471f
add http tests as well
samansmink May 16, 2024
cc77eec
fix ci issue and fix updating over https issue
samansmink May 16, 2024
76acf25
fix extension updating job
samansmink May 16, 2024
c510994
build with tpch
samansmink May 16, 2024
8f484a5
clang tidy, speed up decompress
samansmink May 17, 2024
e0b4e89
small extension updating fixes
samansmink May 17, 2024
3ae45d6
format
samansmink May 17, 2024
1736368
several smaller fixes to extension installation and updating
samansmink May 17, 2024
cdb1c4d
catch install without force from different repo
samansmink May 17, 2024
52ebf66
fix memory leak on exceptions in listfiles
samansmink May 17, 2024
d02f157
format
samansmink May 17, 2024
0ed81d0
add no_extension_autoloading
samansmink May 18, 2024
529759a
Merge branch 'main' into install-extension-version-merged
samansmink May 19, 2024
d14607e
fix generated_extensions script to handle csv format instead of txt f…
samansmink May 19, 2024
b530912
small ci fixes
samansmink May 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup changes to extension updating
  • Loading branch information
samansmink committed May 13, 2024
commit 4bcc31e5cf6816700d2d734d4cfd7cb9463709b1
2 changes: 1 addition & 1 deletion src/include/duckdb/main/database.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ class ObjectCache;
struct AttachInfo;
class DatabaseFileSystem;

class DatabaseInstance : public std::enable_shared_from_this<DatabaseInstance> {
class DatabaseInstance : public enable_shared_from_this<DatabaseInstance> {
friend class DuckDB;

public:
Expand Down
24 changes: 10 additions & 14 deletions src/main/extension/extension_install.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -366,24 +366,20 @@ static unique_ptr<ExtensionInstallInfo> InstallFromHttpUrl(DBConfig &config, con
}

// Install an extension using a hand-rolled http request
static unique_ptr<ExtensionInstallInfo> InstallFromHttpRepository(DBConfig &config, const string &url,
const string &extension_name,
const string &repository_url, const string &temp_path,
const string &local_extension_path,
const string &version, bool force_install,
optional_ptr<HTTPLogger> http_logger) {
static unique_ptr<ExtensionInstallInfo>
InstallFromHttpRepository(DBConfig &config, const string &url, const string &extension_name,
const string &repository_url, const string &temp_path, const string &local_extension_path,
const string &version, bool force_install, optional_ptr<HTTPLogger> http_logger) {
string url_template = ExtensionHelper::ExtensionUrlTemplate(&config, repository_url, version);
string generated_url = ExtensionHelper::ExtensionFinalizeUrlTemplate(url_template, extension_name);
return InstallFromHttpUrl(config, generated_url, extension_name, repository_url, temp_path, local_extension_path,
force_install, http_logger);
}

unique_ptr<ExtensionInstallInfo> ExtensionHelper::InstallExtensionInternal(DBConfig &config, FileSystem &fs,
const string &local_path,
const string &extension, bool force_install,
const string &repository,
const string &version,
optional_ptr<HTTPLogger> http_logger) {
unique_ptr<ExtensionInstallInfo>
ExtensionHelper::InstallExtensionInternal(DBConfig &config, FileSystem &fs, const string &local_path,
const string &extension, bool force_install, const string &repository,
const string &version, optional_ptr<HTTPLogger> http_logger) {
#ifdef DUCKDB_DISABLE_EXTENSION_LOAD
throw PermissionException("Installing external extensions is disabled through a compile time flag");
#else
Expand Down Expand Up @@ -434,8 +430,8 @@ unique_ptr<ExtensionInstallInfo> ExtensionHelper::InstallExtensionInternal(DBCon

if (StringUtil::StartsWith(extension, "http://")) {
// Extension is a full http:// path
return InstallFromHttpUrl(config, extension, extension_name, "", temp_path, local_extension_path,
force_install, http_logger);
return InstallFromHttpUrl(config, extension, extension_name, "", temp_path, local_extension_path, force_install,
http_logger);
} else {
// Extension is a regular remote repository
return InstallFromHttpRepository(config, extension, extension_name, repository_url, temp_path,
Expand Down
10 changes: 6 additions & 4 deletions src/main/extension/extension_load.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -284,13 +284,14 @@ bool ExtensionHelper::TryInitialLoad(DBConfig &config, FileSystem &fs, const str

auto lowercase_extension_name = StringUtil::Lower(filebase);

// Initialize the ExtensionInitResult
result.filebase = lowercase_extension_name;
result.filename = filename;
result.lib_hdl = lib_hdl;

if (!direct_load) {
auto info_file_name = filename + ".info";

result.filebase = lowercase_extension_name;
result.filename = filename;
result.lib_hdl = lib_hdl;

// If there's an info file, we parse that for some more metadata
if (fs.FileExists(info_file_name)) {
auto file_reader = BufferedFileReader(fs, info_file_name.c_str());
Expand All @@ -307,6 +308,7 @@ bool ExtensionHelper::TryInitialLoad(DBConfig &config, FileSystem &fs, const str
result.install_info = make_uniq<ExtensionInstallInfo>();
result.install_info->mode = ExtensionInstallMode::NOT_INSTALLED;
result.install_info->full_path = filename;
result.install_info->version = parsed_metadata.extension_version;
}

return true;
Expand Down
8 changes: 4 additions & 4 deletions test/extension/update_extensions_ci.test
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ query IIII rowsort
SELECT extension_name, install_mode, installed_from, extension_version FROM duckdb_extensions() where installed and extension_name != 'jemalloc'
----
json REPOSITORY ./build/extension_metadata_test_data/repository v0.0.1
parquet (empty) (empty) (empty)
parquet STATICALLY_LINKED (empty) (empty)
tpcds CUSTOM_PATH ./build/extension_metadata_test_data/direct_install/tpcds.duckdb_extension v0.0.1
tpch REPOSITORY ./build/extension_metadata_test_data/repository v0.0.1

Expand All @@ -55,7 +55,7 @@ query IIII rowsort
SELECT extension_name, install_mode, installed_from, extension_version FROM duckdb_extensions() where installed and extension_name != 'jemalloc'
----
json REPOSITORY ./build/extension_metadata_test_data/repository v0.0.1
parquet (empty) (empty) (empty)
parquet STATICALLY_LINKED (empty) (empty)
tpcds CUSTOM_PATH ./build/extension_metadata_test_data/direct_install/tpcds.duckdb_extension v0.0.1
tpch REPOSITORY ./build/extension_metadata_test_data/repository v0.0.2

Expand All @@ -81,7 +81,7 @@ query IIII rowsort
SELECT extension_name, install_mode, installed_from, extension_version FROM duckdb_extensions() where installed and extension_name != 'jemalloc'
----
json REPOSITORY ./build/extension_metadata_test_data/repository v0.0.1
parquet (empty) (empty) (empty)
parquet STATICALLY_LINKED (empty) (empty)
tpcds CUSTOM_PATH ./build/extension_metadata_test_data/direct_install/tpcds.duckdb_extension v0.0.1
tpch REPOSITORY ./build/extension_metadata_test_data/repository v0.0.2

Expand Down Expand Up @@ -165,4 +165,4 @@ FORCE INSTALL '${DIRECT_INSTALL_DIR}/json.duckdb_extension';
statement error
LOAD json;
----
Also, the file was built for the platform 'test_platform', but we can only load extensions built for platform
Also, the file was built for the platform 'test_platform', but we can only load extensions built for platform
86 changes: 0 additions & 86 deletions test/tmp.test

This file was deleted.