Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extension Updating #11677

Merged
merged 62 commits into from
May 21, 2024
Merged
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
d783f00
add syntax to install a versioned extension
samansmink Jan 2, 2024
867547a
fixes to extension versioned install
samansmink Feb 2, 2024
b8657dc
duplicate header
samansmink Feb 2, 2024
d55942b
basic idea of extension updating working
samansmink Mar 6, 2024
445d9b1
extension updating + small refactor
samansmink Mar 7, 2024
308bc10
Merge branch 'main' into install-extension-version
samansmink Apr 3, 2024
22a0b33
Merge branch 'extension_metadata' into install-extension-version-merged
samansmink Apr 10, 2024
0d22196
add repository aliasing
samansmink Apr 10, 2024
c6e0337
rework extension repository mechanism slightly
samansmink Apr 10, 2024
1f0386b
more extension updating improvements
samansmink Apr 15, 2024
5059c5b
switch to dedicated opeator for update extensions
samansmink Apr 15, 2024
228ebc2
add CI test for succesful update extensions statement
samansmink Apr 15, 2024
1de4405
format
samansmink Apr 15, 2024
f00322b
some minor extension updating fixes
samansmink Apr 15, 2024
f775594
remove old test
samansmink Apr 15, 2024
2f294b1
add errors to incorrect version installation as well
samansmink Apr 16, 2024
5cd6002
format
samansmink Apr 16, 2024
6f1bde6
improve extension versioning/updating CI
samansmink Apr 16, 2024
70d416d
disable broken test for now
samansmink Apr 16, 2024
c6082bc
fix broken test
samansmink Apr 16, 2024
96e5ff8
remove old code
samansmink Apr 16, 2024
36299a8
Merge branch 'main' into install-extension-version-merged
samansmink Apr 16, 2024
96f8922
Extension API: Add optional member Version()
carlopi Apr 16, 2024
2a55929
Use ext.Version() to handle versioning info also for built-in extensions
carlopi Apr 16, 2024
c32e54e
CMake: Propagate EXT_VERSION_NAME to hold the CMake computed version …
carlopi Apr 16, 2024
ae5d3eb
In-tree extensions to be passed the versioning info
carlopi Apr 16, 2024
7099214
forward define for python
samansmink Apr 17, 2024
f8a4dac
re-enable test assertion
samansmink Apr 17, 2024
21f5e7e
small CI fixes
samansmink Apr 17, 2024
4f0088a
wip
samansmink Apr 19, 2024
f4d0597
Merge branch 'main' into install-extension-version-merged
samansmink May 10, 2024
5f12217
fix merge conflict issue
samansmink May 10, 2024
df7c860
add accidentally removed cast
samansmink May 13, 2024
4cf2ccb
Merge branch 'main' into install-extension-version-merged
samansmink May 13, 2024
4bcc31e
cleanup changes to extension updating
samansmink May 13, 2024
7d684db
fixing ci for extension update PR
samansmink May 13, 2024
99f87da
add extension version detection
samansmink May 13, 2024
7ba67d3
fix out-of-tree version-detection
samansmink May 14, 2024
5bb8b50
dont set EXT_VERSION_ define if no version is available
samansmink May 14, 2024
09a19cd
add auto-loading for extension install over httpfs
samansmink May 14, 2024
8efd4a5
format
samansmink May 14, 2024
1e8ebdd
fix install gzipped from https urls
samansmink May 15, 2024
9c26dc2
better error handling for malformed or missing info files
samansmink May 15, 2024
9422b49
small tweak to extension install write order
samansmink May 16, 2024
18d0edb
switch to explicit difference between repo and repo url
samansmink May 16, 2024
ee1c21e
add test cases for malformed and missing info files
samansmink May 16, 2024
4aefceb
add gzipped extension install to non-http install
samansmink May 16, 2024
b3c471f
add http tests as well
samansmink May 16, 2024
cc77eec
fix ci issue and fix updating over https issue
samansmink May 16, 2024
76acf25
fix extension updating job
samansmink May 16, 2024
c510994
build with tpch
samansmink May 16, 2024
8f484a5
clang tidy, speed up decompress
samansmink May 17, 2024
e0b4e89
small extension updating fixes
samansmink May 17, 2024
3ae45d6
format
samansmink May 17, 2024
1736368
several smaller fixes to extension installation and updating
samansmink May 17, 2024
cdb1c4d
catch install without force from different repo
samansmink May 17, 2024
52ebf66
fix memory leak on exceptions in listfiles
samansmink May 17, 2024
d02f157
format
samansmink May 17, 2024
0ed81d0
add no_extension_autoloading
samansmink May 18, 2024
529759a
Merge branch 'main' into install-extension-version-merged
samansmink May 19, 2024
d14607e
fix generated_extensions script to handle csv format instead of txt f…
samansmink May 19, 2024
b530912
small ci fixes
samansmink May 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
format
  • Loading branch information
samansmink committed May 14, 2024
commit 8efd4a5ecbb8081f43a3b1f7e93478966a0afb9a
53 changes: 29 additions & 24 deletions src/main/extension/extension_install.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -299,14 +299,16 @@ static unique_ptr<ExtensionInstallInfo> DirectInstallExtension(DBConfig &config,
}

// Throw error on failure
if (!exists) {
if (!fs.IsRemoteFile(file)) {
throw IOException("Failed to copy local extension \"%s\" at PATH \"%s\"\n", extension_name, file);
}
if (StringUtil::StartsWith(file, "https://")) {
throw IOException("Failed to copy remote extension \"%s\" at PATH \"%s\"\n Please install the 'httpfs' extension, then try again.", extension_name, file);
}
}
if (!exists) {
if (!fs.IsRemoteFile(file)) {
throw IOException("Failed to copy local extension \"%s\" at PATH \"%s\"\n", extension_name, file);
}
if (StringUtil::StartsWith(file, "https://")) {
throw IOException("Failed to copy remote extension \"%s\" at PATH \"%s\"\n Please install the 'httpfs' "
"extension, then try again.",
extension_name, file);
}
}

idx_t file_size;
auto in_buffer = ReadExtensionFileFromDisk(fs, file, file_size);
Expand Down Expand Up @@ -387,19 +389,20 @@ static unique_ptr<ExtensionInstallInfo> InstallFromHttpUrl(DBConfig &config, con
}

// Install an extension using a hand-rolled http request
static unique_ptr<ExtensionInstallInfo>
InstallFromRepository(DBConfig &config, FileSystem &fs, const string &url, const string &extension_name,
const string &repository_url, const string &temp_path, const string &local_extension_path,
const string &version, bool force_install, optional_ptr<HTTPLogger> http_logger,
optional_ptr<ClientContext> context) {
static unique_ptr<ExtensionInstallInfo> InstallFromRepository(DBConfig &config, FileSystem &fs, const string &url,
const string &extension_name,
const string &repository_url, const string &temp_path,
const string &local_extension_path, const string &version,
bool force_install, optional_ptr<HTTPLogger> http_logger,
optional_ptr<ClientContext> context) {
string url_template = ExtensionHelper::ExtensionUrlTemplate(&config, repository_url, version);
string generated_url = ExtensionHelper::ExtensionFinalizeUrlTemplate(url_template, extension_name);

// Special handling for http repository: avoid using regular filesystem (note: the filesystem is not used here)
if (StringUtil::StartsWith(repository_url, "http://")) {
return InstallFromHttpUrl(config, generated_url, extension_name, repository_url, temp_path, local_extension_path,
force_install, http_logger);
}
return InstallFromHttpUrl(config, generated_url, extension_name, repository_url, temp_path,
local_extension_path, force_install, http_logger);
}

// Default case, let the FileSystem figure it out
return DirectInstallExtension(config, fs, generated_url, temp_path, extension_name, local_extension_path,
Expand Down Expand Up @@ -436,7 +439,8 @@ ExtensionHelper::InstallExtensionInternal(DBConfig &config, FileSystem &fs, cons
// Install extension from local, direct url
if (ExtensionHelper::IsFullPath(extension) && !FileSystem::IsRemoteFile(extension)) {
samansmink marked this conversation as resolved.
Show resolved Hide resolved

return DirectInstallExtension(config, fs, extension, temp_path, extension, local_extension_path, force_install, "", context);
return DirectInstallExtension(config, fs, extension, temp_path, extension, local_extension_path, force_install,
samansmink marked this conversation as resolved.
Show resolved Hide resolved
"", context);
}

// Install extension from local url based on a repository (Note that this will install it as a local file)
Expand All @@ -457,17 +461,18 @@ ExtensionHelper::InstallExtensionInternal(DBConfig &config, FileSystem &fs, cons
if (IsFullPath(extension)) {
if (StringUtil::StartsWith(extension, "http://")) {
// HTTP takes separate path to avoid dependency on httpfs extension
return InstallFromHttpUrl(config, extension, extension_name, "", temp_path, local_extension_path, force_install,
http_logger);
}
return InstallFromHttpUrl(config, extension, extension_name, "", temp_path, local_extension_path,
force_install, http_logger);
}

// direct installation from local or remote path
return DirectInstallExtension(config, fs, extension, temp_path, extension, local_extension_path, force_install, "", context);
}
return DirectInstallExtension(config, fs, extension, temp_path, extension, local_extension_path, force_install,
"", context);
}

// Repository installation
return InstallFromRepository(config, fs, extension, extension_name, repository_url, temp_path,
local_extension_path, version, force_install, http_logger, context);
return InstallFromRepository(config, fs, extension, extension_name, repository_url, temp_path, local_extension_path,
version, force_install, http_logger, context);
#endif
#endif
}
Expand Down