when using virtual tables, dumpiter generates a broken db script
virtual table entries must be created as master table entries
the sqlite tools dump does that correctly
however pythons iterdump seems to do that rather different and wrong

sqlite3 test.db "create virtual table test using fts4(example);"
-------------------------------
sqlite dump
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
PRAGMA writable_schema=ON;
INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','test','test',0,'CREATE VIRTUAL TABLE test using fts4(example)');
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
PRAGMA writable_schema=OFF;
COMMIT;
------------------------------
python iterdump "import sqlite3;
c=sqlite3.connect("test.db");
print("\n".join(c.iterdump()))"
BEGIN TRANSACTION;
CREATE VIRTUAL TABLE test using fts4(example);
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
COMMIT;

apsw contains code that handles the issues with dumping SQLite databases very well. I plan to integrate this code into pysqlite. We can then later port the fix to the sqlite3 module.

See ghaering/pysqlite#10 for the tasks and
https://github.com/rogerbinns/apsw/blob/master/tools/shell.py#L1012 for the apsw code.

Attached is patch with a fix for this issue. This is not as comprehensive as the solution in apsw but I think this should cover most of the cases. The result for Ronny dump after this fix is:

BEGIN TRANSACTION;
PRAGMA writable_schema=ON;
INSERT INTO sqlite_master(type,name,tbl_name,rootpage,sql)VALUES('table','test','test',0,'CREATE VIRTUAL TABLE test using fts4(example)');
CREATE TABLE 'test_content'(docid INTEGER PRIMARY KEY, 'c0example');
CREATE TABLE 'test_docsize'(docid INTEGER PRIMARY KEY, size BLOB);
CREATE TABLE 'test_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx));
CREATE TABLE 'test_segments'(blockid INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'test_stat'(id INTEGER PRIMARY KEY, value BLOB);
PRAGMA writable_schema=OFF;
COMMIT;

My plan for iterdump is to align it with the implementation in the SQLite shell, but I haven't gotten around to it yet. I think @palaviv's patch is a step in the right direction, and I don't think we should block that patch, just because iterdump also has other problems.

Hm, bug or feature; this issue was originally labeled as a "bug". But, the source code clearly included the comment:

# NOTE: Virtual table support not implemented

In this case, I believe the correct label should have been "feature", not "bug", so I'm inclined not to backport it.

I'm going to close this as resolved. Virtual table support has been added to Python 3.13. If you disagree, please ping me, or re-open. cc. @palaviv @RonnyPfannschmidt @ghaering

I consider it a bug as the output wasn't valid,it should error if it cant make it,

I consider it a bug as the output wasn't valid,it should error if it cant make it,

Hm, yeah, that is a valid point that weighs heavier than the comment.

OTOH, this changes the behaviour, which is an argument not to backport it.

@erlend-aasland personally i prefer people see a error when they dump rather than when they later load that dump

this bug in the worst case makes a broken backup silently appear working - not failing early causes more damage

silent breakage/invalid datagen is imho worse than adding a error where else people have hidden broken data until they test again

ftr - imho its fine to have a backport that just finally raises a NotImplementedError rather than silently dumping broken data

ftr - imho its fine to have a backport that just finally raises a NotImplementedError rather than silently dumping broken data

I'd rather backport the fix. Anyway, you've convinced me; I agree that this is more a bugfix than a new feature. I'll prepare backports through 3.11.

Thanks for you input, @RonnyPfannschmidt.

Backports are queued; marking this as resolved.