Unzip to stdout with filenames and file delims

Question

I have a tar which contains many zip files, each of which contains xml files. I would like to untar, unzip, and then do some stuff with the plaintext in the xml files. This entire program is being written as a bash pipeline.

I need the unzip command to return:

1. uncompressed file contents of each file inside the zip
2. file delimiter (to know when file contents of one file stop and next one start)
3. name of each file inside the zip

The next command in the pipeline needs these 3 things to do its job correctly. The file contents and delimiter have to be in a pipe (stdout), the filenames can either be in the same pipe or in variables or something.

Question: Which implementation of unzip should I use, and how do I do this?

bsdtar works (but AFAIK can't return filenames):

tar -xf ~/tar/0.tar --to-command 'bsdtar -xO --include='*.html' --include='*.xhtml' | iconv -f UTF-8 -t UTF-8//IGNORE | htmlq -tw'

P.S. I am doing the untar, unzip and everything inside a bash pipeline rather than on disk because writing to disk slows down the program by 30x. Each zip contains a lot of small files and trying to find them later hits the disk I/O bottleneck.

P.P.S. I am aware that unzipping requires reading to the end first, so in theory pipes shouldn't help much. In practice this is not slowing down the program much (I'm assuming the entire zip file gets stored in RAM, which is fine).

Answer 1

This should be a comment, but would be overlong. It might provide some guidance.

gzip is fairly smart. It cannot assume how much memory is available (so it won't read the whole file into memory), and it does not know the amount of compression it has achieved until that phase is complete. (It will of course use normal caching.)

While compressing, I believe it accumulates the list of files in the archive (and the stats for each) and appends those after all the data. Then it cunningly writes an epilogue which contains the seek address of the start of that file list. The file stats could also contain the offset in the zip file of the start of each subfile, which would optimise partial extracts too.

So unzip with the -l or -v option can seek to the end, seek back by sizeof(epilogue), and report the file contents without reading anything else. You could verify my conjecture by running a small test file under strace.

You might find it is sufficiently fast to get the filename list with unzip -v, parse that with awk, and extract each file separately, maybe prepending the file details. This would also be an interesting exercise under strace, which would show the seek/read strategy.

You might find it requires to write files to disk, but you could optimise that with a ram-disk as the individual files are small. Also, the zip family may delete some files once they have been processed -- test in some development directory before attempting production.

If this is a production job, I would probably disconnect the extract and upload. Extract files into a pend subdirectory, move completed files into a live subdirectory, and the uploader (maybe in parallel) moves uploaded files into a done directory.

EDIT: Note zcat and gunzip do not fully handle zip archives. zip and unzip are required.

I made a test .zip file and ran a couple of commands inside strace. The results are somewhat confusing because the file structure aligns some things (like the file list) to block boundaries, and unzip also reads complete aligned 8192-byte chunks.

However, I can confirm:

(a) unzip -v Test1.zip makes two seeks and one read to produce the verbose file listing.

(b) unzip Test1.zip csvParse.c makes five seeks and six reads to search for, find and extract one file. It does not read anything that is not required.

My conclusion is that dealing with each extracted file individually will not impact performance significantly, and will simplify the required processing.

The archive contents list appears to be very regularly formatted in columns, and should be simple to parse to get the filenames.