[PC-BSD Dev] Binary patch support for updates.

[Kris Moore]

Dylan Cochran wrote:
> I was looking into adding support for bsdiff to the system updates and
> PBIs, however, I am missing two bits of information.
> 
> 1): I can find no definition of the current PBI file format, other
> then it is the concantenation of the pbistart binary, a header, a png
> file, a header, and two tarballs. To do binary updating properly, you
> need to first decompress any of the input files, as compressed input
> yields patches of the same size as the output file (which defeats the
> purpose). So I have to start by writing a PBI file format unpacker to
> break it into uncompressed components, and then update those. If there
> is any information about querying for the size of each element, and
> the 'proper' way to parse the input file, it would be a great help.

Dylan,

I don't really have any docs on how to query the elements, but this is 
how pbistart does it:

strcpy(awkcmd, "awk '/^__PBI_ARCHIVE__/ { print NR + 1; exit 0; }' '");

We first fine the __PBI_ARCHIVE__ flag, which indicates that we've found 
the first small tarbal, which contains the PBI details / configuration.

Then to extract this data we use tar

# tail +<offset as found above> <file> | head -n <lengh of tar ball> 
tar xvf -C <tmp dir> -f -

During the PBI installation, we then do the same type process for 
__PBI_ARCHIVE2__, which is the flag that indicates we're at the tar / 
lzma compressed file.


http://trac.pcbsd.org/browser/pbibuild/pbi-source/PBCsource/pbistart.c


> 2): I am unable to find any program or script that generates the .upd
> files and the update tarballs. The system updates look to be the
> simplest ones to get started on, as they are guarenteed to have an
> existing reference point, they are simple collections of files, and
> use an extensible metadata format that are simpler to add the extra
> information to. I want to get started on that, but I need the update
> generation and pushing script/programs.

Sadly, there isn't any auto-generator for .upd files yet :( I'm writing 
them by hand, when we need to issue an update, which isn't all that bad, 
since they are fairly easy to make, only a few lines.

The update tarbals are also rolled by hand, we simply put the data we 
want into a patch, and create the update.sh script, which actually 
performs the update, notifies users, etc. Then to make it, I use this 
command:

# tar cvf - . | lzma e -si ../patch-<date>-<name>.lzma

I guess the first thing to do would be to write some sort of generation 
program, correct? I think it'll still need some manual input though, for 
cases when we are updating files which we need to notify the user about, 
or prompt for some reason?


-- 

Kris Moore
PC-BSD Software
http://www.pcbsd.com