[PC-BSD Dev] Binary patch support for updates.

Dylan Cochran a134qaed at gmail.com
Wed Apr 22 12:17:00 PDT 2009

On 4/22/09, Kris Moore <kris at pcbsd.com> wrote:
> Dylan Cochran wrote:
>> I was looking into adding support for bsdiff to the system updates and
>> PBIs, however, I am missing two bits of information.
>> 1): I can find no definition of the current PBI file format, other
>> then it is the concantenation of the pbistart binary, a header, a png
>> file, a header, and two tarballs. To do binary updating properly, you
>> need to first decompress any of the input files, as compressed input
>> yields patches of the same size as the output file (which defeats the
>> purpose). So I have to start by writing a PBI file format unpacker to
>> break it into uncompressed components, and then update those. If there
>> is any information about querying for the size of each element, and
>> the 'proper' way to parse the input file, it would be a great help.
> Dylan,
> I don't really have any docs on how to query the elements, but this is
> how pbistart does it:
> strcpy(awkcmd, "awk '/^__PBI_ARCHIVE__/ { print NR + 1; exit 0; }' '");
> We first fine the __PBI_ARCHIVE__ flag, which indicates that we've found
> the first small tarbal, which contains the PBI details / configuration.
> Then to extract this data we use tar
> # tail +<offset as found above> <file> | head -n <lengh of tar ball>
> tar xvf -C <tmp dir> -f -
> During the PBI installation, we then do the same type process for
> __PBI_ARCHIVE2__, which is the flag that indicates we're at the tar /
> lzma compressed file.
> http://trac.pcbsd.org/browser/pbibuild/pbi-source/PBCsource/pbistart.c

This is enough to write a decompressor, though it will break on any
later versions of the format. Though, popping a version number into
the elf header at a specific place in later versions will be enough
future proofing.
>> 2): I am unable to find any program or script that generates the .upd
>> files and the update tarballs. The system updates look to be the
>> simplest ones to get started on, as they are guarenteed to have an
>> existing reference point, they are simple collections of files, and
>> use an extensible metadata format that are simpler to add the extra
>> information to. I want to get started on that, but I need the update
>> generation and pushing script/programs.
> Sadly, there isn't any auto-generator for .upd files yet :( I'm writing
> them by hand, when we need to issue an update, which isn't all that bad,
> since they are fairly easy to make, only a few lines.
> The update tarbals are also rolled by hand, we simply put the data we
> want into a patch, and create the update.sh script, which actually
> performs the update, notifies users, etc. Then to make it, I use this
> command:
> # tar cvf - . | lzma e -si ../patch-<date>-<name>.lzma
> I guess the first thing to do would be to write some sort of generation
> program, correct? I think it'll still need some manual input though, for
> cases when we are updating files which we need to notify the user about,
> or prompt for some reason?

So would the following workflow suffice:

# cd /
# update_gen sbin/mount_nfs usr/local/etc/rc.d/somescript
PCBSD/kernels/pcbsd-standard.tbz PCBSD/kernels/kern.ini boot/loader
  Name: NFS Security update
  Details URL:  http://www.pcbsd.org/blah

Then it proceeds to work through each file, and inspects the pathname.
If the pathname matches certain patterns, postprocessing should be
added (for example, if it's PCBSD/kernels/pcbsd-standard.tbz, it knows
to extract the tarball to /boot/, if it's in a lib/ directory, it
should prompt the user for a reboot, etc.). I'll write a mini-updater
to download all the patch tarballs, then grab all the files specified
in .upd files, and figure out what the normal patterns of
post-processing are. If a post processing step needs extra input, it
will prompt for it.

I'll keep in mind all the details of binary updating, but first
duplicate the existing output files. That way the update generation
can be used well before the system updater client and binary diff
generator are extended to support binary updating. I will also add
general improvements I can see are needed at the same time, for
example removing the actual reboot message, and just doing flag
passing so the reboot message occurs only once, etc. Am I correct in
assuming that the current system updater only looks at the head of
each line, ie MD5:, and ignores headers that it does not understand?

> --
> Kris Moore
> PC-BSD Software
> http://www.pcbsd.com

More information about the Dev mailing list