[PC-BSD Pbi-dev] All in one approach to modules

Jesse Smith jessefrgsmith at yahoo.ca
Tue Oct 4 13:30:36 PDT 2011


-----Original Message-----
From: Ken Moore <ken at pcbsd.org>
To: pbi-dev at lists.pcbsd.org
Subject: Re: [PC-BSD Pbi-dev] All in one approach to modules
Date: Tue, 04 Oct 2011 16:17:20 -0400

On 10/04/11 15:09, Jesse Smith wrote:
> -----Original Message-----
> From: Ken Moore <ken at pcbsd.org>
> To: pbi-dev at lists.pcbsd.org
> Subject: Re: [PC-BSD Pbi-dev] All in one approach to modules
> Date: Tue, 04 Oct 2011 14:40:40 -0400
>
> On 10/03/11 16:22, Jesse Smith wrote:
>> -----Original Message-----
>> From: Ken Moore <ken at pcbsd.org>
>> To: pbi-dev at lists.pcbsd.org
>> Subject: Re: [PC-BSD Pbi-dev] All in one approach to modules
>> Date: Mon, 03 Oct 2011 12:59:32 -0400
>>
>> On 10/01/11 11:33, Jesse Smith wrote:
>>> We appear to be getting more requests for PBIs there days and it crossed
>>> my mind while looking at the latest list that maybe it would be worth it
>>> to just build a huge number of modules all at once.
>>>
>>> Aside from intimidating the over-worked quality assurance folks, is
>>> there any reason we can't just automate the creation of, say, the entire
>>> "www" and "deskutils" branches of the ports tree?
>>>
>>> It should be fairly simple to run a script to run pbimaker on a large
>>> group of ports and then upload the resulting modules to the build
>>> server. Granted, some PBIs won't build the first time around and some
>>> minor touch-ups might have to be done, but we could import hundreds of
>>> PBIs (many of which would probably work without issue) in one go.
>>>
>>> Before I go and make a few hundred PBI modules, does anyone have any
>>> objections to this idea?
>>>
>>> - Jesse
>>>
>>>
>>> _______________________________________________
>>> Pbi-dev mailing list
>>> Pbi-dev at lists.pcbsd.org
>>> http://lists.pcbsd.org/mailman/listinfo/pbi-dev
>> What you mention sounds like a good idea, however your method of looping
>> through a category needs to be fairly "smart" in order to only build
>> modules for actual programs and not libraries or data sets for other ports.
>> For example, the /usr/ports/games/ category is probably one of the best
>> categories to take this approach to, but if you browse through that
>> category on freshports, you will see that a number of the ports are
>> simply data for a game/program, and not the actual program. This problem
>> gets much worse as you start to look at some of the other categories.
>> You also will have to distinguish which programs are text-based or
>> servers as opposed to graphical applications so that you are (or are
>> not) creating desktop/menu entries for the program. Even after all that,
>> you will still have to go through each module individually to set the
>> proper icons, register mime types, etc.
>>
>> In other words, it would be great if you could pull it off, but there is
>> so much to the module building process that requires a personal touch
>> that I am uncertain you will be able to find a way to automatically
>> produce them.
>>
>> One thing I do recommend before trying is that before mass-producing
>> modules with pbimaker, you might want to check and make sure that the
>> pbimaker format for 9.x modules is up to date. There have been a few
>> changes recently, and the latest (and hopefully final) format for the
>> 9.x PBI modules is up on the wiki
>> (http://wiki.pcbsd.org/index.php/PBI_Module_Builder_Guide#Module_Components).
>> Specifically, I caught a few modules recently that were missing the
>> PBI_MAKEPORT line in pbi.conf and you might want to make sure that it is
>> being put in by pbimaker.
>>
>> Thanks Jesse!
>>
>> ~Ken Moore
>> PCBSD/iXsystems
>>
>>
>>
>>
>>
>> Ken,
>>
>> Thanks for the tip regarding the PBI format. I'll take a look at that.
>> The next version of PBImaker should fix the issue.
>>
>> Regarding the supporting ports and libraries, you raise a good point.
>> Often times there doesn't seem to be any way to tell the difference
>> between a library and a program. At least not directly or by looking at
>> the Makefile. We might be able to weed down the list if we only build
>> modules for ports that have a pkg-plist file that contains entries for
>> bin/
>>
>> Libraries and data support ports shouldn't place anything in the
>> PREFIX/bin/ directory, but most programs will. What if I do a check for
>> this instance and only build modules for ports with this feature?
>>
>> Jesse
>>
>>
>>
>> _______________________________________________
>> Pbi-dev mailing list
>> Pbi-dev at lists.pcbsd.org
>> http://lists.pcbsd.org/mailman/listinfo/pbi-dev
> Jesse,
>
> pkg-plist files are generally not required for a port. They are used
> when there is a large number of files that need to be included, but
> there are many smaller programs that only have a few libraries/files and
> simply list them in the makefile under PLIST=(whatever). However, if a
> pkg-plist exists, I think it is a good idea to look for files in the
> */bin/ category.
>
> The only issue with *requiring* a port to put something in the
> PREFIX/bin/ directory is that some server applications don't have
> binaries in */bin/ because everything is contained in */etc/
>
> In addition, I have been doing a bit more looking into the ports that
> are only data for games and such, and I have come across a couple
> conclusions that might help:
> 1) The name usually involves *-data-* or ends in *-data
> 2) Ports of data files generally do not have depends of any kind in the
> makefile
> These things should be fairly easy to check automatically, and might
> help narrow down what ports you build.
>
> I have not looked at the ports that are just libraries, but there are
> probably some similar trends for those ports.
>
>
>
> Ken,
>
> I agree, weeding out ports with -data in the name is one way to avoid a
> lot of supporting ports.
>
> Regarding the pkg-plist search for "bin", I just want to clear up a
> point. It isn't so much my intention to get every one of the available
> ports, just a lot of them in one go. Let's say, just for argument's
> sake, there are 1,000 games ports. And let's say half of them (500) have
> pkg-plist files. And, of those, only 100 have entries with ^bin/
> entries. That still gives us 100 new PBI modules. If we can do the same
> for the deskutils, emulators and multimedia categories, we're up 400 new
> PBIs. And we can go back to gather up the rest on a per-request basis.
>
> I guess what I'm thinking here is that we can get a lot done as long as
> we don't have any (or not many) false positives when building end-user
> applications.
>
> I think it should be fairly easy to separate the desktop applications
> with console apps by doing a check for files in pkg-plist that have a
> ".desktop" extension.
>
> Jesse
>
>
>
> _______________________________________________
> Pbi-dev mailing list
> Pbi-dev at lists.pcbsd.org
> http://lists.pcbsd.org/mailman/listinfo/pbi-dev

Jesse,

That sounds good. I see your point about just trying to get number of
them, and not necessarily trying to get them all. If it works out well,
you might be able to refine the process to get more later, as well as
filling in the blanks by hand.

So far it sounds like you have a good handle on this. When you get some
outputs, send them my way and I will get the modules onto the build server!




Ken,

I'm testing a script now which should do the "weeding" part. Once it
looks like it's filtering properly I will see about making modules.

Do we have to worry about duplicate modules? For example, if I try to
make modules for the "www" category, it will include Firefox. It would
be a pain to make all these modules and then have to remove, by hand,
each module we already have. Or if you go to upload a module that
already exists, will the build server just reject it?

If duplicates pose a problem I'll grab a list of modules we already have
a use that as an additional filter.

Jesse




More information about the Pbi-dev mailing list