.\" Automatically generated by Pod::Man v1.37, Pod::Parser v1.32 .\" .\" Standard preamble: .\" ======================================================================== .de Sh \" Subsection heading .br .if t .Sp .ne 5 .PP \fB\\$1\fR .PP .. .de Sp \" Vertical space (when we can't use .PP) .if t .sp .5v .if n .sp .. .de Vb \" Begin verbatim text .ft CW .nf .ne \\$1 .. .de Ve \" End verbatim text .ft R .fi .. .\" Set up some character translations and predefined strings. \*(-- will .\" give an unbreakable dash, \*(PI will give pi, \*(L" will give a left .\" double quote, and \*(R" will give a right double quote. | will give a .\" real vertical bar. \*(C+ will give a nicer C++. Capital omega is used to .\" do unbreakable dashes and therefore won't be available. \*(C` and \*(C' .\" expand to `' in nroff, nothing in troff, for use with C<>. .tr \(*W-|\(bv\*(Tr .ds C+ C\v'-.1v'\h'-1p'\s-2+\h'-1p'+\s0\v'.1v'\h'-1p' .ie n \{\ . ds -- \(*W- . ds PI pi . if (\n(.H=4u)&(1m=24u) .ds -- \(*W\h'-12u'\(*W\h'-12u'-\" diablo 10 pitch . if (\n(.H=4u)&(1m=20u) .ds -- \(*W\h'-12u'\(*W\h'-8u'-\" diablo 12 pitch . ds L" "" . ds R" "" . ds C` "" . ds C' "" 'br\} .el\{\ . ds -- \|\(em\| . ds PI \(*p . ds L" `` . ds R" '' 'br\} .\" .\" If the F register is turned on, we'll generate index entries on stderr for .\" titles (.TH), headers (.SH), subsections (.Sh), items (.Ip), and index .\" entries marked with X<> in POD. Of course, you'll have to process the .\" output yourself in some meaningful fashion. .if \nF \{\ . de IX . tm Index:\\$1\t\\n%\t"\\$2" .. . nr % 0 . rr F .\} .\" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .hy 0 .if n .na .\" .\" Accent mark definitions (@(#)ms.acc 1.5 88/02/08 SMI; from UCB 4.2). .\" Fear. Run. Save yourself. No user-serviceable parts. . \" fudge factors for nroff and troff .if n \{\ . ds #H 0 . ds #V .8m . ds #F .3m . ds #[ \f1 . ds #] \fP .\} .if t \{\ . ds #H ((1u-(\\\\n(.fu%2u))*.13m) . ds #V .6m . ds #F 0 . ds #[ \& . ds #] \& .\} . \" simple accents for nroff and troff .if n \{\ . ds ' \& . ds ` \& . ds ^ \& . ds , \& . ds ~ ~ . ds / .\} .if t \{\ . ds ' \\k:\h'-(\\n(.wu*8/10-\*(#H)'\'\h"|\\n:u" . ds ` \\k:\h'-(\\n(.wu*8/10-\*(#H)'\`\h'|\\n:u' . ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'^\h'|\\n:u' . ds , \\k:\h'-(\\n(.wu*8/10)',\h'|\\n:u' . ds ~ \\k:\h'-(\\n(.wu-\*(#H-.1m)'~\h'|\\n:u' . ds / \\k:\h'-(\\n(.wu*8/10-\*(#H)'\z\(sl\h'|\\n:u' .\} . \" troff and (daisy-wheel) nroff accents .ds : \\k:\h'-(\\n(.wu*8/10-\*(#H+.1m+\*(#F)'\v'-\*(#V'\z.\h'.2m+\*(#F'.\h'|\\n:u'\v'\*(#V' .ds 8 \h'\*(#H'\(*b\h'-\*(#H' .ds o \\k:\h'-(\\n(.wu+\w'\(de'u-\*(#H)/2u'\v'-.3n'\*(#[\z\(de\v'.3n'\h'|\\n:u'\*(#] .ds d- \h'\*(#H'\(pd\h'-\w'~'u'\v'-.25m'\f2\(hy\fP\v'.25m'\h'-\*(#H' .ds D- D\\k:\h'-\w'D'u'\v'-.11m'\z\(hy\v'.11m'\h'|\\n:u' .ds th \*(#[\v'.3m'\s+1I\s-1\v'-.3m'\h'-(\w'I'u*2/3)'\s-1o\s+1\*(#] .ds Th \*(#[\s+2I\s-2\h'-\w'I'u*3/5'\v'-.3m'o\v'.3m'\*(#] .ds ae a\h'-(\w'a'u*4/10)'e .ds Ae A\h'-(\w'A'u*4/10)'E . \" corrections for vroff .if v .ds ~ \\k:\h'-(\\n(.wu*9/10-\*(#H)'\s-2\u~\d\s+2\h'|\\n:u' .if v .ds ^ \\k:\h'-(\\n(.wu*10/11-\*(#H)'\v'-.4m'^\v'.4m'\h'|\\n:u' . \" for low resolution devices (crt and lpr) .if \n(.H>23 .if \n(.V>19 \ \{\ . ds : e . ds 8 ss . ds o a . ds d- d\h'-1'\(ga . ds D- D\h'-1'\(hy . ds th \o'bp' . ds Th \o'LP' . ds ae ae . ds Ae AE .\} .rm #[ #] #H #V #F C .\" ======================================================================== .\" .IX Title "OpenBSD::Intro 3p" .TH OpenBSD::Intro 3p "2008-05-30" "perl v5.8.8" "Perl Programmers Reference Guide" .SH "NAME" OpenBSD::Intro \- Introduction to the pkg tools internals .SH "SYNOPSIS" .IX Header "SYNOPSIS" .Vb 2 \& use OpenBSD::PackingList; \& ... .Ve .SH "DESCRIPTION" .IX Header "DESCRIPTION" Note that the \f(CW\*(C`OpenBSD::\*(C'\fR namespace of perl modules is not limited to package tools, but also includes \fIpkg\-config\fR\|(1) and \fImakewhatis\fR\|(8) support modules. This document only covers package tools material. .PP The design of the package tools revolves around a few central ideas: .PP Design modules that manipulate some notions in a consistent way, so that they can be used by the package tools proper, but also with a high-level \s-1API\s0 that's useful for anything that needs to manipulate packages. This was validated by the ease with which we can now update packing\-lists, check for conflicts, and check various properties of our packages. .PP Try to be as safe as possible where installation and update operations are concerned. Cut up operations into small subsets which yields frequent safe intermediate points where the machine is completely functional. .PP Traditional package tools often rely on the following model: take a snapshot of the system, try to perform an operation, and roll back to a stable state if anything goes wrong. .PP Instead, OpenBSD package tools take a computational approach: record semantic information in a useful format, pre-compute as much as can be about an operation, and only perform the operation when we have proved that (almost) nothing can go wrong. As far as possible, the actual operation happens on the side, as a temporary scaffolding, and we only commit to the operation once most of the work is over. .PP Keep high-level semantic information instead of recomputing it all the time, but try to organize as much as possible as plain text files. Originally, it was a bit of a challenge: trying to see how much we could get away with, before having to define an actual database format. Turns out we do not need a database format, or even any cache on the ftp server. .PP Avoid copying files all over the place. Hence the \fIOpenBSD::Ustar\fR\|(3p) module that allows package tools to manipulate tarballs directly without having to extract them first in a staging area. .PP All the package tools use the same internal perl modules, which gives them some consistency about fundamental notions. .PP It is highly recommended to try to understand packing-lists and packing elements first, since they are the core that unlocks most of the package tools. .SH "COMMON NOTIONS" .IX Header "COMMON NOTIONS" .IP "packing-lists and elements" 3 .IX Item "packing-lists and elements" Each package consists of a list of objects (mostly files, but there are some other abstract structures, like new user accounts, or stuff to do when the package gets installed). They are recorded in a \fIOpenBSD::PackingList\fR\|(3p), the module offers everything needed to manipulate packing\-lists. The packing-list format has a text representation, which is documented in \fIpkg_create\fR\|(3p). Internally, packing-lists are heavily structured. Objects are reordered by the internals of \fIOpenBSD::PackingList\fR\|(3p), and there are some standard filters defined to gain access to some commonly used information (dependencies and conflicts mostly) without having to read and parse the whole packing\-list. Each object is an \fIOpenBSD::PackingElement\fR\|(3p), which is an abstract class with lots of children classes. The use of packing-lists most often combines two classic design patterns: one uses Visitor to traverse a packing-list and perform an operation on all its elements (this is where the order is important, and why some stuff like user creation will `bubble up' to the beginning of the list), allied to Template Method: the operation is often not determined for a basic \fIOpenBSD::PackingElement\fR\|(3p), but will make more sense to an \&\fIOpenBSD::PackingElement::FileObject\fR\|(3p) or similar. Packing-list objects have an \*(L"automatic visitor\*(R" property: if a method is not defined for the packing-list proper, but exists for packing elements, then invoking the method on the packing-list will traverse it and apply the method to each element. For instance, package installation happens through the following snippet: .Sp .Vb 1 \& $plist->install_and_progress(...) .Ve .Sp where \f(CW\*(C`install_and_progress\*(C'\fR is defined at the packing element level, and invokes \f(CW\*(C`install\*(C'\fR and shows a progress bar if needed. .IP "package names and specs" 3 .IX Item "package names and specs" Package names and specifications for package names have a specific format, which is described in \fIpackages\-specs\fR\|(7). Package names are handled within \fIOpenBSD::PackageName\fR\|(3p). There is also a framework to organize searches based on \fIOpenBSD::Search\fR\|(3p) objects. Specifications are structured in a specific way, which yields a shorthand for conflict handling through \fIOpenBSD::PkgCfl\fR\|(3p), allows the package system to resolve dependencies in \fIOpenBSD::Dependencies\fR\|(3p) and to figure out package updates in \fIOpenBSD::Update\fR\|(3p). .IP "sources of packages" 3 .IX Item "sources of packages" Historically, \fIOpenBSD::PackageInfo\fR\|(3p) was used to get to the list of installed packages and grab information. This is now part of a more generic framework \fIOpenBSD::PackageRepository\fR\|(3p), which interacts with the search objects to allow you to access packages, be they installed, on the local machines, or distant. Once a package is located, the repository yields a proxy object called \fIOpenBSD::PackageLocation\fR\|(3p) that can be used to gain further info. (There are still shortcuts for installed packages for performance and simplicity reasons.) .IP "update sets" 3 .IX Item "update sets" This is a new notion (introduced around OpenBSD 4.1) which is not yet fully used throughout the package tools. Each operation (installation, removal, or replacement of packages) is cut up into small atomic operations, in order to guarantee maximal stability of the installed system. The package tools will try really hard to only deal with one or two packages at a time, in order to minimize combinatorial complexity, and to have a maximal number of safe points, where an update operation can stop without hosing the whole system. An update set is simply a minimal bag of packages, with old packages that are going to be installed, new packages that are going to replace them, and an area to record related ongoing computations. The old set may be empty, the new set may be empty, and in all cases, the update set shall be small (as small as possible). Currently, the package tools do not deal with all the cases they should. This is still under construction. We have already met with update situations where dependencies between packages invert (A\-1.0 depends on B\-1.0, but B\-0.0 depends on A\-0.0), or where files move between packages, which in theory will require update-sets with two new packages that replace two old packages. Currently, we cheat. .IP "dependency information" 3 .IX Item "dependency information" Dependency information exists at three levels: first, there are source specifications within ports. Then, those specifications turn into binary specifications with more constraints when the package is built by \&\fIpkg_create\fR\|(1), and finally, they're matched against lists of installed objects when the package is installed, and recorded as lists of inter-dependencies in the package system. .Sp At the package level, there are currently two types of dependencies: package specifications, that establish direct dependencies between packages, and shared libraries, that are described below. .Sp Normal dependencies are shallow: it is up to the package tools to figure out a whole dependency tree throughout top-level dependencies. None of this is hard\-coded: this a prerequisite for flavored packages to work, as we do not want to depend on a specific package if something more generic will do. .Sp At the same time, shared libraries have harsher constraints: a package won't work without the exact same shared libraries it needs (same major number, at least), so shared libraries are handled through a want/provide mechanism that walks the whole dependency tree to find the required shared libraries. .Sp Dependencies are just a subclass of the packing\-elements, rooted at the \f(CW\*(C`OpenBSD::PackingElement::Depend\*(C'\fR class. .Sp A specific \f(CW\*(C`OpenBSD::Dependencies::Solver\*(C'\fR object is used for the resolution of dependencies (see \fIOpenBSD::Dependencies\fR\|(3p), the solver is mostly a tree\-walker, but there are performance considerations, so it may need to cache a lot of information). Specificities of shared libraries are handled by \fIOpenBSD::SharedLibs\fR\|(3p). In particular, the base system also provides some shared libraries which are not recorded within the dependency tree. .Sp Lists of inter-dependencies are recorded in both directions (RequiredBy/Requiring). The \fIOpenBSD::RequiredBy\fR\|(3p) module handles the subtleties (removing duplicates, keeping things ordered, and handling pretend operations). .IP "shared items" 3 .IX Item "shared items" Some items may be recorded multiple times within several packages (mostly directories, users and groups). There is a specific \fIOpenBSD::SharedItems\fR\|(3p) module which handles these. Mostly, removal operations will scan all packing-lists at high speed to figure out shared items, and remove stuff that's no longer in use. .IP "virtual file system" 3 .IX Item "virtual file system" Most package operations will lead to the installation and removal of some files. Everything is checked beforehand: the package systeme must verify that no new file will erase an existing file, or that the file system won't overflow during the package installation. The package tools also have a \*(L"pretend\*(R" mode where the user can check what will happen before doing an operation. All the computations and caching are handled through the \fIOpenBSD::Vstat\fR\|(3p) module, which is designed to hide file system oddities, and to perform addition/deletion operations virtually before doing them for real. .SH "BASIC ALGORITHMS" .IX Header "BASIC ALGORITHMS" There are three basic operations: package addition (installation), package removal (deinstallation), and package replacement (update). .PP These operations are achieved through repeating the correct operations on all elements of a packing\-list. .Sh "\s-1PACKAGE\s0 \s-1ADDITION\s0" .IX Subsection "PACKAGE ADDITION" For package addition, \fIpkg_add\fR\|(1) first checks that everything is correct, then runs through the packing\-list, and extracts element from the archive. .Sh "\s-1PACKAGE\s0 \s-1DELETION\s0" .IX Subsection "PACKAGE DELETION" For package deletion, \fIpkg_delete\fR\|(1) removes elements from the packing\-list, and marks `common' stuff that may need to be unregistered, then walks quickly through all installed packages and removes stuff that's no longer used (directories, users, groups...) .Sh "\s-1PACKAGE\s0 \s-1REPLACEMENT\s0" .IX Subsection "PACKAGE REPLACEMENT" Package replacement is more complicated. It relies on package names and conflict markers. .PP In normal usage, pkg_add installs only new stuff, and checks that all files in the new package don't already exist in the file system. By convention, packages with the same stem are assumed to be different versions of the same package, e.g., screen\-1.0 and screen\-1.1 correspond to the same software, and users are not expected to be able to install both at the same time. .PP This is a conflict. .PP One can also mark extra conflicts (if two software distributions install the same file, generally a bad idea), or remove default conflict markers (for instance, so that the user can install several versions of autoconf at the same time). .PP If pkg_add is invoked in replacement mode (\-r), it will use conflict information to figure out which package(s) it should replace. It will then operate in a specific mode, where it replaces old package(s) with a new one. .IP "\(bu" 4 determine which package to replace through conflict information .IP "\(bu" 4 extract the new package 'alongside' the existing package(s) using temporary filenames. .IP "\(bu" 4 remove the old package .IP "\(bu" 4 finish installing the new package by renaming the temporary files. .PP Thus replacements will work without needing any extra information besides conflict markers. pkg_add \-r will happily replace any package with a conflicting package. Due to missing information (one can't predict the future), conflict markers work both way: packages a and b conflict as soon as a conflicts with b, or b conflicts with a. .Sh "\s-1PACKAGE\s0 \s-1UPDATES\s0" .IX Subsection "PACKAGE UPDATES" Package replacement is the basic operation behind package updates. In your average update, each individual package will be replaced by a more recent one, starting with dependencies, so that the installation stays functional the whole time. Shared libraries enjoy a special status: old shared libraries are kept around in a stub .lib\-* package, so that software that depends on them keeps running. (Thus, it is vital that porters pay attention to shared library version numbers during an update.) .PP An update operation starts by figuring out which packages to update (walking the selected packages dependency trees). .PP Then it looks for new packages to replace the old ones. This search relies on stem names first (e.g., to update package foo\-1.0, pkg_add \-u will look for foo\-* in the \s-1PKG_PATH\s0), then it trims the search results by looking more closely inside the package candidates. More specifically, their pkgpath (the directory in the ports tree from which they were compiled). Thus, a package that comes from category/someport/snapshot will never replace a package that comes from category/someport/stable. Likewise for flavors. .PP Finally, pkg_add \-u decides whether the update is needed by comparing the package signatures: a package signature is information is composed of the name of a package, together with relevant dependency information: all wantlib versions, and all run dependencies versions. If anything is different, it means the new package must replace the old one. .PP Currently, pkg_add \-u stops at the first entry in the \s-1PKG_PATH\s0 from which suitable candidates are found. .SH "BUGS AND LIMITATIONS" .IX Header "BUGS AND LIMITATIONS" There are a few desireable changes that will happen in the future: .IP "\(bu" 4 pkg_add \-u should be able to find updates with a stem change. This requires an extra mechanism that would say `oh, this package changed name', as we cannot afford to look into every single package. .IP "\(bu" 4 more complicated update scenarios could happen, once UpdateSets are fully functional. .IP "\(bu" 4 pkg_add has no real notion of version number. Thus, updates work only in so far as you direct your \s-1PKG_PATH\s0 to a more recent repository. .IP "\(bu" 4 there should be some carefully designed mechanisms to register more `global' processing, to avoid exec/unexec. .IP "\(bu" 4 some packages should be able to refuse automatic update, and to interact with the user. Again, to be carefully designed, so that one can still understand what's going on. .SH "LIST OF MODULES" .IX Header "LIST OF MODULES" .IP "OpenBSD::Add" 3 .IX Item "OpenBSD::Add" common operations related to a package addition. .IP "OpenBSD::ArcCheck" 3 .IX Item "OpenBSD::ArcCheck" additional layer on top of \f(CW\*(C`OpenBSD::Ustar\*(C'\fR that matches extra information that the archive format cannot record with a packing\-list. .IP "OpenBSD::CollisionReport" 3 .IX Item "OpenBSD::CollisionReport" checks a collision list obtained through \f(CW\*(C`OpenBSD::Vstat\*(C'\fR against the full list of installed files, and reports origin of existing files. .IP "OpenBSD::Delete" 3 .IX Item "OpenBSD::Delete" common operations related to a package deletion .IP "OpenBSD::Error" 3 .IX Item "OpenBSD::Error" handles external calls through \f(CW\*(C`system(3p)\*(C'\fR, exception mechanism, fatal errors and warnings. .IP "OpenBSD::Getopt" 3 .IX Item "OpenBSD::Getopt" \&\fIGetopt::Std\fR\|(3p)\-like with extra hooks for special options. .IP "OpenBSD::IdCache" 3 .IX Item "OpenBSD::IdCache" caches uid and gid vs. user names and group names correspondences. .IP "OpenBSD::Interactive" 3 .IX Item "OpenBSD::Interactive" handles user questions (to be redesigned) .IP "OpenBSD::Mtree" 3 .IX Item "OpenBSD::Mtree" simple parser for \fImtree\fR\|(8) specifications. .IP "OpenBSD::PackageInfo" 3 .IX Item "OpenBSD::PackageInfo" handles package meta-information (all the +CONTENTS, +DESCR, etc files) .IP "OpenBSD::PackageLocation" 3 .IX Item "OpenBSD::PackageLocation" proxy for a package, either as a tarball, or an installed package. Obtained through \f(CW\*(C`OpenBSD::PackageRepository\*(C'\fR. .IP "OpenBSD::PackageLocator" 3 .IX Item "OpenBSD::PackageLocator" central non-OO hub for the normal repository list (should use a singleton pattern instead). .IP "OpenBSD::PackageName" 3 .IX Item "OpenBSD::PackageName" common operations on package names. .IP "OpenBSD::PackageRepository" 3 .IX Item "OpenBSD::PackageRepository" base class for all package sources. Actual packages instantiate as \&\f(CW\*(C`OpenBSD::PackageLocation\*(C'\fR. .IP "OpenBSD::PackageRepositoryList" 3 .IX Item "OpenBSD::PackageRepositoryList" list of package repository, provided as a front to search objects, because searching through a repository list has \fIld\fR\|(1)\-like semantics (stops at the first repository that matches). .IP "OpenBSD::PackingElement" 3 .IX Item "OpenBSD::PackingElement" all the packing-list elements class hierarchy, together with common methods that do not belong elsewhere. .IP "OpenBSD::Paths" 3 .IX Item "OpenBSD::Paths" hardcoded paths to external programs and locations. .IP "OpenBSD::PkgCfl" 3 .IX Item "OpenBSD::PkgCfl" conflict lists handling in an efficient way. .IP "OpenBSD::PkgSpec" 3 .IX Item "OpenBSD::PkgSpec" ad-hoc search for package specifications. External \s-1API\s0 is stable, but it needs to be updated to use \f(CW\*(C`OpenBSD::PackageName\*(C'\fR objects now that they exist. .IP "OpenBSD::ProgressMeter" 3 .IX Item "OpenBSD::ProgressMeter" handles display of a progress meter when a terminal is available. Provided as a singleton object so that a graphical interface could be used instead. In practice, most input-output of the package tools will need to be abstracted first. .IP "OpenBSD::Replace" 3 .IX Item "OpenBSD::Replace" common operations related to package replacement. .IP "OpenBSD::RequiredBy" 3 .IX Item "OpenBSD::RequiredBy" handles requiredby and requiring lists. .IP "OpenBSD::Search" 3 .IX Item "OpenBSD::Search" search object for package repositories: specs, stems, and pkgpaths. .IP "OpenBSD::SharedItems" 3 .IX Item "OpenBSD::SharedItems" handles items that may be shared by several packages. .IP "OpenBSD::SharedLibs" 3 .IX Item "OpenBSD::SharedLibs" shared library specificities when handled as dependencies. .IP "OpenBSD::Subst" 3 .IX Item "OpenBSD::Subst" conventions used for substituting variables during \fIpkg_create\fR\|(1), and related algorithms. .IP "OpenBSD::Temp" 3 .IX Item "OpenBSD::Temp" safe creation of temporary files as a light-weight module that also deals with signal issues. .IP "OpenBSD::Update" 3 .IX Item "OpenBSD::Update" computes lists of package replacements required by an update. .IP "OpenBSD::UpdateSet" 3 .IX Item "OpenBSD::UpdateSet" common operations to all package tools that manipulate update sets. Lots of things to do. .IP "OpenBSD::Ustar" 3 .IX Item "OpenBSD::Ustar" simple \s-1API\s0 that allows for Ustar (new tar) archive manipulation, allowing for extraction and copies on the fly. .IP "OpenBSD::Vstat" 3 .IX Item "OpenBSD::Vstat" virtual file system (pretend) operations. .IP "OpenBSD::md5" 3 .IX Item "OpenBSD::md5" simple interface to the \fIDigest::MD5\fR\|(3p) module.