autospec¶

autospec is a tool used to assist with the automated creation and maintenance of RPM packaging in Clear Linux* OS. Where a standard RPM build process using rpmbuild requires a tarball and .spec file to start, autospec requires only a tarball and package name to start.

Description
How it works
Examples
Test packaged software
References
Related topics

Description ¶

The autospec tool attempts to infer the requirements of the .spec file by analyzing the source code and Makefile information. It continuously runs updated builds based on new information discovered from build failures until it has a complete and valid .spec file. If needed, you can influence the behavior of autospec and customize the build by providing optional control files to the autospec tool.

autospec uses mock as a sandbox to run the builds. Visit the mock wiki for additional information on using mock.

For a general understanding of how an RPM works, visit the rpm website or the RPM Packaging Guide.

How it works ¶

Learn the autospec tool set up and process.

Prerequisites
Create an RPM

Prerequisites ¶

The setup for building source in Clear Linux OS must be completed before using the autospec tool.

Refer to Setup environment to build source for instructions on completing the setup.

Create an RPM ¶

The basic autospec process is described in the following steps:

The make autospec command generates a .spec file based on the analysis of code and existing control files.

Any control files should be located in the same directory as the resulting .spec file. View the autospec README for more information on control files.
autospec creates a build root with mock config.
autospec attempts to build an RPM from the generated .spec.
autospec detects any missed declarations in the .spec.
If build errors occur, autospec scans the build log to try to detect the root cause.
If autospec detects the root cause and knows how to continue, it restarts the build automatically at step 1 with updated build instructions.
Otherwise, autospec stops the build for user inspection to resolve the errors. Respond to the build process output by fixing source code issues and/or editing control files to resolve issues, which may include dependencies or exclusions. See autospec README for more information on control files.

The user resumes the process at step 1 after errors are resolved.

If a binary dependency doesn’t exist in Clear Linux OS, you must build it before running autospec again.

Following these steps, autospec continues to rebuild the package, based on new information discovered from build failures, until it has a valid .spec. If no build errors occur, RPM packages are successfully built.

Examples ¶

Complete Setup environment to build source before using these examples.

Example 1: Build RPM with an existing spec file
Example 2: Build a new RPM
Example 3: Generate a new spec file with a pre-defined package
Example 4: Provide control files to autospec
Example 5: Update an existing package

Example 1: Build RPM with an existing spec file ¶

This example shows how to build a RPM from a pre-packaged upstream package with an existing spec file. The example uses the dmidecode package.

Navigate to the autospec workspace and clone the dmidecode package:
```
cd ~/clearlinux
make clone_dmidecode
```
Note

You can clone all package repos at once using the following command:
```
make [-j NUM] clone-packages
```
The optional NUM is the number of threads to use.

For a list of available packages, view the ~/clearlinux/projects/common/packages file.
Navigate to the local copy of the dmidecode package and build it:
```
cd ~/clearlinux/packages/dmidecode/
make build
```
The resulting RPMs are in ./rpms. Build logs and additional RPMs are in ./results.

Example 2: Build a new RPM ¶

This example shows how to build a new RPM with no spec file. The example will create a simple helloclear RPM.

Navigate to the autospec workspace and build the helloclear RPM. The Makefile provides a make autospecnew that can automatically generate an RPM package using the autospec tool. You must pass the URL to the source tarball and the NAME of the RPM you wish to create:
```
cd ~/clearlinux
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear"
```
The resulting RPMs are in ./packages/helloclear/rpms. Build logs and additional RPMs are in ./packages/helloclear/results.

Example 3: Generate a new spec file with a pre-defined package ¶

This example shows how to modify an existing package to create a custom RPM. In this example you will make a simple change to the dmidecode package and rebuild the package.

Navigate to the autospec workspace and clone the dmidecode package:
```
cd ~/clearlinux
make clone_dmidecode
```
Navigate into the dmidecode directory:
```
cd packages/dmidecode
```

Open the excludes file with an editor and add these lines:

/usr/bin/biosdecode
/usr/bin/ownership
/usr/bin/vpddecode
/usr/share/man/man8/biosdecode.8
/usr/share/man/man8/ownership.8
/usr/share/man/man8/vpddecode.8

Note

These files aren’t needed by dmidecode, so we can remove them without any issues.

In the dmidecode directory, build the modified dmidecode package:
```
make autospec
```
The resulting RPMs are in ./rpms. Logs are in ./results.

Example 4: Provide control files to autospec ¶

This example shows how to modify control files to correct build failures that autospec is unable to resolve. In this example, you will add a missing license and dependencies so autospec can complete a successful build.

Navigate to the autospec workspace:
```
cd ~/clearlinux
```
If you have not already, clone all upstream package repos:
```
make [-j NUM] clone-packages
```
The optional NUM is the number of threads to use.

Note

In a later step of this example, we will search the cloned package repos for a missing dependency.

Build the opae-sdk RPM:

make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk"

This results in an error for a missing license file:

[FATAL]    Cannot find any license or opae-sdk.license file!

Navigate to the package with build failures:
```
cd packages/opae-sdk
```
Add one or more valid license identifiers from the SPDX License List. In the example below, two different licenses are appropriate based on the opae-sdk project licensing:
```
echo "BSD-3-Clause MIT" > opae-sdk.license
```
Run autospec again:
```
make autospec
```
This results in a generic error:
```
[FATAL]    Build failed, aborting
```

Open the build log to view the error details:

cat ./results/build.log

The build log contains details for the specific failures. In this instance, there are missing dependencies:

CMake Error: The following variables are used in this project, but
they are set to NOTFOUND.  Please set them or make sure they are set and tested correctly in the CMake files:

CJSON_LIBRARY
   linked by target "opae-c++-utils" in directory /builddir/build/BUILD/opae-sdk-0.13.0/tools/c++utilslib
json-c_LIBRARIES
   linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
libuuid_LIBRARIES
   linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae

Search the spec files of upstream Clear Linux OS packages to see if the json-c library is available. In this case, it does exist and we’ll add the json-c ‘dev’ package into the buildreq_add:
```
grep 'json-c\.so$' ~/clearlinux/packages/*/*.spec
echo "json-c-dev" >> buildreq_add
```
Note

This search step works only if the user cloned all of the upstream package repos. In this example, upstream package repos were cloned in a previous step.
Search the spec files of upstream Clear Linux OS packages to see if the libuuid library is available. In this case, it exists in the util-linux package, so we’ll add util-linux-dev package into the buildreq_add:
```
grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec
echo "util-linux-dev" >> buildreq_add
```
Run autospec again and find the successfully-generated RPMs in the rpms directory:
```
make autospec
```
Note

If you need a dependency that does not exist in the Clear Linux OS repo, you must first build it manually (see Example 2: Build a new RPM), then add the repo so that autospec knows the package exists. For example:
```
cd ~/clearlinux/packages/<package-name>
make repoadd
make repostatus
```
You only need to add the dependency to the buildreq_add control file if autospec is not able to automatically find the correct dependency on its own.

Example 5: Update an existing package ¶

The Clear Linux OS team prefers to carry no patches and seeks to make the latest releases work. If we do need patches, we use autospec to add, remove, or manage patches. The autospec control files are integral to the patch management process. Developers can expect a more streamlined approach to managing a large collection of packages with autospec.

Adding and submitting patches¶

To add patches to Clear Linux OS upstream, follow patching source code.
To submit a patch to upstream, follow contributing to an existing software package.

If you maintain a downstream derivative of Clear Linux OS and you want to integrate new or patched packages into your mix, follow the process in mixer.

Assuming you have followed the above process, autospec has generated a new spec file.

Refresh a package and inspect¶

In this example, we use autospec to refresh the m4 package and recreate RPM files.

Navigate to the top-level directory of the workspace
```
cd clearlinux
```
- where clearlinux is the top level of the tooling workspace
Run the make_clone command and then navigate to the package.
```
make clone_m4

cd packages/m4
```
Make desired changes to the package, its control files, or other files.
Finally, run:
```
make autospec
```

To view spec file changes, run:

git show m4.spec

The output shows:

m4: Autospec creation for version 1.4.18

diff --git a/m4.spec b/m4.spec
index f76c78d..97b846a 100644
--- a/m4.spec
+++ b/m4.spec
@@ -6,15 +6,14 @@
#
Name     : m4
Version  : 1.4.18
-Release  : 88
+Release  : 89
URL      : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz
Source0  : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz
-Source99 : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz.sig
+Source1 : http://mirrors.kernel.org/gnu/m4/m4-1.4.18.tar.xz.sig
Summary  : No detailed summary available
Group    : Development/Tools
...

The following commands provide a more complete view of the changes.
- git log -p
- gitk

Test packaged software ¶

After software has been packaged with autospec, the resulting RPMs can be tested for functionality before being integrated and deployed into a Clear Linux OS image with the Mixer tool.

The Clear Linux OS development tooling offers two ways to quickly test autospec generated RPMs.

Note

The methods outlined below should only be used for temporary testing on development systems.

Test in a Clear Linux OS virtual machine¶

The Clear Linux OS development tooling includes a method to install RPMs into a Clear Linux OS virtual machine running on the KVM hypervisor. Using a VM allows testing in a completely isolated environment.

To test an autospec-created package inside a VM:

Download the Clear Linux OS KVM image into the ~/clearlinux directory as clear.img. The location and name clear.img.xz is important for the tooling to work:

cd ~/clearlinux
curl -o clear.img.xz https://download.clearlinux.org/image/$(curl https://download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm)

Extract the downloaded Clear Linux OS KVM image:
```
unxz -v clear.img.xz
```
Copy the QEMU start script and virtual firmware needed for KVM into the ~/clearlinux directory:
```
cp ~/clearlinux/projects/common/start_qemu.sh .
cp /usr/share/qemu/OVMF.fd .
```
Run make install from the package’s autospec directory. The make install command mounts the downloaded Clear Linux OS KVM image and installs the autospec-created RPM into it:
```
cd ~/clearlinux/packages/<package-name>
make install
```
The code that makes this possible can be viewed by searching for the install: target in the Makefile.common file on GitHub.
Return to the ~/clearlinux directory and start the Clear Linux OS VM:
```
cd ~/clearlinux/
sudo ./start_qemu.sh clear.img
```
A new Clear Linux OS VM will launch in the console. Log into the VM as root and set a new password for the VM.
Check that the software is installed in the Clear Linux OS VM as expected and perform any relevant tests.
After testing has been completed, the Clear Linux OS VM can be powered off and deleted:
```
poweroff
rm clear.img
```

Test directly on a development machine¶

The Clear Linux OS development tooling also includes a method to extract autospec-created RPMs locally onto a Clear Linux OS development system for testing. Extracting an RPM directly onto a system offers quicker testing; however conflicts may occur and responsibility to remove the software after testing is up to the developer.

To test an autospec created package directly on the Clear Linux OS development system:

Run make install-local from the package’s autospec directory. The make install-local command extracts the RPM directly onto the filesystem of the running Clear Linux OS system:
```
cd ~/clearlinux/packages/<package-name>
make install-local
```
The code that makes this possible can be viewed by searching for the install-local: target in the Makefile.common file on GitHub.
Check that the software is installed as expected and perform any relevant tests.
After testing has been completed, the software and any related files must be identified and deleted. The swupd repair --picky command can help restore the state of the /usr directory (see swupd) however any other files must be cleaned up manually.

References ¶

Reference the autospec README for details regarding autospec commands and options.

Setup environment to build source¶

Setup of the workspace and tooling used for building source in Clear Linux OS is mostly automated for you with a setup script. It uses tools from the os-clr-on-clr bundle.

The setup script creates a workspace in the clearlinux folder, with the subfolders Makefile, packages, and projects. The projects folder contains the main tools used for making packages in Clear Linux OS autospec and common.

Follow these steps to setup the workspace and tooling for building source:

Install the os-clr-on-clr bundle:
```
sudo swupd bundle-add os-clr-on-clr
```

Download the user-setup.sh script:

curl -O https://raw.githubusercontent.com/clearlinux/common/master/user-setup.sh

Make user-setup.sh executable:
```
chmod +x user-setup.sh
```
Run the script as an unprivileged user:
```
./user-setup.sh
```
After the script completes, log out and log in again to complete the setup process.
Set your Git user email and username for the repos on your system:
```
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
```
This global setting is used by Clear Linux OS tools that make use of Git.