Manjaro-tools


Introduction

manjaro-tools consists of 4 different tools: mkset, buildpkg, buildtree, and buildiso.

mkset and buildpkg include scripts and tools for Manjaro Linux package maintainers. Official packages get built in clean chroots to avoid dependency conflicts with our branches.

buildtree is a little tool to sync Arch abs and Manjaro packages git repositories.

buildiso is a small set of Bash scripts that is capable of building fully functional Manjaro Linux based live medias. It is a very generic tool, so it could potentially be used to generate anything from rescue systems, to install disks, to special interest live CD systems, and who knows what else. Simply put, if it involves Manjaro on a shiny coaster, it can do it.


manjaro-tools is a replacement for devtools and manjaroiso.

A detailed user manual is available at github.


Configuration

manjaro-tools can be configured via editing /etc/manjaro-tools/manjaro-tools.conf


Files

These are the new names for renamed scripts.

  • mkmanjaroroot --> mkchroot
  • manjarobuild --> buildpkg
  • pacstrap --> basestrap
  • genfstab --> fstabgen
  • arch-chroot --> manjaro-chroot


mkset

mkset is used to create build lists. Build lists can be defined in /etc/manjaro-tools/sets/<buildlistname>.set

To create a build list, one can run in the pkgbuilds dir:

mkset -c <name>

This will create a build list containing all directories with PKGBUILDs in them. This generated list can be then edited, like packages can be removed.

For example, in a git clone of https://github.com/manjaro/packages-openrc

$ ls 
consolekit/                  manjaro-tools-git/           openrc-net/
eudev/                       manjaro-tools-git-devel/     openrc-systemdcompat/
eudev-systemdcompat/         manjaro-tools-iso-profiles/  openrc-video/
kde-servicemenus-pkg-tools/  manjaro-tools-livecd/        packages/
keyboardctl-openrc/          networkmanager-openrc/       pkgbuilds-review/
kf5-servicemenus-pkg-tools/  openrc-aur/                  polkit-consolekit/
lib32-eudev/                 openrc-base/                 sddm-consolekit/
lib32-eudev-systemdcompat/   openrc-core/                 sysvinit/
lxqt-lightdm-greeter/        openrc-desktop/              upower-pm-utils/
manjaro-openrc/              openrc-devel/                README.md
manjaro-tools/               openrc-misc/

Running mkset -c openrc does

$ mkset -c openrc
==> Creating [/etc/manjaro-tools/sets/openrc.set] ...
 -> Adding consolekit
 -> Adding eudev
 -> Adding eudev-systemdcompat
 -> Adding kde-servicemenus-pkg-tools
 -> Adding kf5-servicemenus-pkg-tools
 -> Adding lib32-eudev
 -> Adding lib32-eudev-systemdcompat
 -> Adding lxqt-lightdm-greeter
 -> Adding manjaro-openrc
 -> Adding manjaro-tools
 -> Adding manjaro-tools-git
 -> Adding manjaro-tools-git-devel
 -> Adding manjaro-tools-iso-profiles
 -> Adding manjaro-tools-livecd
 -> Adding networkmanager-openrc
 -> Adding openrc-aur
 -> Adding openrc-base
 -> Adding openrc-core
 -> Adding openrc-desktop
 -> Adding openrc-devel
 -> Adding openrc-misc
 -> Adding openrc-net
 -> Adding openrc-systemdcompat
 -> Adding openrc-video
 -> Adding polkit-consolekit
 -> Adding sddm-consolekit
 -> Adding sysvinit
 -> Adding upower-pm-utils


Note
The set name should be different from a directory name in pkgbuilds dir. Anything else should work, eg adding a date to the name.


buildpkg

buildpkg is used to build a particular package or a set. The help looks like the following for i686

$ buildpkg -h
Usage: buildpkg [options] [--] [makepkg args]
   -p <profile>       Set profile or pkg [default: default]
   -a <arch>          Set arch  [default: i686]
   -b <branch>        Set branch [default: stable]
   -r <dir>           Chroots directory [default: /srv/manjarobuild]
   -c                 Recreate chroot
   -w                 Clean up
   -n                 Install and run namcap check
   -s                 Sign packages
   -q                 Query settings and pretend build
   -h                 This help

To build a single package, go into the directory which is one above the package build directory (which contains the PKGBUILD), and run it as:

buildpkg -p <package-name>

For example, we want to build the upower-pm-utils package from the following PKGBUILD folders

$ ls
consolekit                  manjaro-tools          openrc-video
eudev                       networkmanager-openrc  packages
eudev-systemdcompat         openrc-aur             polkit-consolekit
kde-servicemenus-pkg-tools  openrc-base            README.md
keyboardctl-openrc          openrc-core            sddm-consolekit
kf5-servicemenus-pkg-tools  openrc-desktop         sysvinit
lib32-eudev                 openrc-devel           upower-pm-utils
lib32-eudev-systemdcompat   openrc-misc
manjaro-openrc              openrc-net

To test if the options are correct, the -q option can be used.

$ buildpkg -q upower-pm-utils
==> manjaro-tools
 -> version: 0.9.5.6
 -> manjaro_tools_conf: /etc/manjaro-tools/manjaro-tools.conf
==> OPTIONS:
 -> arch: i686
 -> branch: stable
 -> chroots: /srv/manjarobuild
==> ARGS:
 -> mkchrootpkg_args: -r /srv/manjarobuild/stable/i686
 -> makepkg_args: 
 -> clean_first: false
 -> wipe_clean: false
 -> namcap: false
 -> sign: false
==> PATHS:
 -> chrootdir: /srv/manjarobuild/stable/i686
 -> profiledir: /etc/manjaro-tools/sets
 -> pkg_dir: /var/cache/manjaro-tools/stable/i686
 -> pacman_conf: /usr/share/manjaro-tools/pacman-default.conf
 -> makepkg_conf: /usr/share/manjaro-tools/makepkg-i686.conf
 -> mirrors_conf: /usr/share/manjaro-tools/pacman-mirrors-stable.conf
==> BLACKLIST:
 -> blacklist_trigger: eudev lib32-eudev upower-pm-utils eudev-systemdcompat lib32-eudev-systemdcompat
 -> blacklist: libsystemd
==> SETS:
 -> profiles: default
 -> profile: upower-pm-utils
 -> is_profile: false
==> Build queue:
 -> upower-pm-utils

After that to build the package -p can be used.

$ buildpkg -p upower-pm-utils
==> Updating chroot for [testing] (x86_64)...
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
:: Starting full system upgrade...
 there is nothing to do
==> Making package: upower-pm-utils 0.9.23-4 (Mon Jan 12 12:38:16 IST 2015)
==> Retrieving sources...
...
...
==> Leaving fakeroot environment.
==> Finished making: upower-pm-utils 0.9.23-4 (Mon Jan 12 07:08:28 UTC 2015)

To clean the chroot before building -c can be used.

$ buildpkg -p upower-pm-utils -c
==> Creating chroot for [testing] (x86_64)...
 -> Deleting chroot copy 'aaditya'...
 -> Deleting chroot copy 'root'...
==> Creating install root at /mnt/datalinux2/manjaro-tools/testing/x86_64/root
==> Installing packages to /mnt/datalinux2/manjaro-tools/testing/x86_64/root
:: Synchronizing package databases...
 core                     133.7 KiB  30.8K/s 00:04 [######################] 100%
 extra                   1920.2 KiB   392K/s 00:05 [######################] 100%
 community                  2.7 MiB   512K/s 00:05 [######################] 100%
:: There are 25 members in group base-devel:
:: Repository core
  1) autoconf  2) automake  3) binutils  4) bison  5) fakeroot  6) file
  7) findutils  8) flex  9) gawk  10) gcc  11) gettext  12) grep  13) groff
  14) gzip  15) libtool  16) m4  17) make  18) pacman  19) patch
  20) pkg-config  21) sed  22) sudo  23) texinfo  24) util-linux  25) which
...
...
==> Creating clean working copy [aaditya]...
==> Making package: upower-pm-utils 0.9.23-4 (Mon Jan 12 12:37:00 IST 2015)
==> Retrieving sources...
...
...
==> Leaving fakeroot environment.
==> Finished making: upower-pm-utils 0.9.23-4 (Mon Jan 12 07:07:20 UTC 2015)

To build a set the name of the set can be used. The current sets can be queried with the -q option.


buildtree

buildtree is a little tools to sync arch abs and manjaro packages git repos.

The arguments are:

$ buildtree -h
Usage: buildtree [options]
    -s            Sync manjaro tree
    -a            Sync arch abs
    -c            Clean package tree
    -q            Query settings
    -h            This help[/code]

To sync Arch and Manjaro trees:

buildtree -as


buildiso

buildiso is used to build a particular ISO or a set of ISOs.

Overview

The help looks like the following for x86_64:

$ buildiso -h
Usage: buildiso [options]
   -p <profile>       Buildset or profile [default: default]
   -a <arch>          Arch [default: x86_64]
   -b <branch>        Branch [default: unstable]
   -r <dir>           Chroots directory
                      [default: /mnt/datalinux2/manjaro-tools/buildiso]
   -w                 Disable clean iso cache
   -c                 Disable clean work dir
   -x                 Disable clean xorg cache
   -l                 Disable clean lng cache
   -i                 Build images only
   -s                 Generate iso only
                      Requires pre built images (-i)
   -q                 Query settings and pretend build
   -h                 This help


Build profiles for official and community editions of Manjaro can be obtained from here, and this is what the current folder looks like:

$ ls
cinnamon       kde          kde-openrc  lxqt-openrc  netbook     openbox-legacy  shared   xfce-minimal
enlightenment  kde5         lxde        mate         net-openrc  pekwm-openrc    sysmenu  xfce-openbox-openrc
gnome          kde-minimal  lxqt        net          openbox     README.md       xfce

Query Building

To query build an ISO (-q option), for example the xfce-openbox-openrc profile, the following command can be used:

$ buildiso -p xfce-openbox-openrc/ -b stable -q
==> manjaro-tools
 -> version: 0.9.6.4
 -> user_config: /home/aaditya/.config/manjaro-tools.conf
==> PROFILE:
 -> buildsets: community|default|official|openrc
 -> buildset_iso: xfce-openbox-openrc/
 -> is_buildset: false
==> OPTIONS:
 -> arch: x86_64
 -> branch: stable
 -> chroots_iso: /mnt/datalinux2/manjaro-tools/buildiso
==> ARGS:
 -> clean_first: true
 -> clean_cache_xorg: true
 -> clean_cache_lng: true
 -> clean_cache_iso: true
 -> images_only: false
 -> iso_only: false
==> PATHS:
 -> sets_dir_iso: /etc/manjaro-tools/sets/iso
 -> cache_dir_iso: /mnt/datalinux2/manjaro-tools/cache/iso
 -> cache_dir_xorg: /mnt/datalinux2/manjaro-tools/cache/xorg/x86_64
 -> cache_dir_lng: /mnt/datalinux2/manjaro-tools/cache/lng/x86_64
==> IMAGES SETTINGS:
 -> manjaro_kernel: linux318
 -> manjaro_version: 2015.03
 -> plymouth_theme: manjaro-elegant
==> ISO SETTINGS:
 -> iso_label: MJRO0812
 -> iso_version: openrc-2015-03-01
 -> code_name: Ascella
==> LIVECD:
 -> username: manjaro
 -> password: manjaro
 -> addgroups: video,audio,power,disk,storage,optical,network,lp,scanner,wheel
 -> hostname: manjaro
==> BUILD QUEUE:
--> Profile: [xfce-openbox-openrc/] ...
 -> Packages-Custom: Packages-Xfce
 -> initsys: openrc
 -> displaymanager: lxdm
 -> is_plymouth: false
 -> pacman_conf: pacman-multilib.conf
 -> work_dir: /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64
 -> iso_file: manjaro-xfce-openrc-2015-03-01-x86_64.iso
 -> start_openrc: syslog-ng cronie acpid alsasound dbus consolekit wicd bluetooth
 -> start_openrc_live: livecd mhwd-live pacman-init pacman-boot

Building

To actually build the ISO:

$ buildiso -p xfce-openbox-openrc/ -b stable
==> Start building [xfce-openbox-openrc/]
--> Profile: [xfce-openbox-openrc/] ...
 -> Packages-Custom: Packages-Xfce
==> Cleaning up ...
 -> Deleting chroot 'root-image'...
 -> Deleting chroot 'xfce-image'...
 -> Cleaning [/mnt/datalinux2/manjaro-tools/cache/xorg/x86_64] ...
 -> Cleaning [/mnt/datalinux2/manjaro-tools/cache/lng/x86_64] ...
--> Loading Packages: [Packages] ...
==> Prepare [Base installation] (root-image)
==> mkiso : Configuration Settings
 -> working directory: /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64
 -> image name: none
 -> pacman_mirrors: /usr/share/manjaro-tools/pacman-mirrors-stable.conf
==> Creating install root at /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/root-image
==> Installing packages to /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/root-image
:: Synchronizing package databases...
core                                                    132.7 KiB   202K/s 00:01 [##############################################] 100%
extra                                                  1916.0 KiB   939K/s 00:02 [##############################################] 100%
community                                                 2.8 MiB   524K/s 00:05 [##############################################] 100%
multilib                                                140.9 KiB   217K/s 00:01 [##############################################] 100%
...
...
...
 -> Cleaning up what we can
warning: database file for 'core' does not exist
warning: database file for 'extra' does not exist
warning: database file for 'community' does not exist
warning: database file for 'multilib' does not exist
warning: database file for 'openrc-eudev' does not exist
 -> Copying overlay ...
==> Done [Base installation] (root-image)
--> Loading Packages: [Packages-Xfce] ...
==> Prepare [xfce installation] (xfce-image)
 -> mount [root-image] on [xfce-image]
==> mkiso : Configuration Settings
 -> working directory: /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64
 -> image name: none
 -> pacman_mirrors: /usr/share/manjaro-tools/pacman-mirrors-stable.conf
==> Creating install root at /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/xfce-image
==> Installing packages to /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/xfce-image
:: Synchronizing package databases...
...
...
...
Generation complete.
 -> Cleaning up what we can
warning: database file for 'core' does not exist
warning: database file for 'extra' does not exist
warning: database file for 'community' does not exist
warning: database file for 'multilib' does not exist
warning: database file for 'openrc-eudev' does not exist
 -> Copying xfce-overlay ...
==> Configuring [xfce-image]
 -> Configuring Displaymanager ...
 -> Configured: lxdm
--> Congiguring OpenRC ....
 -> Setting syslog-ng ...
 -> Setting cronie ...
 -> Setting acpid ...
 -> Setting alsasound ...
 -> Setting dbus ...
 -> Setting consolekit ...
 -> Setting wicd ...
 -> Setting bluetooth ...
--> Done configuring OpenRC
==> Done configuring [xfce-image]
 -> unmount xfce-image
==> Done [xfce installation] (xfce-image)
--> Loading Packages: [Packages-Livecd] ...
==> Prepare [livecd-image]
 -> mount [livecd-image] on [xfce-image]
 -> append [root-image] on [livecd-image]
==> mkiso : Configuration Settings
 -> working directory: /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64
 -> image name: none
 -> pacman_mirrors: /usr/share/manjaro-tools/pacman-mirrors-testing.conf
==> Creating install root at /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/livecd-image
==> Installing packages to /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/livecd-image
:: Synchronizing package databases...
...
...
...
==> Making bootable image
 -> Removing existing bootable image...
==> Creating ISO image...
xorriso 1.3.8 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev 'stdio:/mnt/datalinux2/manjaro-tools/cache/iso/manjaro-xfce-openrc-2015-03-01-x86_64.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 62.7g free
Added to ISO image: directory '/'='/mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/iso'
xorriso : UPDATE : 115 files added in 1 seconds
xorriso : UPDATE : 115 files added in 1 seconds
xorriso : NOTE : Copying to System Area: 432 bytes from file '/mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc/x86_64/iso/isolinux/isohdpfx.bin'
libisofs: NOTE : Aligned image size to cylinder size by 42 blocks
xorriso : UPDATE :  2.75% done
xorriso : UPDATE :  16.40% done
xorriso : UPDATE :  27.90% done, estimate finish Sun Mar 01 23:10:20 2015
xorriso : UPDATE :  93.66% done
xorriso : UPDATE :  99.20% done
xorriso : UPDATE :  99.46% done
xorriso : UPDATE :  99.47% done
xorriso : UPDATE :  99.67% done
xorriso : UPDATE :  99.68% done
ISO image produced: 343552 sectors
Written to medium : 343552 sectors at LBA 0
Writing to 'stdio:/mnt/datalinux2/manjaro-tools/cache/iso/manjaro-xfce-openrc-2015-03-01-x86_64.iso' completed successfully.

==> Done [Build ISO]
==> Creating [md5sum] ...
 -> md5sum: 78f4b335fc593b7df7edff46a4d91b5d  manjaro-xfce-openrc-2015-03-01-x86_64.iso
==> Done [md5sum]
--> Time compress_images: 9.43 minutes
==> Finished building [xfce-openbox-openrc/]
--> Time make_profile: 13.77 minutes

Building with predownloaded Xorg packages

To build an ISO while retaining the previously downloaded cache of Xorg packages, the -x option can be used:

$ buildiso -p xfce-openbox-openrc/ -b stable -x

Building with a small config change

Note
It is to be verified if the procedure given below works or not.

Supposing something only changed in config, like a setting, instead of building the whole ISO from scratch, the ISO build directory can be modified and the ISO can be rebuilt. For example, supposing one changed xfce-overlay/etc/skel/.conkyrc in the config, one can go into the work directory, modify the said file, and rebuild the ISO using the -cs option.

The work directory can be found using the -q option:

$ buildiso -p xfce-openbox-openrc/ -q
==> manjaro-tools
 -> version: 0.9.6.4
 -> user_config: /home/aaditya/.config/manjaro-tools.conf
==> PROFILE:
 -> buildsets: community|default|official|openrc
 -> buildset_iso: xfce-openbox-openrc/
 -> is_buildset: false
==> OPTIONS:
 -> arch: x86_64
 -> branch: unstable
 -> chroots_iso: /mnt/datalinux2/manjaro-tools/buildiso
...
...
==> BUILD QUEUE:
--> Profile: [xfce-openbox-openrc/] ...
 -> Packages-Custom: Packages-Xfce
 -> initsys: openrc
 -> displaymanager: lxdm
 -> is_plymouth: false
 -> pacman_conf: pacman-multilib.conf
 -> work_dir: /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64
 -> iso_file: manjaro-xfce-openrc-2015-03-01-x86_64.iso

(work directory highlighted in bold)

The required file can be edited as root, for example:

$ sudo nano /mnt/datalinux2/manjaro-tools/buildiso/xfce-openbox-openrc//x86_64/xfce-image/etc/skel/.conkyrc

There is a catch involved here though; the file one wants to edit may be present in multiple images, like root-image, livecd-image, and DE-image (eg, xfce-image), and would need to be edited in the respective image directories.


Then the ISO then can be rebuilt as:

$ buildiso -p xfce-openbox-openrc/ -cs

Alternative

First only the chroot for the ISO could be created, using the -i option, then the changes can be made, and finally an ISO can be built with the -sc option.

For example:

$ buildiso -p xfce-openrc/ -b stable -i
==> Start building [xfce-openrc/]
--> Profile: [xfce-openrc/]
 -> Packages-Custom: Packages-Xfce
==> Cleaning up ...
...
...
==> Done [manjaro/iso/isolinux]
==> Creating [isomounts]
 -> Writing livecd entry ...
 -> Writing pkgs entry ...
 -> Writing xfce entry ...
 -> Writing root entry ...
==> Done creating [isomounts]
--> Time build_images: 4.28 minutes
==> WARNING: Continue compress: buildiso -p xfce-openrc/ -sc ...

After this the changes in the work directory can be made, and the ISO can be generated with:

$ buildiso -p xfce-openrc/ -sc

Further reading