linux:server:automated_aur_repo

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
linux:server:automated_aur_repo [2022/09/07 20:25] – removed - external edit (Unknown date) 127.0.0.1linux:server:automated_aur_repo [2022/09/07 20:25] (current) – ↷ Page moved from linux:home:automated_aur_repo to linux:server:automated_aur_repo michaelbromilow
Line 1: Line 1:
 +====== Automated AUR pacman repo ======
 +The goal of this page is to detail how to set up an AUR package repo using [[https://github.com/AladW/aurutils|aurutils]] which automatically updates AUR packages, and automatically rebuilds them if their dependent packages are updated. This results in a pacman repository that can be added on a remote Arch system to install AUR packages seamlessly, like official Arch packages.
  
 +__**This guide is written for a btrfs file system. It's applicable to other filesystems, but you'll have to figure out where to change installation instructions yourself**__
 +
 +===== Prerequisites =====
 +==== Install LXD ====
 +Simply install using the snap:
 +  $ snap install lxd
 +
 +Run the initialisation process:
 +  $ lxd init
 +
 +It will probably be helpful to add your user to the ''lxd'' group for permission to use LXD.
 +  $ usermod -a -G lxd $USER
 +
 +Log into your session again, or run ''newgrp lxd''.
 +
 +===== Set up an Arch container on the server =====
 +I opted to use LXD to run the Arch container. I assumed that since my server runs Ubuntu server 20.04 LTS, the differing kernel versions would be an issue, but everything seems to work fine.
 +
 +==== Filesystem ====
 +Since my server runs entirely on btrfs, I created two subvolumes; one for all of my containers to live in, and one to be mounted inside the container for my local pacman repository:
 +  # mount -o subvolid=0 /dev/sdb1 /mnt
 +  # cd /mnt
 +  # btrfs subvol create @lxd
 +  # btrfs subvol create @aur-repo
 +  # mkdir /vol/lxd
 +  # mkdir /vol/aur-repo
 +  # cd /
 +  # umount /mnt
 +
 +I added these to my ''/etc/fstab'':
 +<file>
 +UUID=... /vol/lxd btrfs compress=zstd,noatime,nodiratime,space_cache,commit=30,subvol=@lxd 0 0
 +UUID=... /vol/aur-repo btrfs compress=zstd,noatime,nodiratime,space_cache,commit=30,subvol=@aur-repo 0 0
 +</file>
 +  # mount -a
 +
 +I made a new group ''aur-user'' on the host and changed the ownership of ''/vol/aur-repo'' to ''www-data:aur-user'', with permissions ''575'', which allows the webserver read-only access and anyone in the ''aur-user'' group read-write access.
 +  # groupadd aur-user
 +  # chown -R www-data:aur-user /vol/aur-repo/
 +  # chmod -R 575 /vol/aur-repo/
 +
 +==== Create the container ====
 +After installing LXD, the btrfs subvolume for containers can be added as a new storage pool:
 +  $ lxc storage create default btrfs source=/vol/lxd/
 +  
 +Create a new LXD profile for the arch container, and open it for editing:
 +  $ lxc profile create aur-repo-archlinux
 +  $ lxc profile edit aur-repo-archlinux
 +
 +Add the following content:
 +<file>
 +config:
 +  security.nesting: "true"
 +  boot.autostart: "true"
 +description: Arch Linux Container Profile for AUR package repository
 +devices:
 +  aur-repo:
 +    path: /aur-repo
 +    source: /vol/aur-repo
 +    type: disk
 +  eth0:
 +    name: eth0
 +    nictype: bridged
 +    parent: lxdbr0
 +    type: nic
 +  root:
 +    path: /
 +    pool: default
 +    type: disk
 +name: archlinux
 +used_by: []
 +</file>
 +<callout>
 +''security.nesting'' is required in order to enable aurutils to use ''systemd-nspawn'' from within the container.
 +
 +The ''aur-repo'' device mounts the directory ''/vol/aur-repo'' to ''/aur-repo'' inside the container.
 +</callout>
 +
 +Create and launch the Arch Linux container with the new profile by downloading from the ''images'' image server:
 +  $ lxc launch images:archlinux archlinux-aur -p aur-repo-archlinux
 +
 +You can now get a root shell as follows:
 +  $ lxc exec archlinux-aur -- /bin/bash
 +
 +If you set up a user, get a user shell with:
 +  $ lxc exec archlinux-aur -- su --login username
 +
 +==== Setting up Arch ====
 +(inside the container)
 +The system locale will need to be set up; edit ''/etc/locale.gen'', uncomment ''en_US.UTF-8'' (and other locales you want), and run ''locale-gen''.
 +You will also need a non-root user to run ''makepkg'' to install aurutils.
 +  # useradd -m username
 +  # passwd username
 +  # usermod -a -G wheel username
 +
 +Install the following packages:
 +  * ''sudo'' - easy root access from an administrator account.
 +  * ''base-devel'', ''git'' - required for installing ''aurutils'' from the AUR.
 +  * ''devtools'' - required for building packages in a chroot with ''aurutils''.
 +  * ''btrfs-progs'' - required to allow ''aurutils'' to use btrfs subvolumes for its chroots.
 +
 +  # pacman -S --needed sudo base-devel git devtools btrfs-progs
 +  
 +Create a system user ''aur-user'' for aurutils to use:
 +  # useradd -r -m -s /usr/bin/nologin -U aur-user
 +
 +You should add your user to the new ''aur-user'' group so you have the same level of access:
 +  # usermod -a -G aur-user username
 +
 +<popover trigger="focus" title="To-do for this section" content="  * Pacman configuration (mirrors)"><btn type="danger">To-do for this section</btn></popover>
 +
 +==== Setting up the shared mount ====
 +Since LXD runs containers unprivileged by default, they are namespaced by the Linux kernel so that their UID/GIDs are outside of the range of the host's UID/GIDs - therefore, even ''root'' on the container only has ''nobody'' permissions on the host. This is problematic with file permissions when mounting directories, so we have to configure mapping such that certain host UIDs and GIDs appear as usable UIDs and GIDs in the container.
 +
 +To find the UID of a user, use:
 +  $ id -u [username]
 +
 +e.g.
 +  $ id -u michael
 +<file>
 +1000
 +</file>
 +
 +Likewise, to find the GID of a group, use:
 +  $ getent group groupname
 +
 +e.g.
 +  $ getent group sudo
 +<file>
 +sudo:x:27:michael
 +</file>
 +
 +Using these commands it can be found that **on the host** the UID of ''www-data'' is 33, and the GID of the ''aur-user'' group is 1010.
 +
 +Now add the necessary UID/GIDs to remap to ''/etc/subuid'' and ''/etc/subgid'' to allow LXD the permissions to remap them in namespaces (unprivileged containers) if requested:
 +  $ echo "root:33:1" | sudo tee -a /etc/subuid
 +  $ echo "root:1010:1" | sudo tee -a /etc/subgid
 +
 +Now, **in the Arch container**, find the UIDs/GIDs you wish to map to - in this case, the UID and GID of ''aur-user'':
 +  $ id -u aur-user
 +<file>
 +975
 +</file>
 +  $ getent group aur-user
 +<file>
 +  aur-user:x:975:
 +</file>
 +
 +Back on the host, let's add the relevant configuration line to our LXD profile:
 +  $ lxc profile edit aur-repo-archlinux
 +<file>
 +config:
 +  raw.idmap: |-
 +    uid 33 975
 +    gid 1010 975
 +</file>
 +
 +This remaps (inside the container) both the UID of the host's ''www-data'' (33) to the container's ''aur-user'' (975), and the GID of the host's ''aur-user'' (1010) to the container's ''aur-user'' (975).
 +
 +<callout>
 +If you were remapping two equal UID/GIDs on the host to two equal UID/GIDs on the container, you could use the following shorthand syntax:
 +<file>
 +config:
 +  raw.idmap: both 1000 1000
 +</file>
 +This would remap the host's 1000 UID and GID to the container's 1000 UID and GID respectively.
 +</callout>
 +
 +Now, restart the ''lxd'' daemon to use the changes in ''/etc/sub(u/g)id'':
 +  # systemctl restart snap.lxc.daemon
 +
 +Next time the container is started, the new mapping should now be applied. You can confirm this by looking at the owner of ''/aur-repo'': following the configuration exactly described above leaves the folder owned by the user and the group ''aur-user'':
 +  $ ls -laph / | grep aur-repo
 +<file>
 +dr-xrwxr-x   1 aur-user aur-user  124 Sep 10 23:25 aur-repo/
 +</file>
 +
 +===== Setting up aurutils =====
 +Assume instructions in this section are run in the Arch container unless otherwise specified.
 +
 +==== Installing aurutils ====
 +The easiest way to do this is to install the AUR package like any other; something like:
 +  $ mkdir ~/aur
 +  $ cd ~/aur
 +  $ git clone https://aur.archlinux.org/aurutils.git
 +  $ cd aurutils
 +  $ makepkg -si
 +  
 +You will probably need to import the PGP key before running ''makepkg'':
 +  $ gpg --recv-keys [Key ID]
 +<callout>
 +If gpg fails with ''keyserver receive failed: General error'', try specifying a keyserver with ''--keyserver'', e.g. ''--keyserver pool.sks-keyservers.net''.
 +</callout>
 +
 +==== Creating a custom local repository ====
 +Add a custom repository to ''/etc/pacman.conf'':
 +<file>
 +[mbromilow]
 +SigLevel = Optional TrustAll
 +Server = file:///aur-repo
 +</file>
 +
 +Create the repository root and database:
 +  # install -d /aur-repo -o aur-user
 +  # sudo -u aur-user repo-add /aurutils-repo/mbromilow.db.tar
 +
 +Synchronise pacman:
 +  # pacman -Syu
 +
 +It's probably a good idea to add your database to your webserver now. Here's my nginx configuration to do this:
 +  /etc/nginx/sites-available/55-aur.bromilow.uk
 +<file>
 +# Custom Arch User Repository
 +server {
 +    server_name aur.bromilow.uk;
 +    root /vol/aur-repo;
 +
 +    location / {
 +        # Enable folder indexing
 +        autoindex on;
 +        # We just try to get the URL as a file.
 +        try_files $uri $uri/ =404;
 +    }
 +
 +    listen 443 ssl;
 +    listen [::]:443 ssl;
 +    ssl_certificate /etc/letsencrypt/live/aur.bromilow.uk/fullchain.pem;
 +    ssl_certificate_key /etc/letsencrypt/live/aur.bromilow.uk/privkey.pem;
 +    include /etc/letsencrypt/options-ssl-nginx.conf;
 +    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
 +}
 +
 +server {
 +    server_name aur.bromilow.uk;
 +    listen 80;
 +    listen [::]:80;
 +
 +    return 301 https://$host$request_uri;
 +}
 +</file>
 +
 +==== Adding the repository to new systems ====
 +Following the above setup, this is quite simple. In my case, I simply add to ''/etc/pacman.conf'':
 +  [mbromilow]
 +  SigLevel = Optional TrustAll
 +  Server = https://aur.bromilow.uk/
 +
 +==== Modifying pacstrap to work in the unprivileged container ====
 +This one-liner replaces ''pacstrap'''s ''devtmpfs'' mount with a bind-mount of the container's ''/dev'' into the chroot's ''/dev'':
 +  # sed -i 's/chroot_add_mount udev \"$1\/dev\" -t devtmpfs/chroot_add_mount \/dev \"$1\/dev\" --bind/' /usr/bin/pacstrap
 +
 +<callout>
 +This is a **required** step if you want to build your AUR packages in a clean chroot (e.g. using ''aur sync -c'').
 +</callout>
 +
 +===== Management =====
 +In the Arch container...
 +
 +==== Adding Packages ====
 +  $ aur sync -c package_name
 +
 +Import any missing GPG keys as usual.
 +
 +==== Updating Packages ====
 +  $ aur sync -cu
 +
 +===== Troubleshooting =====
 +=== Error: Common start logic: Failed to change ACLs on /var/snap/lxd/common/lxd/storage-pools/default/containers/archlinux-aur/rootfs/var/log/journal ===
 +Delete the folder ''/var/log/journal'' in the container's rootfs (e.g. in ''/vol/lxd/containers/archlinux-aur/rootfs/var/log/journal/'')