Fresh soft for your Amazon AMI. Part 1. Stealing from Fedora


Posted:   |  More posts about linux amazon linux yum   |   Source

I recently started using Amazon Linux AMI as the main platform for deployment. So far I used Debian and Ubuntu distributions for long time, and very soon I was disappointed how outdated some software, provided by standard Amazon Linux AMI and EPEL repositories is.

Then I decided to to find out, how easy is to build your own software for Amazon Linux AMI. Luckily, it turned out to be easier, when you don't start from scratch, but use existing packages as a leverage. Here and in the Part 2. Publishing your own work I share my experience.

These instructions should work for all RHEL-based distributions, and to build rpm packages for CentOS in particular.

Motivation

Yeah, the software in Amazon Linux AMI sometimes is too old. In particular, what I found outdated:

  • python-pip 0.8, released August 2010 (currently available 1.2)
  • python-virtualenv 1.7, released November 2011 (currently available 1.8.4)
  • redis 2.4, released October 2011 (currently available 2.6)
  • supervisor 2, released March 2007 (most frightening example, currently available 3.0b1)

Not to mention that default Python is still 2.6 (released in 2008), and Python 3 is not in repository yet (released 2008-2009).

Actually, it's not that bad, after all. For example nginx is only two months old, and so on, but if you got accustomed to the luxury of being on the "bleeding edge", you'd feel slightly uncomfortable in a new environment, as I did.

Options available

I discovered that people tend to resort to following three decisions:

  • Send a request for a new package to Amazon AMI staff. It usually works, but maybe not with something which touches core system and python packaging, as python is used extensively in the RH-based distributions.
  • Build soft from source, install in a separate location. Then use it without telling anyone, that they turned rpm-based system to a slackware-alike distribution.
  • Install third-party rpm, designed to work in foreign environment. Then hope that it will work, and they will find updates later.

Two last options are kind of "improper". The rightful way would be to build a new rpm package, and to distribute it to your machines with a yum repository.

First, I consider how we can make it properly and at the same time with no big hassle. In the second part I describe more advanced techniques: how to use mock with SCM and how to publish your own yum repository on Amazon S3.

Rebuild rpm. Preparation

Usually all the software is already packaged. The only problem is that is is outdated sometimes. Here I describe how I re-built the latest version of python-pip package from Fedora repository.

I use Fedora Mock to build packages in a chrooted environment.

# yum install --enablerepo=epel mock

Create a separate user for builds

# useradd -s /bin/bash mockbuild
# usermod -a -G mock mockbuild
# newgrp mock

Then try to reset your privileges to a mockbuild and initialize a new chroot environment.

$ mock -r default --init

The -r option means "root" and defines the way how chroot will be created. Descriptions of roots you may find /etc/mock directory.

Assuming that you work in a Linux AMI instance, and your instance is a 64-bit, the directory /var/lib/mock/epel-6-x86_64/root/ contains the virtual root, and the directory /var/lib/mock/epel-6-x86_64/result/ will contain the output srpm and rpm files (not now, currently it should be empty).

Rebuild existing srpm from Fedora Rawhide

Rawhide is the version of Fedora, containing the most recent versions of the packages. These packages aren't meant to work in Amazon Linux AMI, and to be honest, sometimes aren't meant to work at all (it's a very unstable branch, actually). Nonetheless, as it's the "cheapest" way for us to build a package, we start with it.

Find out the Fedora rawhide repository file, and add it to the list of your repositories. Download whatever you like from rpmfind.net.

# cd /tmp
# wget ftp://rpmfind.net/linux/fedora/linux/development/rawhide/x86_64/os/Packages/f/fedora-release-rawhide-19-0.1.noarch.rpm
# rpm2cpio fedora-release-rawhide-19-0.1.noarch.rpm | cpio -idmv
# cp etc/yum.repos.d/fedora-rawhide.repo /etc/yum.repos.d/

This set of command downloads the rpm package for you and extract it to current directory. With the last command you copy the repository description to the standard repos path.

You may accidentally turn your system to garbage with the simple yum install, unless you disable rawhide repository. We will only enable rawhide temporary, to download srpms.

sed -i 's/enabled=1/enabled=0/g' /etc/yum.repos.d/fedora-rawhide.repo

Then find which version of the package Rawhide provides. Below is the python-pip example. It's cool that you can enable and disable yum repositories on the fly, by the way.

$ yum info --enablerepo='rawhide*' python-pip

Then download the srpm. Script yumdownloader is the part of yum-utils package which should be already installed in your system:

$ yumdownloader --enablerepo='rawhide*' --source python-pip

Rebuild the package

$ mock -r default --rebuild ./python-pip-1.2.1-2.fc19.src.rpm

If the build has passed successfully, you may find your packages in the /var/lib/mock/epel-6-x86_64/result/ directory. You may install it immediately with rpm -ihv <path/to/python-pip.rpm> or yum install <path/to/python-pip.rpm>

Final notes

How kosher is the way of building rpm packages I described here? Not so much, actually. First, it's hard to distribute your package. Second, you don't have any control over the package contents, and unfortunatelu, the successful rebuild don't guarantee that it will work. And lastly, the only small subset of packages you can rebuild without any changes this way.

In the part 2 I try to explain, how you can re-build packages "consciously", and make them availabe via yum repository on github.

Comments powered by Disqus
Contents © 2013 Roman Imankulov - Powered by Nikola