Copying Debian package selections to a new machine

Copying Debian package selections to a new machine


Most of us will install our GNU/Linux system once or twice and then use the excellent package management systems to upgrade when new releases of our chosen distribution come out. Users of Debian and Debian-based systems (such as Ubuntu) will be quite used to the idea that you only need to install it once. But what happens when you want to replicate one Debian system on another machine? Do you use cloning tools? Yes you can but only if the hardware is similar on the two machines. What if one has an Intel Pentium-based processor and the other has an AMD64? In that case what you need is some way to replicate the package selection but use the appropriate ones for the new architecture. Enter dpkg.

dpkg?

dpkg is one of the core applications of Debian and essential to its package management. Whatever you use to manage packages will in fact use dpkg in the background to install, remove, upgrade and handle post-installation processes. So whether you use apt, aptitude, synaptic or one of the others, chances are dpkg is involved somewhere. It has some nice features but, to be honest, is cumbersome for day to day use which is why there are clients on top of it. In particular dpkg does nothing about dependencies - leaving that to apt or whatever you use. It's a shell-based tool so if you consider yourself to be part of what some people refer to as "the rest of us", you might have to get you hands onto the keyboard more than you are used to. I'm afraid there isn't really a point and click version of this but the process I'm describing is not complex. It just involves typing.

The process

Simply put the process has four steps:

  1. install and setup your first Debian system as you prefer
  2. make a list of the packages installed
  3. install a basic Debian system on the new machine
  4. use the list of packages from step 2 to complete the process.

There is also a step 0 which should go without saying: backup before you begin. I am not responsible if you hose your system trying this out and lose your data

I'm going to describe the process from a Debian point of view. I am sure this can work on Debian-based systems such as Ubuntu but I don't have enough experience of installing those to speak about installing minimal systems etc. There is also a step 0 which should go without saying: backup before you begin. I am not responsible if you hose your system trying this out and lose your data.

Step 1 - install and setup your first Debian system

I suspect many of you will have done the first step over time. Sometimes you will have removed some packages which are part of the base install but aren't crucial to your machine. For example I've often purged a lot of the additional xserver-xorg-video- packages as they are graphics drivers for hardware I don't have on that machine. Using this process will save me having to do that each time I setup a Debian box.

Step 2 - make a list of the packages installed

The second step is where dpkg comes in. Fire up a terminal and switch to root using su. Then run this:

dpkg --get-selections

This command will present with a long list of installed packages. Obviously I'm not expecting you to write them down -- this is the shell we can redirect the output into a text file. Try this:

dpkg --get-selections > /tmp/package_list

This command gets the same list of installed packages and puts them into a file: /tmp/package_list. Note that custom configuration is not saved, so your new system will need those same setups applied. Save this file to a USB key/flash drive or similar as we'll be needing it later.

Step 3 - install a basic Debian system on the new machine

I'm not going to go into how to install Debian on a new machine -- there's plenty of places describing how to do that, but you need to ensure you install the most minimal system you can. This means not selecting any tasks during the installation. When you install a new Debian system, you are at one stage presented with a list of "tasks" which aid the installation of packages. For example there will be Desktop Environment, Mail Server, Print Server etc. This is one of the great things about Debian in that groups of packages required for a specific type of system are grouped and ready for you to install. In this instance we don't want any of those, so when you get to this stage deselect everything - including the Standard System one. This will result in a bare-bones Debian system.

Do not select any of the tasks during this step in the installationDo not select any of the tasks during this step in the installation

Incidentally when installing for this purpose I tend to use one of the Debian "business-card" .iso's. These are minimal (35Mb) iso's which then get the remaining required packages via a net connection.

Step 4 - use the package list we created in Step 2

Once you have installed your new system and rebooted you should be looking at a login prompt. Login as root and perform an upgrade -- just in case:

apt-get update
apt-get upgrade

Now we need to tell our new system what packages we require. This again is where dpkg comes in. Put your USB key/flash drive into the new machine and mount it. Remember that as we have a minimal system automounting won't work. Again I won't go into details here about how to mount USB drives. Once the drive is mounted and you have found the packages_list file, run this:

dpkg --set-selections < /path/to/packages_list
apt-get -u dselect-upgrade

The first command uses dpkg to preselect the packages in the packages_list file. The second command tells apt-get to upgrade the system using the packages supplied by dpkg. It might take a while but by the end of the process you should have a new system with the same packages as the first one. The best part is that dependencies will be retrieved and installed as well.

This process is quite flexible and while it won't be the apt-get undo that Rosalyn Hunter longs for it might help. You could grab the package list of a hosed Sid system and then use that on a new Testing system. I wouldn't recommend going too far that direction though. Going from Testing to Stable might not work as well because some of the Testing packages may not exist in Stable. You could also have the dpkg --get-selections > /tmp/package_list command run as a cron job or run it immediately before an apt-get dist-upgrade and thus if your system gets really hosed you can backup your data and start afresh confident that you can install the same packages again easily.

As I said I wouldn't use this if I had a lot of identical machines to install -- that's where imaging and cloning comes into its own. But I would and do use this to setup similar machines or reinstall a Debian system after installing a new HDD for example particularly if I wanted to get the latest versions of the packages I was previously using.

Category: 

Comments

maruadventurer's picture

Mr. Cartwright,

Good article. Couple of points. --

1) Its obvious that this could be scripted. One for grabbing the selections. Second for deploying to the new machine.

2) Cloning is great for that all 30 units are twins. But guess what? More often than not what is labeled as a twin is not. Dell in particular is fond of switching peripherals in the same production run, substituting a Deskstar for a Seagate drive. Or a different MB entirely on some occasions. So I drop back to the get-selections routine when that happens.

On a batch of 40, I might have 2-3 that fail a successful cloning.

3) Unless one is running Debian one might consider making sure you have the latest OS rev before going thru all the trouble of creating a series of clones.

Ryan Cartwright's picture

Its obvious that this could be scripted. One for grabbing the selections. Second for deploying to the new machine.

Indeed and I have done this. In fact if you are doing a few then scripting is part of the process.

Cloning is great for that all 30 units are twins. But guess what? More often than not what is labeled as a twin is not. Dell in particular is fond of switching peripherals in the same production run, substituting a Deskstar for a Seagate drive. Or a different MB entirely on some occasions. So I drop back to the get-selections routine when that happens.

Sensible. I did mention that cloning only works if the hardware is identical. In my experience relying on what the supplier says is not always best. For example I have seen two graphics cards with identical "version numbers" that used entirely different chipsets!

Unless one is running Debian one might consider making sure you have the latest OS rev before going thru all the trouble of creating a series of clones.

I wasn't really talking about cloning here but the extra info is appreciated and this piece really is about Debian. I guess it will work on Debian-based machines but I've not tried it.

--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

ajt's picture
Submitted by ajt on

On a modern Debian system you really should be using aptitude not apt-get. Can you do the same thing with aptitude instead of apt-get?

--
It's not magic, it's work!

andyjpb's picture
Submitted by andyjpb on

Hi,

> On a modern Debian system you really should be using aptitude not apt-get.
> Can you do the same thing with aptitude instead of apt-get?

This is not the case. The two tools serve different purposes. Whilst aptitude is good for day-to-day package management, it's not so good in this instance. The way the two tools calculate package dependencies is different and, AIUI, aptitude by default will install "recommended" packges as well as "required" packages.

In this particular case, one should use apt-get.

For more information see http://jpb.li/gYWIw

Regards,
@ndy

Gary Richmond's picture

Ryan,

You must be psychic! I will be installing Meerkat soon and I was planning the "strategy" for it. I vaguely recalled some while back that there was a series of commands using apt-get to replicate and upgrade package selections from one machine to another, but I couldn't remember the specifics.

A Google session would have turned up the stuff I needed but you've come along just at the right time and saved me all that trouble. Great article. Thanks.

One of your readers has speculated as to whether or not Aptitude could be used instead. I don't know if it can be or not, but even if it is possible I suspect it would be easier just to use apt-get. As a command-line interface it is uncluttered and relatively simple. Ncurses-based Aptitude is, in my opinion, lethally unfriendly and not very intuitive and anytime I have fired it up I have shied away uncertain and slightly nervous that I might hose something important. I'll stick will the venerable apt-get. Hard to beat.

Ryan Cartwright's picture

You must be psychic! I will be installing Meerkat soon and I was planning the "strategy" for it.

This piece has been knocking around on my machine for a while. Roslyn's woes prompted me to finish it. I'm guessing this is installing Meerkat on a different machine and not upgrading from 8.04?

--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

r_a_trip's picture
Submitted by r_a_trip on

@Gary Richmond

Aptitude will start in curses mode if you don't give it any parameters. It threw me off as well before I knew that I could use it as a pure CLI command by simply giving it parameters.

aptitude update && aptitude full-upgrade will yield the same result as apt-get update && apt-get dist-upgrade

# touch universe
# chmod +rwx universe
# ./universe

Ryan Cartwright's picture

Thanks for the comments all.

with regards Aptitude, yes it can be used on the command line aptitude update; aptitude upgrade for example. Aptitude from the shell (rather than curses) is quite useful and has some nice features. Namely suggesting what to do in times of dependency issues and also giving you a one key press chance to enact one of those options. It also tidies up after itself quite nicely.

But aptitude upgrade has one big difference to apt-get upgrade. The former will remove packages whereas the latter will hold them back until you do a dist-upgrade. This means if you are doing a big upgrade, although aptitude lists the packages to be removed the list can be missed in the deluge of text and before you know it you have hit Y and hosed your system.

I disagree that a "modern" Debian system should use aptitude mostly because I don't know what "modern" means. There are many that say modern systems use GUI tools only, I disagree there too but it proves my point. One person's "modern" is another person's "new fangled". I've used aptitude (not the curses interface though) and Synaptic and whatever the Kubuntu package manager is but it all depends upon the system I am on, what I have available and how much time I have. In the end it's "horses for courses" - although I recently discovered that a lot of people younger than me have no idea what that means!

In this case though aptitude is not suitable as it doesn't support dselect.

cheers
Ryan

--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

ajt's picture
Submitted by ajt on

I should have been more precise, apt-get is formally deprecated and aptitude is now preferred. I agree that using the word "modern" is loaded and I shouldn't have said that.

However it's still a great idea even if it only works with apt-get.

--
It's not magic, it's work!

Ryan Cartwright's picture

Interesting. "Deprecated" has significant connotations. It implies the package is to disappear altogether in the near future. Do you have a link where apt-get is announced as formally deprecated? All I can find is that aptitude is now the recommended command-line tool over apt-get.

Dselect is no longer recommended but I can't find a suitable alternative for it. Personally I prefer apt-get over aptitude for the differences in "upgrade" I gave above. Yes I know aptitude has safe-upgrade but I'm just used to apt-get.

cheers
Ryan
--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

Gary Richmond's picture

Ryan,

Yes, just to confirm. The Meerkat install will be on a different machine. Thanks to you and r_a_trip for the heads up on using Aptitude on the command line. I should have realised that was an option. This is GNU/Linux for Heaven's sake!

Well, I've backed up a list of the installed packages on my version of Ubuntu courtesy of your piece and copied it to my USB stick and reviewed the contents in Gedit. So far so fine.

Obviously, the fine detail of your tutorial will work for Debian but as I've never tried this before on Ubuntu I might need to tack a little here. I'm not (yet) aware of any minimal version of Ubuntu that I could use to do a "barebones" install. I suspect that if I install a 700MB disc of Maverick Meerkat, even before I do any update(s) I will have software more up to date than my saved package list. It will be interesting to see what happens if I try to install that list.

Additionally, although I spotted the Opera browser in that list of packages for installation I know that I installed it as a stand alone binary for my particular distro from the Opera website. It's not something you will find in the official Ubuntu repositories so I suspect I will have to save it (and other stuff too) and install each package by hand. and then there is the small matter of migrating e-mail but as my last article on Thunderbird, Evolution and Gmail explained there are strategies for dealing with that.

Yes, this upgrade/migration to a fresh install on another machine will prove to be very interesting indeed! Methinks I will need to put pen to paper and make a detailed list of all the things I need to do. However, the beauty of your apt-get commands is that if it all go pear shaped I have the package list on the stick and can do another fresh install again and again until I get it right.

Gary Richmond's picture

Further to my last comment re the above, I think I may have stumbled across some extra goodies to improve the process of package installation.

I keep a small stash of back issues of Linux Format Magazine in the smallest room in the house (well, I like to make the most productive use of my time).

This morning I read a very interesting article by Neil Borthwick about strategies for switching distros. Using your instructions to redirect the Debian package list to a text file for later reinstallation on another machine worked fine but I noticed that it included a lot of library and development files as well as the programmes and distro-dependent dependencies too.

These may not be need in a fresh install, cause clutter and possible problems. Borthwick suggests that the best path to take is to exclude all this stuff and just let the package manager handle all the libraries and dependencies instead. That makes sense to me. He suggests the following command:

dpkg --list | grep -v -e '-dev' -e '^ii lib' >packages.txt

He also suggests a similar strategy to cover the binary packages which usually install to their own directory on /opt. So:

ls -l /opt >binary_packages.txt

If you've installed anything from source with ./configure, make and make install just do

ls -l /usr/local/bin >source_packages.txt

(assuming you haven't been using PREFIX with ./configure).

I tested all three commands and they worked fine.

Ryan Cartwright's picture

Some good points Gary. Remember the aim here is not so much to resolve installing a second machine with a different distro or even the same one. It's for a replacement machine or possibly a clone setup.

Remember that this method does not specify versions of the packages. It presumes you want the latest ones. That means the dependencies will be handled by apt. Yes you will be specifying a lot of the libraries anyway with the package list but apt will deal with any dependencies you missed. Any libraries or packages that were dependencies but are no longer required can be handled with aptitude (on the command line of course).

Ryan
--
Equitas IT Solutions - fairness, quality, freedom
http://www.equitasit.co.uk

Author information

Ryan Cartwright's picture

Biography

Ryan Cartwright heads up Equitas IT Solutions who offer fair, quality and free software based solutions to the voluntary and community (non-profit) and SME sectors in the UK. He is a long-term free software user, developer and advocate. You can find him on Twitter and Identi.ca.