Knapsack problem

src: faculty.chas.uni.edu

The knapsack problem or rucksack problem is a problem in combinatorial optimization: Given a set of items, each with a weight and a value, determine the number of each item to include in a collection so that the total weight is less than or equal to a given limit and the total value is as large as possible. It derives its name from the problem faced by someone who is constrained by a fixed-size knapsack and must fill it with the most valuable items.

The problem often arises in resource allocation where there are financial constraints and is studied in fields such as combinatorics, computer science, complexity theory, cryptography, applied mathematics, and daily fantasy sports.

The knapsack problem has been studied for more than a century, with early works dating as far back as 1897. The name "knapsack problem" dates back to the early works of mathematician Tobias Dantzig (1884-1956), and refers to the commonplace problem of packing the most valuable or useful items without overloading the luggage.

Video Knapsack problem

Applications

A 1998 study of the Stony Brook University Algorithm Repository showed that, out of 75 algorithmic problems, the knapsack problem was the 19th most popular and the 3rd most needed after suffix trees and the bin packing problem.

Knapsack problems appear in real-world decision-making processes in a wide variety of fields, such as finding the least wasteful way to cut raw materials, selection of investments and portfolios, selection of assets for asset-backed securitization, and generating keys for the Merkle-Hellman and other knapsack cryptosystems.

One early application of knapsack algorithms was in the construction and scoring of tests in which the test-takers have a choice as to which questions they answer. For small examples it is a fairly simple process to provide the test-takers with such a choice. For example, if an exam contains 12 questions each worth 10 points, the test-taker need only answer 10 questions to achieve a maximum possible score of 100 points. However, on tests with a heterogeneous distribution of point values--i.e. different questions are worth different point values-- it is more difficult to provide choices. Feuerman and Weiss proposed a system in which students are given a heterogeneous test with a total of 125 possible points. The students are asked to answer all of the questions to the best of their abilities. Of the possible subsets of problems whose total point values add up to 100, a knapsack algorithm would determine which subset gives each student the highest possible score.

Maps Knapsack problem

Definition

The most common problem being solved is the 0-1 knapsack problem, which restricts the number x_i of copies of each kind of item to zero or one. Given a set of n items numbered from 1 up to n, each with a weight w_i and a value v_i, along with a maximum weight capacity W,

maximize ${\displaystyle \sum _{i=1}^{n}v_{i}x_{i}}$

subject to ${\displaystyle \sum _{i=1}^{n}w_{i}x_{i}\leq W}$ and ${\displaystyle x_{i}\in \{0,1\}}$.

Here x_i represents the number of instances of item i to include in the knapsack. Informally, the problem is to maximize the sum of the values of the items in the knapsack so that the sum of the weights is less than or equal to the knapsack's capacity.

The bounded knapsack problem (BKP) removes the restriction that there is only one of each item, but restricts the number ${\displaystyle x_{i}}$ of copies of each kind of item to a maximum non-negative integer value ${\displaystyle c}$:

maximize ${\displaystyle \sum _{i=1}^{n}v_{i}x_{i}}$

subject to ${\displaystyle \sum _{i=1}^{n}w_{i}x_{i}\leq W}$ and ${\displaystyle 0\leq x_{i}\leq c}$

The unbounded knapsack problem (UKP) places no upper bound on the number of copies of each kind of item and can be formulated as above except for that the only restriction on ${\displaystyle x_{i}}$ is that it is a non-negative integer.

maximize ${\displaystyle \sum _{i=1}^{n}v_{i}x_{i}}$

subject to ${\displaystyle \sum _{i=1}^{n}w_{i}x_{i}\leq W}$ and ${\displaystyle x_{i}\geq 0}$

One example of the unbounded knapsack problem is given using the figure shown at the beginning of this article and the text "if any number of each box is available" in the caption of that figure.

src: i.ytimg.com

Computational complexity

The knapsack problem is interesting from the perspective of computer science for many reasons:

• The decision problem form of the knapsack problem (Can a value of at least V be achieved without exceeding the weight W?) is NP-complete, thus there is no known algorithm both correct and fast (polynomial-time) in all cases.
• While the decision problem is NP-complete, the optimization problem is NP-hard, its resolution is at least as difficult as the decision problem, and there is no known polynomial algorithm which can tell, given a solution, whether it is optimal (which would mean that there is no solution with a larger V, thus solving the NP-complete decision problem).
• There is a pseudo-polynomial time algorithm using dynamic programming.
• There is a fully polynomial-time approximation scheme, which uses the pseudo-polynomial time algorithm as a subroutine, described below.
• Many cases that arise in practice, and "random instances" from some distributions, can nonetheless be solved exactly.

There is a link between the "decision" and "optimization" problems in that if there exists a polynomial algorithm that solves the "decision" problem, then one can find the maximum value for the optimization problem in polynomial time by applying this algorithm iteratively while increasing the value of k . On the other hand, if an algorithm finds the optimal value of the optimization problem in polynomial time, then the decision problem can be solved in polynomial time by comparing the value of the solution output by this algorithm with the value of k . Thus, both versions of the problem are of similar difficulty.

One theme in research literature is to identify what the "hard" instances of the knapsack problem look like, or viewed another way, to identify what properties of instances in practice might make them more amenable than their worst-case NP-complete behaviour suggests. The goal in finding these "hard" instances is for their use in public key cryptography systems, such as the Merkle-Hellman knapsack cryptosystem.

src: i.ytimg.com

Solving

Several algorithms are available to solve knapsack problems, based on dynamic programming approach, branch and bound approach or hybridizations of both approaches.

Dynamic programming in-advance algorithm

Unbounded knapsack problem

If all weights (${\displaystyle w_{1},\ldots ,w_{n}}$) are nonnegative integers, the knapsack problem can be solved in pseudo-polynomial time using dynamic programming. The following describes a dynamic programming solution for the unbounded knapsack problem.

To simplify things, assume all weights are strictly positive (${\displaystyle w_{i}>0}$). We wish to maximize total value subject to the constraint that total weight is less than or equal to ${\displaystyle W}$. Then for each ${\displaystyle w\leq W}$, define ${\displaystyle m[w]}$ to be the maximum value that can be attained with total weight less than or equal to ${\displaystyle w}$. Then, ${\displaystyle m[W]}$ is the solution to the problem.

Observe that ${\displaystyle m[w]}$ has the following properties:

• ${\displaystyle m[0]=0\,\!}$ (the sum of zero items, i.e., the summation of the empty set)
• ${\displaystyle m[w]=\max _{w_{i}\leq w}(v_{i}+m[w-w_{i}])}$

where ${\displaystyle v_{i}}$ is the value of the i-th kind of item.

(To formulate the equation above, the idea used is that the solution for a knapsack is the same as the value of one correct item plus the solution for a knapsack with smaller capacity, specifically one with the capacity reduced by the weight of that chosen item.)

Here the maximum of the empty set is taken to be zero. Tabulating the results from ${\displaystyle m[0]}$ up through ${\displaystyle m[W]}$ gives the solution. Since the calculation of each ${\displaystyle m[w]}$ involves examining ${\displaystyle n}$ items, and there are ${\displaystyle W}$ values of ${\displaystyle m[w]}$ to calculate, the running time of the dynamic programming solution is ${\displaystyle O(nW)}$. Dividing ${\displaystyle w_{1},\,w_{2},\,\ldots ,\,w_{n},\,W}$ by their greatest common divisor is a way to improve the running time.

The ${\displaystyle O(nW)}$ complexity does not contradict the fact that the knapsack problem is NP-complete, since ${\displaystyle W}$, unlike ${\displaystyle n}$, is not polynomial in the length of the input to the problem. The length of the ${\displaystyle W}$ input to the problem is proportional to the number of bits in ${\displaystyle W}$, ${\displaystyle \log W}$, not to ${\displaystyle W}$ itself.

0/1 knapsack problem

A similar dynamic programming solution for the 0/1 knapsack problem also runs in pseudo-polynomial time. Assume ${\displaystyle w_{1},\,w_{2},\,\ldots ,\,w_{n},\,W}$ are strictly positive integers. Define ${\displaystyle m[i,w]}$ to be the maximum value that can be attained with weight less than or equal to ${\displaystyle w}$ using items up to ${\displaystyle i}$ (first ${\displaystyle i}$ items).

We can define ${\displaystyle m[i,w]}$ recursively as follows:

• ${\displaystyle m[0,\,w]=0}$
• ${\displaystyle m[i,\,w]=m[i-1,\,w]}$ if ${\displaystyle w_{i}>w\,\!}$ (the new item is more than the current weight limit)
• ${\displaystyle m[i,\,w]=\max(m[i-1,\,w],\,m[i-1,w-w_{i}]+v_{i})}$ if ${\displaystyle w_{i}\leqslant w}$.

The solution can then be found by calculating ${\displaystyle m[n,W]}$. To do this efficiently we can use a table to store previous computations.

The following is pseudo code for the dynamic program:

This solution will therefore run in ${\displaystyle O(nW)}$ time and ${\displaystyle O(nW)}$ space.

Meet-in-the-middle

Another algorithm for 0-1 knapsack, discovered in 1974 and sometimes called "meet-in-the-middle" due to parallels to a similarly named algorithm in cryptography, is exponential in the number of different items but may be preferable to the DP algorithm when ${\displaystyle W}$ is large compared to n. In particular, if the ${\displaystyle w_{i}}$ are nonnegative but not integers, we could still use the dynamic programming algorithm by scaling and rounding (i.e. using fixed-point arithmetic), but if the problem requires ${\displaystyle d}$ fractional digits of precision to arrive at the correct answer, ${\displaystyle W}$ will need to be scaled by ${\displaystyle 10^{d}}$, and the DP algorithm will require ${\displaystyle O(W10^{d})}$ space and ${\displaystyle O(nW10^{d})}$ time.

Meet-in-the-middle algorithm

  input:       a set of items with weights and values    output:       the greatest combined value of a subset    partition the set {1...n} into two sets A and B of approximately equal size    compute the weights and values of all subsets of each set    for each subset of A      find the subset of B of greatest value such that the combined weight is less than W    keep track of the greatest combined value seen so far  

The algorithm takes ${\displaystyle O(2^{n/2})}$ space, and efficient implementations of step 3 (for instance, sorting the subsets of B by weight, discarding subsets of B which weigh more than other subsets of B of greater or equal value, and using binary search to find the best match) result in a runtime of ${\displaystyle O(n2^{n/2})}$. As with the meet in the middle attack in cryptography, this improves on the ${\displaystyle O(n2^{n})}$ runtime of a naive brute force approach (examining all subsets of {1...n}), at the cost of using exponential rather than constant space (see also baby-step giant-step).

Approximation algorithms

As for most NP-complete problems, it may be enough to find workable solutions even if they are not optimal. Preferably, however, the approximation comes with a guarantee on the difference between the value of the solution found and the value of the optimal solution.

As with many useful but computationally complex algorithms, there has been substantial research on creating and analyzing algorithms that approximate a solution. The knapsack problem, though NP-Hard, is one of a collection of algorithms that can still be approximated to any specified degree. This means that the problem has a polynomial time approximation scheme. To be exact, the knapsack problem has a fully polynomial time approximation scheme (FPTAS).

Greedy approximation algorithm

George Dantzig proposed a greedy approximation algorithm to solve the unbounded knapsack problem. His version sorts the items in decreasing order of value per unit of weight, ${\displaystyle v_{i}/w_{i}}$. It then proceeds to insert them into the sack, starting with as many copies as possible of the first kind of item until there is no longer space in the sack for more. Provided that there is an unlimited supply of each kind of item, if ${\displaystyle m}$ is the maximum value of items that fit into the sack, then the greedy algorithm is guaranteed to achieve at least a value of ${\displaystyle m/2}$. However, for the bounded problem, where the supply of each kind of item is limited, the algorithm may be far from optimal.

Fully polynomial time approximation scheme

The fully polynomial time approximation scheme (FPTAS) for the knapsack problem takes advantage of the fact that the reason the problem has no known polynomial time solutions is because the profits associated with the items are not restricted. If one rounds off some of the least significant digits of the profit values then they will be bounded by a polynomial and 1/? where ? is a bound on the correctness of the solution. This restriction then means that an algorithm can find a solution in polynomial time that is correct within a factor of (1-?) of the optimal solution.

An Algorithm for FPTAS

 input:      ? ? (0,1]     a list A of n items, specified by their values, ${\displaystyle v_{i}}$, and weights   output:     S' the FPTAS solution  

 P := max ${\displaystyle \{v_{i}\mid 1\leq i\leq n\}}$  // the highest item value   K := ? ${\displaystyle {\frac {P}{n}}}$   for i from 1 to n do      ${\displaystyle v'_{i}}$ := ${\displaystyle \left\lfloor {\frac {v_{i}}{K}}\right\rfloor }$   end for   return the solution, S', using the ${\displaystyle v'_{i}}$ values in the dynamic program outlined above  

Theorem: The set ${\displaystyle S'}$ computed by the algorithm above satisfies ${\displaystyle \mathrm {profit} (S')\geq (1-\varepsilon )\cdot \mathrm {profit} (S^{*})}$, where ${\displaystyle S^{*}}$ is an optimal solution.

Dominance relations

Solving the unbounded knapsack problem can be made easier by throwing away items which will never be needed. For a given item i, suppose we could find a set of items J such that their total weight is less than the weight of i, and their total value is greater than the value of i. Then i cannot appear in the optimal solution, because we could always improve any potential solution containing i by replacing i with the set J. Therefore, we can disregard the i-th item altogether. In such cases, J is said to dominate i. (Note that this does not apply to bounded knapsack problems, since we may have already used up the items in J.)

Finding dominance relations allows us to significantly reduce the size of the search space. There are several different types of dominance relations, which all satisfy an inequality of the form:

${\displaystyle \qquad \sum _{j\in J}w_{j}\,x_{j}\ \leq \alpha \,w_{i}}$, and ${\displaystyle \sum _{j\in J}v_{j}\,x_{j}\ \geq \alpha \,v_{i}\,}$ for some ${\displaystyle x\in Z_{+}^{n}}$

where ${\displaystyle \alpha \in Z_{+}\,,J\subsetneq N}$ and ${\displaystyle i\not \in J}$. The vector ${\displaystyle x}$ denotes the number of copies of each member of J.

Collective dominance

The i-th item is collectively dominated by J, written as ${\displaystyle i\ll J}$, if the total weight of some combination of items in J is less than w_i and their total value is greater than v_i. Formally, ${\displaystyle \sum _{j\in J}w_{j}\,x_{j}\ \leq w_{i}}$ and ${\displaystyle \sum _{j\in J}v_{j}\,x_{j}\ \geq v_{i}}$ for some ${\displaystyle x\in Z_{+}^{n}}$, i.e. ${\displaystyle \alpha =1}$. Verifying this dominance is computationally hard, so it can only be used with a dynamic programming approach. In fact, this is equivalent to solving a smaller knapsack decision problem where ${\displaystyle V=v_{i}}$, ${\displaystyle W=w_{i}}$, and the items are restricted to J.

Threshold dominance

The i-th item is threshold dominated by J, written as ${\displaystyle i\prec \prec J}$, if some number of copies of ${\displaystyle i}$ are dominated by J. Formally, ${\displaystyle \sum _{j\in J}w_{j}\,x_{j}\ \leq \alpha \,w_{i}}$, and ${\displaystyle \sum _{j\in J}v_{j}\,x_{j}\ \geq \alpha \,v_{i}\,}$ for some ${\displaystyle x\in Z_{+}^{n}}$ and ${\displaystyle \alpha \geq 1}$. This is a generalization of collective dominance, first introduced in and used in the EDUK algorithm. The smallest such ${\displaystyle \alpha }$ defines the threshold of the item ${\displaystyle i}$, written ${\displaystyle t_{i}=(\alpha -1)w_{i}}$. In this case, the optimal solution could contain at most ${\displaystyle \alpha -1}$ copies of ${\displaystyle i}$.

Multiple dominance

The i-th item is multiply dominated by a single item ${\displaystyle j}$, written as ${\displaystyle i\ll _{m}j}$, if ${\displaystyle i}$ is dominated by some number of copies of ${\displaystyle j}$. Formally, ${\displaystyle w_{j}\,x_{j}\ \leq w_{i}}$, and ${\displaystyle v_{j}\,x_{j}\ \geq v_{i}}$ for some ${\displaystyle x_{j}\in Z_{+}}$ i.e. ${\displaystyle J=\{j\},\alpha =1,x_{j}=\lfloor {\frac {w_{i}}{w_{j}}}\rfloor }$. This dominance could be efficiently used during preprocessing because it can be detected relatively easily.

Modular dominance

Let b be the best item, i.e. ${\displaystyle {\frac {v_{b}}{w_{b}}}\geq {\frac {v_{i}}{w_{i}}}\,}$ for all ${\displaystyle i}$. This is the item with the greatest density of value. The i-th item is modularly dominated by a single item ${\displaystyle j}$, written as ${\displaystyle i\ll _{\equiv }j}$, if ${\displaystyle i}$ is dominated by ${\displaystyle j}$ plus several copies of b. Formally, ${\displaystyle w_{j}+tw_{b}\leq w_{i}}$, and ${\displaystyle v_{j}+tv_{b}\geq v_{i}}$ i.e. ${\displaystyle J=\{b,j\},\alpha =1,x_{b}=t,x_{j}=1}$.

src: slideplayer.com

Variations

There are many variations of the knapsack problem that have arisen from the vast number of applications of the basic problem. The main variations occur by changing the number of some problem parameter such as the number of items, number of objectives, or even the number of knapsacks.

Multi-objective knapsack problem

This variation changes the goal of the individual filling the knapsack. Instead of one objective, such as maximizing the monetary profit, the objective could have several dimensions. For example, there could be environmental or social concerns as well as economic goals. Problems frequently addressed include portfolio and transportation logistics optimizations.

As an example, suppose you ran a cruise ship. You have to decide how many famous comedians to hire. This boat can handle no more than one ton of passengers and the entertainers must weigh less than 1000 lbs. Each comedian has a weight, brings in business based on their popularity and asks for a specific salary. In this example you have multiple objectives. You want, of course, to maximize the popularity of your entertainers while minimizing their salaries. Also, you want to have as many entertainers as possible.

Multi-dimensional knapsack problem

In this variation, the weight of knapsack item ${\displaystyle i}$ is given by a D-dimensional vector ${\displaystyle {\overline {w_{i}}}=(w_{i1},\ldots ,w_{iD})}$ and the knapsack has a D-dimensional capacity vector ${\displaystyle (W_{1},\ldots ,W_{D})}$. The target is to maximize the sum of the values of the items in the knapsack so that the sum of weights in each dimension ${\displaystyle d}$ does not exceed ${\displaystyle W_{d}}$.

Multi-dimensional knapsack is computationally harder than knapsack; even for ${\displaystyle D=2}$, the problem does not have EPTAS unless P${\displaystyle =}$NP. However, the algorithm in is shown to solve sparse instances efficiently. An instance of multi-dimensional knapsack is sparse if there is a set ${\displaystyle J=\{1,2,\ldots ,m\}}$ for ${\displaystyle m<D}$ such that for every knapsack item ${\displaystyle i}$, ${\displaystyle \exists z>m}$ such that ${\displaystyle \forall j\in J\cup \{z\},\ w_{ij}\geq 0}$ and ${\displaystyle \forall y\notin J\cup \{z\},w_{iy}=0}$. Such instances occur, for example, when scheduling packets in a wireless network with relay nodes. The algorithm from also solves sparse instances of the multiple choice variant, multiple-choice multi-dimensional knapsack.

The IHS (Increasing Height Shelf) algorithm is optimal for 2D knapsack (packing squares into a two-dimensional unit size square): when there are at most five square in an optimal packing.

Multiple knapsack problem

This variation is similar to the Bin Packing Problem. It differs from the Bin Packing Problem in that a subset of items can be selected, whereas, in the Bin Packing Problem, all items have to be packed to certain bins. The concept is that there are multiple knapsacks. This may seem like a trivial change, but it is not equivalent to adding to the capacity of the initial knapsack. This variation is used in many loading and scheduling problems in Operations Research and has a Polynomial-time approximation scheme.

Quadratic knapsack problem

As described by Wu et al.:

The quadratic knapsack problem (QKP) maximizes a quadratic objective function subject to a binary and linear capacity constraint.

The quadratic knapsack problem was discussed under that title by Gallo, Hammer, and Simeone in 1980. However, Gallo and Simeone attribute the first treatment of the problem to Witzgall in 1975.

Subset-sum problem

The subset sum problem is a special case of the decision and 0-1 problems where each kind of item, the weight equals the value: ${\displaystyle w_{i}=v_{i}}$. In the field of cryptography, the term knapsack problem is often used to refer specifically to the subset sum problem and is commonly known as one of Karp's 21 NP-complete problems.

The generalization of subset sum problem is called multiple subset-sum problem, in which multiple bins exist with the same capacity. It has been shown that the generalization does not have an FPTAS.

src: i.ytimg.com

In popular culture

• Neal Stephenson provides an example of the knapsack problem in chapter 70 of his novel Cryptonomicon to distribute family heirlooms.
• The knapsack problem occurs commonly in role-playing games, both digital and paper-based (prominent examples include The Elder Scrolls series and the Dungeons and Dragons game, respectively), where the player character is constrained by their encumbrance threshold when carrying items and treasure, which regularly forces the player to evaluate the items' value-to-weight ratio in order to bring only the most value-dense items to a merchant.
• Web comic xkcd #287 - NP-Complete
• In Charles Stross' Accelerando, the main character Manfred refers to the 'blind knapsack problem' in chapter 2, presumably a generalization or more complex version of the regular knapsack problem.

src: slideplayer.com

See also

src: i.ytimg.com

Notes

src: slideplayer.com

References

• Garey, Michael R.; David S. Johnson (1979). Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman. ISBN 0-7167-1045-5.  A6: MP9, pg.247.
• Kellerer, Hans; Pferschy, Ulrich; Pisinger, David (2004). Knapsack Problems. Springer. doi:10.1007/978-3-540-24777-7. ISBN 3-540-40286-1. MR 2161720. 
• Martello, Silvano; Toth, Paolo (1990). Knapsack problems: Algorithms and computer implementations. Wiley-Interscience. ISBN 0-471-92420-2. MR 1086874. 

src: i.ytimg.com

External links

• Free download of the book "Knapsack problems: Algorithms and computer implementations", by Silvano Martello and Paolo Toth
• Lecture slides on the knapsack problem
• PYAsUKP: Yet Another solver for the Unbounded Knapsack Problem, with code taking advantage of the dominance relations in an hybrid algorithm, benchmarks and downloadable copies of some papers.
• Home page of David Pisinger with downloadable copies of some papers on the publication list (including "Where are the hard knapsack problems?")
• Knapsack Problem solutions in many languages at Rosetta Code
• Dynamic Programming algorithm to 0/1 Knapsack problem
• Knapsack Problem solver (online)
• Solving 0-1-KNAPSACK with Genetic Algorithms in Ruby
• Codes for Quadratic Knapsack Problem

Source of the article : Wikipedia