"Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)"
In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/CocoaPods/Specs/tree/master/Specs ) which every user downloads.
Package management and sync seems to suffer really badly from NIH. Dpkg is over 20 years old and yum is over a decade old. What's up with this particular wheel that people keep reinventing it seemingly without improvement?
Debian's sync may be nicer, but their client-side solution leaves a bit to be desired.
Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes, whereas the same operation takes several seconds on a modern laptop.
It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when uncompressed, comprising more than 41,000 packages, and it has to be parsed and assembled into a dependency tree for every apt operation. This problem screams for SQLite.
A side project I've started looking into is to make a transparent apt proxy which provides a trimmed down Packages.gz (e.g., removing anything which uses X11), which would be a lot easier that rewriting apt to use a SQLite backend.
This is precisely why yum/dnf has been switching from XML for repodata to SQLite. In fact, the only thing that is still XML-only is the comps file which just lists package groups, is updated rarely and "only" weighs in at half a MB.
> Trivial apt operations (e.g. trying to install a package which is already installed) on an NSLU2 (an ancient 266MHz ARM machine) take several minutes
Actually, I'm surprised one can actually run a modern Linux on the NSLU2 given its shameful lack of RAM and slooow USB port. But it was a nice gadget when it came out and it was fun to experiment with it.
> It turns out this is due to the fact that Debian "main" (Packages.gz) has ballooned to 32MB of plain text when uncompressed, comprising more than 41,000 packages, and it has to be parsed and assembled into a dependency tree for every apt operation. This problem screams for SQLite.
Correct me if I'm wrong, but isn't apt (and dpkg) basically composed out of a ton of different (perl/shellscript) modules? So it should be possible to create an interface-compatible sqlite data store.
It isn't just package search which is the problem, it is everything which has to parse the packages file, which is basically every apt command. So if you can trim "main" down to 10,000 packages, suddenly every part of apt is faster, and no one has to install any custom apt replacements.
I believe scaling this could happen with either: 1) lightweight filesystem\directory versioning support, like how btrfs allows you to mount snapshots. This way, peers could update whichever version of a torrent they have. Or 2) very reliable means to update to the latest torrent release (as reliable as syncing with peers), which afaict means smarter bittorrent clients that can perform DHT-based "crawling". Those recent defcon(?) "hacks" to query peers for similar torrents based on user pools and connection histories (or something like that) would make sense here.
A cool side-note: In one of my few experiences diving into `.git`, I diff'd it before and after making changes to its tracked sources, like adding files and modifying them. It looked like a torrent that included version control data would make out just fine if an updated torrent expected similar data in the same location. Again, a smarter bittorrent client would need to sort some of this out. See also 0': Updating Torrents Via Feed URL. Anyway, most users would probably leave that part out, in favor of only which parts they need.
Another cool side-note: This would also allow for easily adding repos from multiple sources... Look at how many ( non-automated :-( ) merge requests com.github/CocoPods/Specs's caregivers have reviewed: 13,331 as of now (0'').
> Arch and Debian contributors have tried a good approach for package management..
> 0: p2pacman - Bittorrent powered pacman wrapper
> 1: pacman & torrent, feasible?
> 2: DebTorrent
That's about distributing packages via p2p. The problematic repository doesn't store any package data, it stores package metadata (it's the cocoapods index if you will).
I see metadata very much as "regular" data (in terms of needs and tooling); practically speaking, even from the same-ish data set. Simply put, it just looks different.. above the surface.
As a data point, "dnf" is the successor to yum. Started using it recently with Fedora 23... and it's pretty decent.
(It may not have been earlier on, I really don't know. ;>)
Something nifty about the new dnf is several of the older yum commands (eg builddep, yum-downloader) are now integrated directly so don't need extra utils installed. Seems like refinement is still happening.
If only my fingers didn't keep typing "dns" instead of "dnf" all the time, it would be great. :D
I just keep typing "yum", since it's an alias of dnf (albeit with an annoying nag message) and since I work on CentOS servers a lot and am automatically used to typing "yum".
Perhaps because Cocoapods is not an OS package manager or anything close to it. It installs libraries within the context of an XCode project, regardless of the host system or what is installed for other projects.
It is a package manager, though? The fundamental idea of downloading a list of available options of which the user picks some, and the system pulls in dependencies, is almost exactly how dpkg and yum work. The location to which the packages are installed is a detail.
A language package manager must be able to "install" the same packages over and over again (and possibly "install" multiple versions of the same package in the same environment), and the ability to push packages is generally considered part of their duty, not so for OS package managers, you don't use dpkg to send a package to debian's repositories.
> The fundamental idea of downloading a list of available options of which the user picks some, and the system pulls in dependencies, is almost exactly how dpkg and yum work.
If you reduce it to the fundamentals you don't need yum or dpkg either to do that, just a dependency solver and curl.
The fundamentals for package management also move the package into it's final resting place(s), where it's going to do it's work. Curl doesn't do that, it just gives you a single file somewhere.
I'd also consider removing a package to be a fundamental part of a manager. The two items you describe would be a 'package grabber'.
CocoaPods is much more akin to RubyGems or PyPI or CPAN, all of which are established as useful tools outside of OS level package managers. There's a need for a iOS/Cocoa package manager (that understands Xcode!) and CocoaPods has so far been the most successful.
A package manager for a project can be the same os package manager with reduced dependency tree and default to install with a prefix (that is the project root or the vendor directory).
Actually it seems very likely that one or more of the popular linux distro package manager ecosystems would fare well on other OSs. Arch Linux's pacman was ported to Windows, e.g..
"Not having to develop a system that somehow syncs required data at all means we get to spend more time on the work that matters more to us, in this case. (i.e. funding of dev hours)"
In other words, using github as a free unlimited CDN lets them be as inefficient as they like. Such as having 16k entries in a directory ( https://github.com/CocoaPods/Specs/tree/master/Specs ) which every user downloads.
Package management and sync seems to suffer really badly from NIH. Dpkg is over 20 years old and yum is over a decade old. What's up with this particular wheel that people keep reinventing it seemingly without improvement?