In this post I demonstrate an effective way to create iterators and generators in PHP and provide an example of a scenario in which using them makes sense.<\/p>\n
Generators<\/a> have been around since PHP 5.5, and iterators<\/a> have been around since the Planck epoch. Even so, a lot of PHP developers do not know how to use them well and cannot recognize situations in which they are helpful. In this blog post I share insights I have gained over the years, that when sharing, always got an interested response from colleague developers. The post goes beyond the basics, provides a real world example, and includes a few tips and tricks. To not leave out those unfamiliar with Iterators the post starts with the “What are Iterators” section, which you can safely skip if you can already answer that question.<\/p>\n PHP has an Why would you bother implementing an Imagine you have a directory with a bunch of text files. One of the files contains an ASCII NyanCat ( We can get all the files by doing a While this approach is decoupled<\/strong>, a big drawback is that now we need to fetch the contents of all files and keep all of that in memory<\/strong> before we even start executing any of our logic. If NyanCat is hiding in the first file, we’ll have fetched the contents of all others for nothing. We can avoid this by using an Our I wrote more about basic Let’s start by looking at PHP’s type hierarchy for collections as of PHP 7.1. These are the core types that I think are most important:<\/p>\n At the very top we have Finally we have Generators are a nice and easy way to create iterators. Often you’ll only loop over them once, and not have any problem. However beware that generators create iterators that are not rewindable, which means that if you loop over them more than once, you’ll get an exception.<\/p>\n Imagine the scenario where you pass in a generator to a service that accepts an instance of The service class in which This blows up if the provided There is a simple and easy way to get around it though. This is where If you call By using For cases where it does not make sense to share a Edit: You can, and probably should, use PHPs native CachingIterator<\/a> instead.<\/p>\n A few months ago I refactored some code part of the Wikimedia\u00a0Deutschland fundraising codebase<\/a>. This code gets the filesystem paths of email templates by looking in a set of specified directories.<\/p>\n This code made the class bound to the filesystem, which made it hard to test. In fact, this code was not tested. Furthermore, this code irked me, since I like code to be on the functional side. The This was refactored using the awesome Much easier to read\/understand code, no state mutation whatsoever, good separation of concerns, easier testing and reusability of this collection building code elsewhere.<\/p>\nWhat are Iterators<\/h3>\n
Iterator<\/code> interface that you can implement to represent a collection. You can loop over an instance of an Iterator<\/code> just like you can loop over an array:<\/p>\nfunction doStuff(Iterator $things) {\n foreach ($things as $thing) { \/* ... *\/ }\n}<\/pre>\nIterator<\/code> subclass rather than just using an array? Let’s look at an example.<\/p>\n~=[,,_,,]:3<\/code>). It is the task of our code to find which file the NyanCat is hiding in.<\/p>\nglob( $path . '*.txt' )<\/code> and we can get the contents for a file with a file_get_contents<\/code>. We could just have a foreach going over the glob result that does the file_get_contents<\/code>. Luckily we realize this would violate separation of concerns and make the “does this file contain NyanCat” logic hard to test since it will be bound to the filesystem access code. Hence we create a function that gets the contents of the files, and ones with our logic in it:<\/p>\nfunction getContentsOfTextFiles(): array {\n \/\/ glob and file_get_contents\n}\n\nfunction findTextWithNyanCat(array $texts) {\n foreach ($texts as $text) { if ( \/* ... *\/ ) { \/* ... *\/ } }\n}\n\nfunction findNyanCat() {\n findTextWithNyanCat(getContentsOfTextFiles());\n}<\/pre>\nIterator<\/code>, as they can fetch their values on demand: they are lazy<\/strong>.<\/p>\nclass TextFileIterator implements Iterator {\n \/* ... *\/\n public function current() {\n \/\/ return file_get_contents\n }\n \/* ... *\/\n}\n\nfunction findTextWithNyanCat(Iterator $texts) {\n foreach ($texts as $text) { if ( \/* ... *\/ ) { \/* ... *\/ } }\n}\n\nfunction findNyanCat() {\n findTextWithNyanCat(new TextFileIterator());\n}<\/pre>\nTextFileIterator<\/code> gives us a nice place to put all the filesystem code, while to the outside just looking like a collection of texts. The function housing our logic, findTextWithNyanCat<\/code>, does not know that the text comes from the filesystem. This means that if you decide to get texts from the database, you could just create a new DatabaseTextBlobIterator and pass it to the logic function without making any changes to the latter. Similarly, when testing the logic function, you can give it an ArrayIterator<\/code>.<\/p>\nfunction testFindTextWithNyanCat() {\n \/* ... *\/\n findTextWithNyanCat(new ArrayIterator(['test text', '~=[,,_,,]:3']));\n \/* ... *\/\n}<\/pre>\nIterator<\/code> functionality in Lazy iterators in PHP and Python<\/a> and Some fun with iterators<\/a>. I also blogged about a library that provides some (Wikidata specific) iterators<\/a> and a CLI tool build around an Iterator<\/a>. For more on how generators work, see the off-site post Generators in PHP<\/a>.<\/p>\nPHP’s collection type hierarchy<\/h3>\n
\n
\n
\n
\n
iterable<\/code>, the supertype of both array<\/code> and Traversable<\/code><\/a>. If you are not familiar with this type or are using a version of PHP older than 7.1, don’t worry, we don’t need it for the rest of this blog post.<\/p>\nIterator<\/code> is the subtype of Traversable<\/code>, and the same goes for IteratorAggregate<\/a><\/code>. The standard library iterator_<\/code> functions such as iterator_to_array<\/code> all take a Traversable<\/code>. This is important since it means you can give them an IteratorAggregate<\/code>, even though it is not an Iterator<\/code>. Later on in this post we’ll get back to what exactly an IteratorAggregate<\/code> is and why it is useful.<\/p>\nGenerator<\/code>, which is a subtype of Iterator<\/code>. That means all functions that accept an Iterator<\/code> can be given a Generator<\/code>, and, by extension, that you can use generators in combination with the Iterator classes in the Standard PHP Library<\/a> such as LimitIterator<\/code> and CachingIterator<\/code>.<\/p>\nIteratorAggregate + Generator = <3<\/h3>\n
Traversable<\/code>:<\/p>\n$aGenerator = function() { \/* ... yield ... *\/ };\n$aService->doStuff($aGenerator());<\/pre>\npublic function doStuff(Traversable $things) {\n foreach ($things as $thing) { \/* ... *\/ }\n}<\/pre>\ndoStuff<\/code> resides does not know it is getting a Generator<\/code>, it just knows it is getting a Traversable<\/code>. When working on this class, it is entirely reasonable to iterate though $things<\/code> a second time.<\/p>\npublic function doStuff(Traversable $things) {\n foreach ($things as $thing) { \/* ... *\/ }\n foreach ($things as $thing) { \/* ... *\/ } \/\/ Boom if Generator!\n}<\/pre>\n$things<\/code> is a Generator<\/code>, because generators are non-rewindable. Note that it does not matter how you iterate through the value. Calling iterator_to_array<\/code> with $things<\/code> has the exact same result as using it in a foreach loop. Most, if not all, generators I have written, do not use resources or state that inherently prevents them from being rewindable. So the double-iteration issue can be unexpected and seemingly silly.<\/p>\nIteratorAggregate<\/code> comes in. Classes implementing IteratorAggregate<\/code> must implement the getIterator()<\/code> method, which returns a Traversable<\/code>. Creating one of these is extremely trivial:<\/p>\nclass AwesomeWords implements IteratorAggregate {\n public function getIterator() {\n yield 'So';\n yield 'Much';\n yield 'Such';\n }\n}<\/pre>\ngetIterator<\/code>, you’ll get a Generator<\/code> instance, just like you’d expect. However, normally you never call this method. Instead you use the IteratorAggregate<\/code> just as if it was an Iterator<\/code>, by passing it to code that expects a Traversable<\/code>. (This is also why usually you want to accept Traversable<\/code> and not just Iterator<\/code>.) We can now call our service that loops over the $things<\/code> twice without any problem:<\/p>\n$aService->doStuff(new AwesomeWords()); \/\/ no boom!<\/pre>\n
IteratorAggregate<\/code> we did not just solve the non-rewindable problem, we also found a good way to share our code. Sometimes it makes sense to use the code of a Generator<\/code> in multiple classes, and sometimes it makes sense to have dedicated tests for the Generator<\/code>. In both cases having a dedicated class and file to put it in is very helpful, and a lot nicer than exposing the generator via some public static function.<\/p>\nGenerator<\/code> and you want to keep it entirely private, you might need to deal with the non-rewindable problem. For those cases you can use my Rewindable Generator<\/a> library, which allows making your generators rewindable by wrapping their creation function:<\/p>\n$aGenerator = function() { \/* ... yield ... *\/ };\n$aService->doStuff(new RewindableGenerator($aGenerator));<\/pre>\nA real-world example<\/h3>\n
private function getMailTemplatesOnDisk(array $mailTemplatePaths): array {\n $mailTemplatesOnDisk = [];\n\n foreach ($mailTemplatePaths as $path) {\n $mailFilesInFolder = glob($path . '\/Mail_*');\n array_walk($mailFilesInFolder, function(&$filename) {\n $filename = basename($filename); \/\/ this would cause problems w\/ mail templates in sub-folders\n } );\n $mailTemplatesOnDisk = array_merge($mailTemplatesOnDisk, $mailFilesInFolder);\n }\n\n return $mailTemplatesOnDisk;\n}<\/pre>\narray_walk<\/code> mutates its by-reference variable and the assignment at the end of the loop mutates the return variable.<\/p>\nIteratorAggregate<\/code> + Generator<\/code> combo:<\/p>\nclass MailTemplateFilenameTraversable implements IteratorAggregate {\n public function __construct(array $mailTemplatePaths) {\n $this->mailTemplatePaths = $mailTemplatePaths;\n }\n\n public function getIterator() {\n foreach ($this->mailTemplatePaths as $path) {\n foreach (glob($path . '\/Mail_*') as $fileName) {\n yield basename($fileName);\n }\n }\n }\n}<\/pre>\n