php check folder for new files
Re: php check folder for new files
https://www.simplicity.be/article/watch ... ories-php/
WATCHING FILES AND DIRECTORIES IN PHP
18 January 2016 7 minutes
Some time ago I got the question if it was possible for a PHP scrip to react on changes within a given folder. The problem was that I knew it was possible using node or the fact that even .NET has an API.aspx) to catch file system events, but I couldn’t come up with a clean solution in PHP that didn’t involved some hackery. Then a couple of days later I had an epiphany…
Reading an article about signals in Linux that was shared on Hacker News, reminded me of the fact that Linux also has a kernel system that fires notices when file system actions are happening. Sometimes my mind needs an indirect jolt to unlock some stored information :-)
Inotify
The Linux subsystem that I’m talking about is called inotify. It was created by a guy called John McCutchan and was merged in the Linux kernel version 2.6.13. It is a replacement for an older system called dnotify.
It is mostly used by desktop utilities that do search. The reason being that its functionality of notifying changes, results in tools that doesn’t need to scan the complete file system over and over again.
Inotify has some CLI tools but (being a Linux kernel system) also includes header files in C, so you can talk with it through code or worst case write a PHP extension around it.
PHP Extension Community Library or just PECL
The first thing you mostly do (or I do) in those situations is to Google for inotify in combination with PHP. And behold I found a function reference to inotify and not really a lot of PHP code unfortunately.
If you look at the description of the function reference it is a clear why that could be. It isn’t really written with findability in mind. You need to be aware of inotify and what it does, ergo the reason why I’m writing this article.
It is a PECL extension so another requirement is that you will need to be able to install and compile PECL extensions or ask your web host to do it for you.
It may sound daunting but on a Debian based system it is just a matter of installing some packages (build-essentials, php5-dev, …) and run
PHP
pecl install inotify
and enable the exension by adding the following in your php.ini
PHP
exension=inotify.so
A simple example
We start of with a simple example where we just watch a given directory for file creation (IN_CREATE) and deletion (IN_DELETE) notices. A complete list of possible flags can be found here.
PHP
#!/usr/local/bin/php
<?php
// directory to watch
$dirWatch = 'watch_dir';
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file is created or deleted in our directory to watch
$watch_id = inotify_add_watch($inoInst, $dirWatch, IN_CREATE | IN_DELETE);
// not the best way but sufficient for this example :-)
while(true){
// read events (
// which is non blocking because of our use of stream_set_blocking
$events = inotify_read($inoInst);
// output data
print_r($events);
}
// stop watching our directory
inotify_rm_watch($inoInst, $watch_id);
// close our inotify instance
fclose($inoInst);
?>
When we run the script and we create a file called helloworld.txt within our watch_dir directory, it will print the following
PHP
Array
(
[0] => Array
(
[wd] => 1
[mask] => 256
[cookie] => 0
[name] => helloworld.txt
)
)
A more complex example
In some cases you want to do a certain action if a file has been deleted or a different action when a file is created. You may also want to watch more than one directory. The following example shows how that can be done.
PHP
#!/usr/local/bin/php
<?php
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file is created or deleted in our directory to watch
$watch_id1 = inotify_add_watch($inoInst, 'watch_dir', IN_CREATE | IN_DELETE);
// add a secton watch to another dir
$watch_id2 = inotify_add_watch($inoInst, 'watch_dir2', IN_CREATE | IN_DELETE);
// not the best way but sufficient for this example :-)
while(true){
// read events
$events = inotify_read($inoInst);
// if the event is happening within our 'watch_dir'
if ($events[0]['wd'] === $watch_id1){
// a file was created
if($events[0]['mask'] === IN_CREATE){
printf("Created file: %s in watch_dir\n", $events[0]['name']);
// a file was deleted
} else if ($events[0]['mask'] === IN_DELETE){
printf("Deleted file: %s in watch_dir\n", $events[0]['name']);
}
// if the event is happening within our 'watch_dir2'
} else if ($events[0]['wd'] === $watch_id2){
// a file was created
if($events[0]['mask'] === IN_CREATE){
printf("Created file: %s in watch_dir2\n", $events[0]['name']);
// a file is deleted
} else if ($events[0]['mask'] === IN_DELETE){
printf("Deleted file: %s in watch_dir2\n", $events[0]['name']);
}
}
}
// stop watching our directories
inotify_rm_watch($inoInst, $watch_id1);
inotify_rm_watch($inoInst, $watch_id2);
// close our inotify instance
fclose($inoInst);
?>
When we create and delete files in our directories that the script is watching, the output could look like this
PHP
Created file: hello.txt in watch_dir2
Deleted file: hello.txt in watch_dir2
Created file: another_hello.txt in watch_dir
Deleted file: another_hello.txt in watch_dir
Bonus: html5 video conversion service
2 years ago around this time, I was working on a problem for a history department of the city of Bruges. They had a big video archive with some videos on-line, but the problem was that they were in low resolution and in Flash.
Fortunately the original footage was available so I needed to batch convert the raw footage to something modern like h.264 video and update the website so it pointed to the new videos. The project was in the end a success, but it involved some “creative” scripting to say the least ;-)
But now with the new knowledge, we can easily create a service that watches a certain folder for video files and convert them automatically every time a new video is uploaded.
It is worth noticing that we are watching for the flag IN_CLOSE_WRITE, the reason being that we only want to start converting when the whole file has been uploaded.
PHP
#!/usr/local/bin/php
<?php
require __DIR__ . '/vendor/autoload.php';
use Symfony\Component\Process\Process;
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file opened for writing was closed in our incoming folder
// you want to start the conversion when the complete file has been uploaded
$watch_id = inotify_add_watch($inoInst, 'incoming', IN_CLOSE_WRITE);
// not the best way but sufficient for this example :-)
while(true){
// read events
$events = inotify_read($inoInst);
// if event happened
if(!empty($events)){
// get filename
$filename = $events[0]['name'];
// be sure that is a quicktime movie
if (pathinfo($filename, PATHINFO_EXTENSION) === "mov"){
// ffmpeg cmd to convert our quicktime movie to html5 video
$ffmpegCmd = sprintf("ffmpeg -i incoming/%s -vcodec h264
-acodec aac -strict -2 processed/%s.mp4", $filename, $filename);
// execute our ffmpeg command
$process = new Process($ffmpegCmd);
$process->run();
}
}
}
// stop watching our directories
inotify_rm_watch($inoInst, $watch_id);
// close our inotify instance
fclose($inoInst);
?>
Limitations
There are some limitations that I need to talk about.
First of all it is a Linux system so it naturally only works on Linux, which isn’t a big deal if you use a typical LAMP stack. Your kernel version should be at least version 2.6.3, but as this was released in 2005 it is safe to say the majority of Linux servers fits the bill :-)
Inotify also doesn’t watch directories recursively. So you will need to add a separate inotify watch for each subdirectory that you want to watch.
It requires that the Linux kernel is aware of all relevant file system events. This means that there are cases in particular when using networked file systems (think NFS), where it may not be immediately always aware of any changes.
Possible improvements
These are use cases that could possible benefit from a framework like ReactPHP. Something that at the moment I don’t have any experience with unfortunately.
You can also easily create a wrapper around inotify that makes it a bit more comprehensible as not everybody knows what Inotify does.
WATCHING FILES AND DIRECTORIES IN PHP
18 January 2016 7 minutes
Some time ago I got the question if it was possible for a PHP scrip to react on changes within a given folder. The problem was that I knew it was possible using node or the fact that even .NET has an API.aspx) to catch file system events, but I couldn’t come up with a clean solution in PHP that didn’t involved some hackery. Then a couple of days later I had an epiphany…
Reading an article about signals in Linux that was shared on Hacker News, reminded me of the fact that Linux also has a kernel system that fires notices when file system actions are happening. Sometimes my mind needs an indirect jolt to unlock some stored information :-)
Inotify
The Linux subsystem that I’m talking about is called inotify. It was created by a guy called John McCutchan and was merged in the Linux kernel version 2.6.13. It is a replacement for an older system called dnotify.
It is mostly used by desktop utilities that do search. The reason being that its functionality of notifying changes, results in tools that doesn’t need to scan the complete file system over and over again.
Inotify has some CLI tools but (being a Linux kernel system) also includes header files in C, so you can talk with it through code or worst case write a PHP extension around it.
PHP Extension Community Library or just PECL
The first thing you mostly do (or I do) in those situations is to Google for inotify in combination with PHP. And behold I found a function reference to inotify and not really a lot of PHP code unfortunately.
If you look at the description of the function reference it is a clear why that could be. It isn’t really written with findability in mind. You need to be aware of inotify and what it does, ergo the reason why I’m writing this article.
It is a PECL extension so another requirement is that you will need to be able to install and compile PECL extensions or ask your web host to do it for you.
It may sound daunting but on a Debian based system it is just a matter of installing some packages (build-essentials, php5-dev, …) and run
PHP
pecl install inotify
and enable the exension by adding the following in your php.ini
PHP
exension=inotify.so
A simple example
We start of with a simple example where we just watch a given directory for file creation (IN_CREATE) and deletion (IN_DELETE) notices. A complete list of possible flags can be found here.
PHP
#!/usr/local/bin/php
<?php
// directory to watch
$dirWatch = 'watch_dir';
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file is created or deleted in our directory to watch
$watch_id = inotify_add_watch($inoInst, $dirWatch, IN_CREATE | IN_DELETE);
// not the best way but sufficient for this example :-)
while(true){
// read events (
// which is non blocking because of our use of stream_set_blocking
$events = inotify_read($inoInst);
// output data
print_r($events);
}
// stop watching our directory
inotify_rm_watch($inoInst, $watch_id);
// close our inotify instance
fclose($inoInst);
?>
When we run the script and we create a file called helloworld.txt within our watch_dir directory, it will print the following
PHP
Array
(
[0] => Array
(
[wd] => 1
[mask] => 256
[cookie] => 0
[name] => helloworld.txt
)
)
A more complex example
In some cases you want to do a certain action if a file has been deleted or a different action when a file is created. You may also want to watch more than one directory. The following example shows how that can be done.
PHP
#!/usr/local/bin/php
<?php
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file is created or deleted in our directory to watch
$watch_id1 = inotify_add_watch($inoInst, 'watch_dir', IN_CREATE | IN_DELETE);
// add a secton watch to another dir
$watch_id2 = inotify_add_watch($inoInst, 'watch_dir2', IN_CREATE | IN_DELETE);
// not the best way but sufficient for this example :-)
while(true){
// read events
$events = inotify_read($inoInst);
// if the event is happening within our 'watch_dir'
if ($events[0]['wd'] === $watch_id1){
// a file was created
if($events[0]['mask'] === IN_CREATE){
printf("Created file: %s in watch_dir\n", $events[0]['name']);
// a file was deleted
} else if ($events[0]['mask'] === IN_DELETE){
printf("Deleted file: %s in watch_dir\n", $events[0]['name']);
}
// if the event is happening within our 'watch_dir2'
} else if ($events[0]['wd'] === $watch_id2){
// a file was created
if($events[0]['mask'] === IN_CREATE){
printf("Created file: %s in watch_dir2\n", $events[0]['name']);
// a file is deleted
} else if ($events[0]['mask'] === IN_DELETE){
printf("Deleted file: %s in watch_dir2\n", $events[0]['name']);
}
}
}
// stop watching our directories
inotify_rm_watch($inoInst, $watch_id1);
inotify_rm_watch($inoInst, $watch_id2);
// close our inotify instance
fclose($inoInst);
?>
When we create and delete files in our directories that the script is watching, the output could look like this
PHP
Created file: hello.txt in watch_dir2
Deleted file: hello.txt in watch_dir2
Created file: another_hello.txt in watch_dir
Deleted file: another_hello.txt in watch_dir
Bonus: html5 video conversion service
2 years ago around this time, I was working on a problem for a history department of the city of Bruges. They had a big video archive with some videos on-line, but the problem was that they were in low resolution and in Flash.
Fortunately the original footage was available so I needed to batch convert the raw footage to something modern like h.264 video and update the website so it pointed to the new videos. The project was in the end a success, but it involved some “creative” scripting to say the least ;-)
But now with the new knowledge, we can easily create a service that watches a certain folder for video files and convert them automatically every time a new video is uploaded.
It is worth noticing that we are watching for the flag IN_CLOSE_WRITE, the reason being that we only want to start converting when the whole file has been uploaded.
PHP
#!/usr/local/bin/php
<?php
require __DIR__ . '/vendor/autoload.php';
use Symfony\Component\Process\Process;
// Open an inotify instance
$inoInst = inotify_init();
// this is needed so inotify_read while operate in non blocking mode
stream_set_blocking($inoInst, 0);
// watch if a file opened for writing was closed in our incoming folder
// you want to start the conversion when the complete file has been uploaded
$watch_id = inotify_add_watch($inoInst, 'incoming', IN_CLOSE_WRITE);
// not the best way but sufficient for this example :-)
while(true){
// read events
$events = inotify_read($inoInst);
// if event happened
if(!empty($events)){
// get filename
$filename = $events[0]['name'];
// be sure that is a quicktime movie
if (pathinfo($filename, PATHINFO_EXTENSION) === "mov"){
// ffmpeg cmd to convert our quicktime movie to html5 video
$ffmpegCmd = sprintf("ffmpeg -i incoming/%s -vcodec h264
-acodec aac -strict -2 processed/%s.mp4", $filename, $filename);
// execute our ffmpeg command
$process = new Process($ffmpegCmd);
$process->run();
}
}
}
// stop watching our directories
inotify_rm_watch($inoInst, $watch_id);
// close our inotify instance
fclose($inoInst);
?>
Limitations
There are some limitations that I need to talk about.
First of all it is a Linux system so it naturally only works on Linux, which isn’t a big deal if you use a typical LAMP stack. Your kernel version should be at least version 2.6.3, but as this was released in 2005 it is safe to say the majority of Linux servers fits the bill :-)
Inotify also doesn’t watch directories recursively. So you will need to add a separate inotify watch for each subdirectory that you want to watch.
It requires that the Linux kernel is aware of all relevant file system events. This means that there are cases in particular when using networked file systems (think NFS), where it may not be immediately always aware of any changes.
Possible improvements
These are use cases that could possible benefit from a framework like ReactPHP. Something that at the moment I don’t have any experience with unfortunately.
You can also easily create a wrapper around inotify that makes it a bit more comprehensible as not everybody knows what Inotify does.