nav-left cat-right
cat-right

Installing Gearman and gearmand on Windows with Cygwin

Recently I've come face-to-face with a significant processing task for a web application written in PHP.  I haven't worked with process control very much, so I started researching ways of distributing the calculations to multiple processes.  PHP offers several libraries for doing this (pcntl, POSIX), but it quickly became clear that if you're running Windows these libraries are not an option, and unfortunately at work I have a Windows machine.  After a lot more research, I came across Gearman.

Gearman is essentially a distributed processing framework, and seems to have community support for many programming languages.  It consists of two main components: the job server, and a Client and Worker API.  The Client and Worker API can be used in a wide variety of languages, but the job server is only available as a C library or a Perl library.  This makes it a bit tougher to get the server running on Windows, especially when you start running into some of the dependencies that it requires to build.  As well, the Client/Worker API for PHP can only be installed as a PECL extension, or a very-out-of-date PEAR extension called Net_Gearman.

Nonetheless, after yet more research I decided that I would give it a shot by using Cygwin to get the job server running (if you haven't used Cygwin before, be sure to read about it before attempting to install Gearman this way), and PEAR to use the API.  Pre-built PECL extensions aren't available for Windows anymore, and the build process for PHP extensions can be pretty painful, so it makes PEAR look good by comparison even if the code will be out of date.

I had a pretty frustrating time finally getting everything up and running due to various dependency issues, so I went back through the whole process and wrote it out step-by-step.  I used a Windows XP SP3 machine for this, but I also got it working on a Windows 7 machine as well.

Installing the Gearman job server (gearmand) on Windows with Cygwin

Installing Cygwin

  1. If you don't have Cygwin already, you can get it from http://www.cygwin.com.  The setup file is located here, and the setup process is pretty straightforward; run it and follow the wizard.  Full installation instructions are available at the Cygwin site.
  2. Keep the Cygwin setup.exe file handy after you've installed the default software packages, as you'll need it in the future to add packages, similar to apt-get, yum, and other Linux/UNIX package managers.
  3. Cygwin installs with some basic packages, including a Cygwin Bash Shell that goes into your Start Menu.  I prefer the mintty emulator instead, as it has less of a DOS Command Prompt feel and better terminal features.  Feel free to use whatever shell you like of course.  You can get mintty by re-running the setup.exe, and at the package selection screen, type 'mintty' into the Search bar at the top left.  Expand the "Shells" category, and click on the word "Skip" under the "New" column beside the mintty package to select it before continuing the install process.

Installing Cygwin Package Dependencies needed for Gearman

If you're not already in the Cygwin setup, re-run the Cygwin setup.exe and go through to the package selection screen.  The following is a list of dependency packages you will need in order to build the Gearman job server (gearmand).  None of these packages were installed by default with Cygwin:

  • gcc
  • make
  • libuuid1-devel
  • libiconv

There's a good installation tutorial here that walks through getting gcc and make installed for people unfamiliar with Cygwin.  Finding the others is pretty straightforward, the Search bar in the package selector works well.

Installing libevent

Gearmand requires an event notification library called libevent that you cannot get as a Cygwin package, which means it has to be installed from source.  You can get the source here.

  1. Download and unpack the latest libevent stable release.  At the time of this writing, I used libevent-1.4.14b-stable.
    NOTE: Download and unpack to a directory that does not contain spaces in the name, such as "C:/cygwin/home/Zombat/libevent-1.4.14b-stable".  If you unpack to something with spaces like "C:/Documents and Settings/Zombat/", the build process might not be able to install libevent correctly (libtool has a hard time with spaces)!
  2. Open a Cygwin shell and cd to the unpacked libevent directory.
  3. Run the following commands:

./configure
make
make install

libevent should now be installed and ready to be used when compiling the Gearman job server.

Installing the Gearman job server, gearmand.exe

  1. Download and unpack the C implementation of gearmand from http://gearman.org/index.php?id=download
  2. Open a cygwin shell and cd to your unpacked gearmand directory.  Same rules apply as before, make sure you've unpacked in a directory with no spaces in the path!  libtool hates that, and your build may fail.
  3. Run the following commands:

./configure
make
make install

The Gearman job server should now be installed and ready to use!  Mine was installed at /usr/local/sbin/gearmand.exe, and running it with a "triple verbose" flag (-vvv) should produce the following:

gearmand.exe startup debug output

That's it for the job server.  When you want to start it, simply open a Cygwin shell and run gearmand.exe.  Running it with the -d flag will cause the server to run as a daemon in the background, and running with --help will show you the full option list.

Installing the Gearman Client and Worker API (Net_Gearman)

I chose to install the PEAR Client and Worker API, as it is native PHP and doesn't involve compiling PECL extensions.  The PEAR package is called Net_Gearman, and was originally written by Joe Stump at Digg.com.  It is old and out of date now, although there appears to be a more recent fork at http://github.com/brianlmoon/net_gearman.  I stuck with the older version, as I suspect it will meet my needs, and was readily available as a PEAR package.

This also makes installation relatively painless.  Assuming you've previously set PEAR up, then all you have to do is open a command window (not a Cygwin shell) and run:

pear install Net_Gearman-alpha

The "-alpha" portion is necessary, as Net_Gearman apparently never made it to a stable release version.  That being said, it has functioned well for me so far.  Perhaps someone will pick the project up in the future.

I'll write more about getting started with the Client and Worker API in the next article, so we can actually use Gearman to get some work done.

Be Sociable, Share!

21 Responses to “Installing Gearman and gearmand on Windows with Cygwin”

  1. Güncel Blog says:

    thanks amına koyam

  2. David says:

    Thank you for your tutorial.
    Can you tell me how to run Net_Gearman?

    Thanks,
    David

  3. blake says:

    Hi David,

    I'm planning on doing a follow-up article at some point on how to utilize the basics of Net_Gearman once you've gotten the server up and running. Haven't had a chance to do it yet.

  4. James says:

    Great article but i am running into some problems with libevent. It looks like I might need to compile it as a shared library? I am using the verion you specified in your article. Any ideas? Thanks!

    *** Warning: This system can not link to static lib archive /usr/local/lib/libev
    ent.la.
    *** I have the capability to make that library automatically link in when
    *** you link to this library. But I can only do this if you have a
    *** shared version of the library, which you do not appear to have.
    CC libtest/server.lo
    CCLD libtest/libserver.la
    CC libtest/test.lo
    CCLD libtest/libtest.la
    CXX bin/arguments.o
    CXX bin/function.o
    CXX bin/gearman.o
    CXX util/error.o
    CXX util/pidfile.o
    CXXLD bin/gearman.exe
    util/error.o:/cygdrive/c/cygwinpackages/gearmand-0.17/util/error.cc:60: undefine

  5. James says:

    I take back my last comment. This might actually have to do with the latest version of cygwin. This is the actual error:

    CXXLD bin/gearman.exe
    util/error.o:/cygdrive/c/cygwinpackages/gearmand-0.17/util/error.cc:60: undefine
    d reference to `____xpg_strerror_r'
    collect2: ld returned 1 exit status
    make[1]: *** [bin/gearman.exe] Error 1
    make[1]: Leaving directory `/cygdrive/c/cygwinpackages/gearmand-0.17'
    make: *** [all] Error 2

  6. Wazzup Dear, What you ˙just written here really have me interested up to the last word, and I must tell you I rarely finish the full length post of blogs as I often got sick and tired of the junk that is presented in the junkyard of the world wide web on a daily basis and then I just end up checking out the headlines and maybe the first lines etc. But your headline and the first paragraphs were so cool and it immediately got me hooked. So, I just wanna say: nice and rare job! Thanks, really.

  7. jose says:

    Thanks man. I love you. I have gearman running on bsd and being meaning to run it on my windows 7 machine for a LONG TIME. A couple of things I am using gearman.23. During the configure process it asked for boost headers and libuuid. I just search on cygwin and install anything related to those names. After that I was able to complete the gearman install. I am still not done yet. Have to work with the php client libraries now. On my BSD machine I have the gearman.so and not pear. I need both environments to be the same. Thank you for the detailed post

  8. power says:

    I cann't install gearmand-0.28.Maybe miss some cygwin packages.
    $ make install
    CXX libtest/libtest_libtest_la-binaries.lo
    In file included from ./libtest/test.hpp:58:0,
    from ./libtest/common.h:56,
    from libtest/binaries.cc:23:
    ./libtest/cmdline.h:24:19: fatal error: spawn.h: No such file or directory
    compilation terminated.
    Makefile:4157: recipe for target `libtest/libtest_libtest_la-binaries.lo' failed
    make: *** [libtest/libtest_libtest_la-binaries.lo] Error 1

  9. Rik says:

    Hey power,

    I have the exact same problem. Any clue on how to solve it?

  10. benny says:

    Hi all,
    Have same problem.
    I made a re-install with all packages included. So I asume it's not a missing package issue. File "spawn.h" is missing in the "libtest" folder. Anyone have a glue?

    Error with #include in libtest/cmdline.
    —————————————–
    benny@computer /cygdrive/c/dev/src/gearmand
    $ make
    make all-am
    make[1]: Map '/cygdrive/c/dev/src/gearmand' wordt binnengegaan
    CXX libtest/libtest_libtest_la-binaries.lo
    In file included from ./libtest/test.hpp:58:0,
    from ./libtest/common.h:56,
    from libtest/binaries.cc:23:
    ./libtest/cmdline.h:24:19: fatal error: spawn.h: No such file or directory
    compilation terminated.
    Makefile:4157: recipe for target `libtest/libtest_libtest_la-binaries.lo' failed
    make[1]: *** [libtest/libtest_libtest_la-binaries.lo] Error 1
    make[1]: Map '/cygdrive/c/dev/src/gearmand' wordt verlaten
    Makefile:2072: recipe for target `all' failed
    make: *** [all] Error 2
    Error with #include in libtest/cmdline.
    —————————————–
    Error when header file removed in libtest/cmdline.
    —————————————–

    benny@computer /cygdrive/c/dev/src/gearmand
    $ make
    make all-am
    make[1]: Map '/cygdrive/c/dev/src/gearmand' wordt binnengegaan
    CXX libtest/libtest_libtest_la-binaries.lo
    In file included from ./libtest/test.hpp:58:0,
    from ./libtest/common.h:56,
    from libtest/binaries.cc:23:
    ./libtest/cmdline.h:58:24: error: 'posix_spawn_file_actions_t' has not been declared
    Makefile:4157: recipe for target `libtest/libtest_libtest_la-binaries.lo' failed
    make[1]: *** [libtest/libtest_libtest_la-binaries.lo] Error 1
    make[1]: Map '/cygdrive/c/dev/src/gearmand' wordt verlaten
    Makefile:2072: recipe for target `all' failed
    make: *** [all] Error 2

  11. ankit says:

    Facing exactly the same problem mentioned above

  12. fengclient says:

    i went into quite a few issues when building gearman v0.3x against cygwin. fine to build v0.14.

    to other visitors, please use v0.14 if u don't want to dive into these build errors.

  13. Yura says:

    Can you tell me how to run Net_Gearman?

  14. Yura says:

    How add gearman.so in php.ini

  15. Dave says:

    ./configure
    make
    make install
    worked for libevent-2.0.19-stable

    but for gearmand-0.33 ./configure hangs on "checking for dpkg-gensymbols…"

    Any clues what to look for?

    Output
    ======
    Dave@PC_VAN_DAVE ~/gearmand-0.33
    $ ./configure
    checking build system type… i686-pc-cygwin
    checking host system type… i686-pc-cygwin
    checking target system type… i686-pc-cygwin
    checking for a BSD-compatible install… /usr/bin/install -c
    checking whether build environment is sane… yes
    checking for a thread-safe mkdir -p… /usr/bin/mkdir -p
    checking for gawk… gawk
    checking whether make sets $(MAKE)… yes
    checking how to create a ustar tar archive… gnutar
    checking for style of include used by make… GNU
    checking for gcc… gcc
    checking whether the C compiler works… yes
    checking for C compiler default output file name… a.exe
    checking for suffix of executables… .exe
    checking whether we are cross compiling… no
    checking for suffix of object files… o
    checking whether we are using the GNU C compiler… yes
    checking whether gcc accepts -g… yes
    checking for gcc option to accept ISO C89… none needed
    checking dependency style of gcc… gcc3
    checking dependency style of gcc… (cached) gcc3
    checking how to run the C preprocessor… gcc -E
    checking for grep that handles long lines and -e… /usr/bin/grep
    checking for egrep… /usr/bin/grep -E
    checking for ANSI C header files… yes
    checking for sys/types.h… yes
    checking for sys/stat.h… yes
    checking for stdlib.h… yes
    checking for string.h… yes
    checking for memory.h… yes
    checking for strings.h… yes
    checking for inttypes.h… yes
    checking for stdint.h… yes
    checking for unistd.h… yes
    checking minix/config.h usability… no
    checking minix/config.h presence… no
    checking for minix/config.h… no
    checking whether it is safe to define __EXTENSIONS__… yes
    checking for g++… g++
    checking whether we are using the GNU C++ compiler… yes
    checking whether g++ accepts -g… yes
    checking dependency style of g++… gcc3
    checking dependency style of g++… (cached) gcc3
    checking whether gcc and cc understand -c and -o together… yes
    checking how to create a ustar tar archive… (cached) gnutar
    checking whether __SUNPRO_C is declared… no
    checking whether __ICC is declared… no
    checking "C Compiler version–yes"… "gcc (GCC) 3.4.4 (cygming special, gdc 0.1
    2, using dmd 0.125)"
    checking "C++ Compiler version"… "g++ (GCC) 3.4.4 (cygming special, gdc 0.12,
    using dmd 0.125)"
    checking whether byte ordering is bigendian… no
    checking for an ANSI C-conforming const… yes
    checking for inline… inline
    checking for working volatile… yes
    checking for C/C++ restrict keyword… __restrict
    checking whether time.h and sys/time.h may both be included… yes
    checking whether struct tm is in sys/time.h or time.h… time.h
    checking for size_t… yes
    checking for special C compiler options needed for large files… no
    checking for _FILE_OFFSET_BITS value needed for large files… no
    checking for library containing clock_gettime… none required
    checking sys/socket.h usability… yes
    checking sys/socket.h presence… yes
    checking for sys/socket.h… yes
    checking size of off_t… 8
    checking size of size_t… 4
    checking size of long long… 8
    checking if time_t is unsigned… no
    checking whether the compiler provides atomic builtins… no
    checking assert.h usability… yes
    checking assert.h presence… yes
    checking for assert.h… yes
    checking whether to enable assertions… yes
    checking whether it is safe to use -fdiagnostics-show-option… no
    checking whether it is safe to use -floop-parallelize-all… no
    checking whether it is safe to use -Wextra… yes
    checking whether it is safe to use -Wformat… no
    checking whether it is safe to use -Wconversion… no
    checking whether it is safe to use -Wmissing-declarations from C++… no
    checking whether it is safe to use -Wframe-larger-than… no
    checking whether it is safe to use -Wlogical-op… no
    checking whether it is safe to use -Wredundant-decls from C++… yes
    checking whether it is safe to use -Wattributes from C++… no
    checking whether it is safe to use -Wno-attributes… no
    checking for perl… perl
    checking for dpkg-gensymbols…

  16. Dave says:

    Okay ./configure works now.

    Configuration summary for gearmand version 0.33

    * Installation prefix: /usr/local
    * System type: pc-cygwin
    * Host CPU: i686
    * C Compiler: gcc (GCC) 4.5.3
    * Assertions enabled: yes
    * Debug enabled: no
    * Warnings as failure: no
    * Building with libsqlite3 no
    * Building with libdrizzle no
    * Building with libmemcached no
    * Building with libpq yes
    * Building with tokyocabinet no
    * Building with libmysql yes

    But now it hangs on make at "CXXLD libgearman/libgearman.la":

    libgearman/vector.cc:277:1: warning: visibility attribute not supported in this
    configuration; ignored
    CXX libgearman/libgearman_libgearman_la-worker.lo
    CXXLD libgearman/libgearman.la

    Anyone knows how to debug this?

  17. olga says:

    Hello! First of all I'd like to sorry for my english. It's not my native language :)

    My experience with the install process was hard, because I've got a lot of troubles during the installation. For example, "cannot find boost headers version" or "couldn't find folder libuuid" or "va_list has not been declared".

    Of course, I did everything according to this manual.

    Now I want to notice some things in addition to this manual.

    ******************************
    Step "Installing Cygwin Package Dependencies needed for Gearman".
    In addition to these packages:
    gcc
    make
    libuuid1-devel
    libiconv

    I advise to install
    libuuid
    boost
    **************************
    Step "Installing the Gearman job server, gearmand.exe"

    Firstly I downloaded the latest version of gearmand. And I've got error messages during make and make install.
    I advise to download gearmand.exe v.0.14
    as posted by fengclient.

    I wish my experience will be usefull for you.

    Thx.

  18. Nausher says:

    Another error I saw when running configure was the following :-

    $ ./configure
    ./configure: line 16: $'\r': command not found
    ./configure: line 31: syntax error near unexpected token `newline'

    I was able to fix this by converting the configure, config/config.sub, config/config.guess, config/config.rpath, config/missing & depcomp files into unix EOL format (I used Notepad++ to do this manually).

    Along with this step I had to follow Olga's suggestions to add 'boost' to and using version v0.14 of gearman.

  19. [...] which share an API interface. Gearman works best in *nix environments, but with a little hacking can be setup on windows machines also. Check here for some sample of code using Gearman and here for integrating Gearman with the Zend [...]

  20. Could You upload the compiled binaries?
    I am using Win7 x86 – thanks :)

  21. Tom says:

    Hi,

    Fine to compile both 0.14 and 0.23 with libevent 1.4.14, but when I start gearmand with -t option, I have a memory Leak.
    Is anybody having the same problem?

    Thxs

  22. GeertVW says:

    I tried the above. The server part works with v0.14.
    The Net_Gearman installs well.
    But then, it still does not recognize the new commands in the php, like $gmworker= new GearmanWorker();
    I guess what we need is still to add an extension dll in php.ini?? But how to get that? phpize is not available. Have been searching quite a bit, but find no working way to have this imported in php on Windows. Any help really welcome.
    But the worker part does not

Leave a Reply